Well after a few hours with the 3 inch wire wheel on the drill press I got sick and tired of it. Ran to the store picked up two pulleys and grabbed an old motor and shaft I had laying about. Ended up with a great bench wire wheel that I can just swap out the brush on the end to whatever size I need or want. Total cost of project $18.67.
I only have a 5 inch wire wheel on it right now and haven't worked out what kinda stand I am gonna put it on. But it will hold up to a 12 inch wheel. Also I can throw a grinder wheel on there and use it for that too.
Just wanted to share. Figured it might help someone down the road when they need an idea.