Heh, I've done it here and there. If you can pull it off it would be strange, but sounding like a native Japanese from anywhere is still sounding like a native Japanese (plus the people in 関西 get a kick out of when you try). When I was in Boston I heard there was a Japanese guy who had spent lots of time in south Louisiana and spoke English with a southern drawl, and the guy telling me about it said it was strange. I unfortunately never got to hear it, although I wanted to. I'm sure it would have taken some getting used to, but it would have been pretty cool.
I once saw an east Asian guy speaking fluent Spanish with some people, and that threw me off, because those sounds should not have been coming out of that guy's mouth from my experience at the time. But the strangeness wears off when you're faced with something all the time, so the people you know would find if normal after a period of time if you spoke 関西弁, but if you sounded foreign, you'd always just sound foreign. Speaking of which, a westerner speaking Japanese period will be strange to lots of Japanese people.
About your age, that guy was probably going by the sound of your voice more than anything else.