And how long have you lived in Japan, JohnAshleigh?
I must say that I'm surprised at the amount of people claiming that the cure to this problem is to get really good at Japanese. Once again, I've only been to Japan for travelling once, for a week, and thus I've never actually lived there but I've lived in Taiwan for a time (I'm entering my 6th year now) and I was thinking that at least in this regard, both mentalities/cultures should be at least a bit similar. So, I don't know if you people are right or not but I can tell you that in Taiwan, no matter how well you speak Chinese, you will always get remarks on how incredibly well you speak it (from the "hello nice to meet you" level to this one where you can comfortably discuss about quantum physics in Chinese). This is not due to your level but to your face: may you be white, black or green, you don't have the right skin color to be a native speaker in their mind (no racism here, just habits: Chinese-speaking though foreign-looking people don't really make a majority, to say the least). Proof to it is that Asian-looking foreigners don't encounter this problem.
It the same logic that makes people never ask you for directions in Chinese. I mean why would they? Chances are even that you're only a tourist and won't be able to help them. The only two times people asked me for directions it was to find a tourist attraction that, they knew, had to be in the area, and they did it in English. Again, it's only logical: what indicates them that you speak Chinese and have lived in the area long enough to be able to answer such questions? It's not written on your forehead (except if you are wearing a t-shirt saying, in Chinese, "white people don't understand this" and on the back "neither do black people" and they understand what irony means).
Of course it only works with people who hear you speak the language for the first time: you'll startle them, in a way. They don't expect you to speak Chinese but English. Sometimes in their mind they'll be like "what I'm actually understanding what this guy is saying" and be distracted from the message you're trying to convey. Then they'll ask you to repeat and once you're done they'll compliment you on your Chinese before answering your question. It's not a fatality of course, and it doesn't happen all the time with everyone, but it will happen now and then, no matter how good you can get at speaking the language.
You can only avoid it by talking on the phone or writing emails, simply because the person doesn't see you.
Now again, maybe you're right and it's different in Japan but I kind of doubt it.
Another annoying thing is like when you go to the supermarket and the elevator is kaput. The guy in charge of redirecting you to the stairs will stop for a second when he sees you and then he'll start searching for his words in English. How to put it? That's fate. Only surgery can solve it.
And all of it is perfectly normal and understandable. What is less, though, now that I think of it, is these people who absolutely don't know you but find and interest in trying to do so and approach you by asking "where
are you come from?" without even saying hello. That would be something that I can't understand nor tolerate. And to say that Taiwanese people have a reputation for making numerous detours before getting to a point. Think of a more straightforward way to make you understand that you don't belong. And the worse is that these people too are usually well-intentioned. Well-intentioned but terribly awkward, to say the least. And there's a bunch of them out there, trust me.
Now think for a second about these children from white American couples that were born and raised in Taiwan. There's not many of them but there are and have been for at least 30 years, probably more. I met one of them once, at a pub. His Chinese was the best I had ever heard from a white-looking person (talk about a statement of the obvious

) but when he goes shopping Taiwanese parents still tell their kids to go and speak English to him for exercise.