Which Japanese?
Managers who have foreign workers?
Co-teachers?
Shop keepers?
Spouses?
The average Taro on the street?
Kids with foreign teachers/ALTs?
Say "arigato" or "domo", and the average Japanese will say your language skills are fantastic. It's a way of complimenting people (over the top, of course), and in my opinion this is done simply because they don't expect a single word of Japanese to come out of your mouth. Granted, Japanese is difficult to learn (written form especially, because you can't sound out letters that you can't even pronounce), but IMO the Japanese have an over-inflated opinion of how hard it is to learn to speak. If you can manage a line or two, people visibly appear more comfortable, but that's only natural, and it shows (again, IMO) that they weren't expecting anything in the first place.
There are always those who MIGHT expect you to know it, though, simply based on your job situation. Others (like waitresses and clerks) will often defer automatically to the Japanese friend/spouse that accompanies you, and thereby totally ignore you before a single word comes out of your mouth.
Overall, I'd say the Japanese are very polite about accepting people with little to no language skill, but that might be a cultural viewpoint they are expressing