Two years ago, I participated in a group tour of Japan with a native guide. For most other members, it was simply yet another country to see the sights of, and so, they behaved in public like they would at home: talking loudly everywhere they went, including on trains and buses, too engrossed in conversation to notice everyone else spoke softly or not at all.
Near the end of the tour, the guide confided to me that she often stayed at a distance from the group during travel - because she was ashamed to be associated with them.
That said, if you don't make too much of a scene, people will generally ignore you. In the total of six weeks I've been there so far, I've never had anyone stare at me, make remarks behind my back, or subtly distantiate themselves from me in public baths. So as long as you try to act at least a bit like the Romans Japanese do, I don't think you'll have much to worry about as a tourist.