The YouTuber named Paolo makes a very good point in the video linked to below. The idea of teamwork is very strong in Japan, and one way this manifests is how Japanese people are very courteous and try not to inconvenience other people. At 2:53 in the video Paolo tells us how Japanese do not stand in the way on a sidewalk and try to move to the side so they don't block the sidewalk. When foreigners do this, Japanese people see this as very rude. Look at 3:24 and you will see how Japanese people stand to the side to keep the sidewalk clear for people walking by.
It is these kinds of things that cause many Japanese people to 'have a problem with foreigners'.