The custom actually varies geographically in the US. In the Midwest, where I grew up and most of my relatives live, people don't take off their shoes when entering a home. My poor Japanese wife had difficulty convincing visitors to do so. No, I should say, I did; my wife was too polite to ask, so she expected me to do it.
Here in Alaska, everyone removes their shoes. It's a given when visiting someone's home, just like Japan. I think practical considerations make the difference. In the Midwest the streets and walks tend to be washed clean by heavy rains. Here, we have snow and road silt (the city and state dump silty sand on roads to help with slick driving conditions) to deal with six months of the year, and it never rains hard enough to clean the stuff off in summer.