It's a strange difference, but tall is the actual height of something from top to bottom, while high is the level of something over the earth, in an altitude sort of way. In that way, Everest is the highest mountain, but it's not the tallest mountain. The tallest mountain is one in the Pacific ocean somewhere, mostly under the water.
For lots of English speakers, they are interchangeable, technically, they aren't. Perhaps Fuji is also the tallest?