Think of it as: here, there, and over yonder
Seriously, though, it can be highly subjective and it really isn't worth worrying about too much. The first apartment I lived in here was on land newly reclaimed from the sea and our building was one of only two or three that had even been built yet. There were no bus lines, no trains, and I often had to take a taxi home from the station. When we reached a certain point, I would say to the taxi driver (while pointing), そのアパート, and he would confirm, あのアパート?....then I would feel dejected at having gotten it wrong. Determined to turn it into a learning experience, the next time I would say (at the same point), あのアパート....only to have the driver confirm, そのアパート? I made sure to conduct this little exercise at the same distance from the apartment each time, and what I learned was, "It's subjective". Don't lose sleep over it.
The differences between それ/その and あれ/あの....yes, you do have to get those right.