Where on earth did you get the idea that ツ静ヲ can only be used for time? It is used for describing location of physical objects as well.
The problem with these two is that we early on learn that ツ静ヲ means "before" in English, which is an inaccurate oversimplification. It is best to think of it as meaning "beyond" some referenced point. In the case of time, it can be either before something or after it. But it is always on the other side of a referenced point as viewed from your current position on the timeline and not from the viewpoint of seeing the timeline as unidirectional, starting from the earliest point and continuing to the latest point. "Before" is always an absolute reference, while ツ静ヲ is a relative reference. Once you realize that, it's dead easy.