That's because ie is also used in a sense rather than just referring to the physical building itself. It can also mean household/family/"clan".
Consider the use in terms such as 本家 and 分家 and as a collective indicator after family names, and indeed as a component of the word 家族 in order to get a better idea of the concept of 家. Although we usually translate it as "house" or "home" in English, that really doesn't give a complete or exact sense of how it is used in Japanese,