What Toritoribe-san means is that she could be describing the situation from the perspective of the others who have only just learned that 十香 moved nearby.
"(As far as they know), you guys haven't walked to school once yet since she moved to this part of town. (Since 士道 was originally keeping it secret that they were actually living together, they don't know that she actually moved from closer to 士道 to slightly farther away. Just that she recently moved to around here and thus it's strange that they don't go to school together even though they live next door to each other.)"