I thought it was OK to have it directly before しました, if you really want to put the focus on it being
yesterday that you played tennis with your brother. Of course, it may be better to just phrase that as (私は)弟とテニスをしたのは昨日だった。but anyway...
Can I get some confirmation on this? I may be coming from 岡山 or 北九州 with this one too, or maybe it will be 青森 or somewhere else in 東北 this time.