Although the Yamaguchi gumi's head office is only a 10minute walk from my House I don't really see anything. There was just one time when I went to Nada onsen (5min south from their head office) when I saw a guy covered with tattoos.
And yeah, the chimpira guys wouldn't dare to be in the HQ area.