It's a bit too elusive. I think they are kanji, but just too stylized to read. They kind of resemble bonji (Siddham script) characters, which are sometimes found on Japanese items. But none of them really resemble any specific bonji, and 4 bonji on a yosegaki flag would be extremely unusual indeed. So I think its a 4-character idiom (super common on these kinds of flags, as you will know because there are many smaller ones scattered around the flag).