Hello, my initial thought is that the name that is written the largest, Inage Eisaburō, would be the intended recipient's name. (The flag has several signatures from people in the Inage family). However there is no "honorific" attached to the name, so that feels sort of strange to me - usually you would expect some kind of honorific attached to the name.
Regardless, you have the location name, and you have several signatures which should be sufficient for someone with resources to be able to search out at least the exact location and the descendants of some of these families. I presume you've already checked out the "Obon Society" website. Also the Japanese embassy has started to accept items like these for repatriation back to the families of the original owners. I don't know which one is better, faster, or more satisfactory.