I seriously doubt that you were headed to a mixed onsen. as stated above, these are rare, and people will were bathing suits. It sounds like you backed out before you saw there were two doors.
There is the option of "family only" in some hotels. Still, I don't think your family was going to make you do that. With older kids, I doubt they do that when you are not there.
While we are on the onsen thing, Japanese people do not wash before entering the onsen. First they get in the onsen and soak, get out and clean, then get back in. This is oposite from most of the books and discussions i have had. I have been to many onsens, and I guess i was so busy making sure that I washed first that I didn't notice that other people didn't.
I say most people beacuse it seems that younger adults do wash first, but my father-in-law gets right in along just like the other older men.
I wash first, and I don't think I will ever change that.