How polite is too polite?

I'm afraid I still find the levels of politeness in Japanese language hard to ascertain sometimes. I've been friends with a guy for a few years who I got to know through other friends. We're not exactly close, but we've hung out a lot with other friends there, and we've had a few more intimate talks (he basically liked me, but I wanted to be friends - yes I know I suck). However, I only recently found out he's 3 years older than me (he has a baby face). I thought he was younger than me the whole time and was talking to him in plain Japanese speech.

Now I'm unsure if I was being rude. Should I revert to polite speech or would that seem like I'm putting up a stiff, polite barrier? I had another friend who always talked to an older girl with the ~です and ~ます form, but he talked to me in plain speech even though I was older and said it's because we had a more casual friendship. Does that mean it's ok for me to use plain speech with friends who are older than me? Sorry for the essay, just wanted to give background.

I highly recommend to keep using casual forms. If I were him and heard you suddenly started talking in polite speech to me, I would wonder if I might do something wrong to you and be anxious about it. It would no doubt sound unfriendly. No problem to use plain forms between casual friends, as one of your friends said. It doesn't sound rude even if he is older than you.

Ok, thank you very much for your advice, Toritoribe. I appreciate it.