Making the VERY top of my list:
1. Xenogears (PS1)
Quite possibly the best story ever told in an RPG. I forgave it's flaws (battles needed tweaking, dialog was too slow), and still loved it immensely. Though written by the same person in a similar universe, the Xenosaga series never lived up to it in the first two, and was cut from a single storyline spanning 6 games down to 3 games... then the 3rd game had to go and own me. I'm playing through it right now, and it quite possibly has the best dramatic storytelling I've EVER seen in a videogame (this one is a rising star on my list)
2. Final Fantasy 6 (SNES)
This one is often forgotten, as it was neither the first final fantasy most people played (e.g. FF4 and FF7), but it had everything you wanted - an ensamble cast, a fun and likable villain (Kefka), and a ton of secrets. It was not revolutionary or new - it was the embodiment of all the experience and polish that those who made games had collected over time.
3. Grandia II (Dreamcast)
Hands-down the best battle system of any turn-based system in any RPG. It improved the system from Grandia I, and added a really intriguing story and fun, fun, fun! A classic that should not be forgotten.
4. Dragon Quest 4 (NES)
This game was so incredibly awesome. It did incredible things on the NES, had a massive world, and some of the best storytelling of the day. It's chapter system gave variety, and gave you a story that brought together a lot of people. This one was largely unknown outside of Japan, however.
5. Paper Mario: The Thousand Year Door (GC)
You're probably saying "huh?" at this point, but this game is so incredibly fun you wouldn't believe it without trying. Between the incredibly beautiful visual style (2d polys in a 3d world), and the funny, quirky world that Mario and friends live in, this game is really, really good.