Very good choice to avoid Golden Week!
While you might be too late for cherry blossom season, you might want to check out shibazakura or wisteria. As for shibazakura, you have several options.
My favorite one is the
Fuji Shibazakura Festival, but I also liked the ones at
Hitsujiyama Park in Chichibu. Both can be easily done as a day trip from Tokyo.
As for 3 weeks, I think 50% in Tokyo (+
day trips) and 50% in Kansai (e.g. 1 day in Osaka, a few days in Kyoto and a few day trips to Kobe, Himeji, Nara etc.) might be a good balance.
There are several festivals in May.
Maybe you can check out the
Kanda Matsuri (May 13-14) in Tokyo or the
Aoi Matsuri (May 15) in Kyoto.
If you decide to visit
Himeji Castle and want a more local festival experience, I recommend the dragon boat race festival in Aioi called "
Aioi Peron Matsuri" (May 27-28).
You're probably already in Japan now and I hope you enjoy your trip to the fullest! ^___^
Maybe you can report back and tell us how it was and where you went in the end.
Thank you.