I've recently started studying more seriously with the goal in mind of passing JLPT1 by next December of 2009. Based on the number of hours generally accepted as being the required amount to do so, I have enough time and certainily enough 'yarigai'. So far I'm about 2/3 of the way through my JLPT2 grammar mondaishuu and I've picked up 90% of the kanji introduced up to that point. Still a long way to go though, a long long way.
You'll find that as you start to get into the thick stuff, the pay-out is at first slim. You start to understand much more in terms of formal structures (stuff in newscasts, newspapers and articles, etc), and reading-wise a lot of that is increasing your kanji knowledge base of course. But speaking-wise, because it's not everday convo. type stuff, you don't feel like you're improving much because you're still stuck saying most of the same things as before. Actually to really understand everyday convo a whole different kind of grammar is required, stuff you may not likely find in any JLPT book. For that you might need some speech-coaching from a native speaker, or simply to live here and absorb it more organically. Everyday convo. grammar is a trimmed down, leaner version that's meant to help one navigate high-context conversations with as little extraneous and unnecessary speech as possible. For that, the shortened forms of formal structures are belted out in rapid succession, which is why as you said, it sounds like 3 billion words a minute and very little is intelligible at first.
If you want to break through and really become an intermediate/advanced USER of Japanese (literacy and speech), I suggest first trying to study for the JLPT so 1) after everythings said and done you have some useful accreditation that actually gauges your literacy and could be helpful in getting a job some day and 2) you then have a solid base of vocab and grammar to work with and hammer out your speaking skills as you go along. To that end, native-speaker exposure is def. necessary and living here would be extremely beneficial.