Not being physically in the country and experienceing Japanese first hand, it is very difficult, if not downright impossible. One must be exposed to it almost 24/7/365 to become aurally and orally proficient.
As I lived in Japan for many years and have now lived in the states for almost 20 years, I can safely say that, although my speaking ability has decreased alot, my listening comprehension is still near 80-90%. Why? I have no answer, I can tell you this though: If I watch a Japanese drama or anime on TV with no sub-titles, I can understand 80-90% still to this day! But ask me to repeat the same thing in Japanese again and I would have difficulty thinking of the proper sentence structure and words.
However, once I return to Japan for visits, for some odd reason, after a few days, it all seems to come back again and I am able to speak coherently. Weird, but true. Maybe someone with more linguistic ability can explain this phenomenon.