Everyone is different and it depends on how bad one wants to learn the language. While it is easier to learn the language while living there and immersed in the culture, it does not mean you will learn it. After one learns that they can get around quite well without hardly knowing the language, many abandon their efforts and go the lazy route where they just know and use elementary words. These people can easily be identified in, for example a bar or other establishment where they will say something like, "Watashi beer dozo."
I've known more than my share of foreigners who have lived in Japan for ten years or more and know nothing of the language save for a few sentences here and there and a basic vocabulary with horrible pronunciation.
If you have a fairly basic understanding of the language it would be best to live alone, away from other foreigners and limit your contact with them and you will have no choice but to learn the language as it will be required in your daily activities of shopping, banking and getting around and, as they say, practice make perfect. Living among the Japanese with limited access to English will greatly help in your efforts to learn the language fairly quickly if that is what you desire.