I was much like Mikawa Onsen above. I landed in Japan with a little bit of Japanese knowledge, and went straight to the Japanese countryside where very few people spoke English. These were the pre-internet days. I carried around a small Japanese-English dictionary everywhere I went, and practiced writing kanji in my free time. I survived largely because of the generosity and kindness of other people. Kindness at the post office, the gas station, the photo-developing place, the train station, the taxi driver, etc...
A 10-word vocabulary quickly became a 20-word vocabulary, which then became a 100-word vocabulary. Sentences were made using nouns and verbs, sometimes in the correct order, sometimes not. Then, I gradually became aware of more complex sentence structures, verb conjugations, levels of politeness, etc... I got "conversational" (kind of) within a year. But I didn't get good until many years later - maybe 5 years. And by "good" I mean I could say a lot of things, but it wasn't always correct and wasn't always pretty. Took me 10 years before I realized how crappy my Japanese was, which allowed me to start getting better. 10 years was about the point at which I could buy a book (in Japanese) explaining the Japanese real estate tax system. But then, you get an event like a possible nuclear meltdown, which requires a highly specialized vocabulary, and you realize once again how much you don't know.
There was a comical novel about an expat in Japan called "Max Danger: The Adventures of an Expat in Tokyo" which was written by a foreigner with a keen eye for the absurdities of the privileged life of a foreign executive in Tokyo in the late 1980s. One of the lines in it stuck with me - it was an observation that the protagonist had used the Ginza subway station every day of his multi-year stay in Japan, but he had never learned how to write the kanji for "Ginza". For some reason I didn't want to be that guy, but it took me a while to learn how to write Ginza. I'm still not 100% confident that I can do it with the correct stroke order.