This might sound like a stupid question, and I really am not wanting to sound sarcastic... Did you have some ideas about what job you would like and/or be able to do, when you move to Japan? You mentioned that you were going to be a physicist in the US, so maybe you could use those skills to find work in Japan? :? And learning Japanese at the same time
But, if you don't have a job at the moment, it's difficult because of the money... You feel maybe different to me, but I can only say, if I was in your place, first I would find some job in the US, it doesn't matter what job, but just to earn some money. Then, to spend some money on taking a Japanese class and save up the rest for your trip.
Doing a crap or boring job can be depressing though... don't want you to get sick again ☝
But, if you have an aim for the future, it is some different motivation, so it is seeing beyond the moment, and especially if you know the time is limited, you are working towards your goal so maybe don't feel bad....
Also, when you go to Japan (whether it is straight away or later) your Japanese skills will surely improve quickly because if you live in a country, that is only natural! Even if you think your languages ability is average, I am sure you would learn fast if you were living there.
But, of course if you're going to work in Japan, you will need to have some Japanese language. Anyway, you can try home study if there is no class near you, and also find Japanese penfriends, can improve your language skill. It sounds like you are determined in what you want to do, so I say, Good Luck!
But, it is well to remember, please make sure you are in good physical and mental health before you start on the big adventurous project... One thing is, if you get sick in another country you could find it more difficult to explain what is the problem and understand the treatments for you... Another thing is, you move away and leave behind friends, family and social life, and you have to start another social life from nothing. This can be difficult if you don't know the language well. So, maybe you feel lonely for a while... so it's important to be strong enough to get through that time. Maybe you don't feel very close to family or friends even in the US, but, when you move, it feels very different in surprising ways.
I think I have been preaching enough! :sorry: I hope you get some good advice from people on this forum