It's really difficult to answer the question without knowing you personally.
For instance, if you judge "worth it" by how useful the language will be to you, it depends if you are thinking of moving to Japan, going there on vacations, doing business with Japanese people, and so on and so forth. If that's the case, then I should think it would be pretty useful!
If you want to learn Japanese for the pleasure of learning a language, well, I guess it depends on the way your mind works, how you learn, and that kind of thing. Personally I'm really enjoying learning Japanese! But to someone else, it might seem like a torment - I have no idea!
Anyway, to take another point... I guess that to learn a language reasonably well, you need vocabulary + grammar + speaking/understanding skills. And in the case of Japanese you need to know the scripts too. It seems like your method of learning English worked pretty well for you. I would think that a similar method of learning Japanese would work, too. But you WILL need to learn kanji, as well as hiragana and katakana. But don't let that put you off! Because learning kanji is fun, and not as hard as it looks before you start.
Then again, perhaps my brain works in such a way that it absorbs kanji easily; I don't know your strength on learning stuff like that. So the only way you can know is by trying it.
I would definitely say "go for it"!! Even if you decide not to carry on, you have only gained knowledge and not lost anything.