Put the manga away for a while. Work your way through at least Genki I (and Genki II would be even better) before you attempt decrypting Japanese text. Right now you know just about enough to confuse yourself and you'll have to re-plow the same ground to unlearn the stuff you're mis-learning with this current approach.
It's going to be a few years before you can read manga as you would like to, so there's need to get in a rush on the front end.