I've never heard you speak Japanese, but I doubt you sound like a (Japanese) girl. More likely, you sound like an American guy trying to speak Japanese.
This reminds me of a classmate of mine back in my college Japanese class. He somehow got it in his head that the audio materials we were using were giving him a Kansai (western Japan) accent, instead of the "proper" and "standard" Tokyo intonation that he wanted to learn. Never mind that he hadn't been studying the language long enough to distinguish between various accents/dialects, much less accurately imitate them.
He brought this up one day in class in a somewhat insulting and accusatory manner. Our instructor's response was, "Let me assure you that the speakers in the audio materials are Tokyo natives who were chosen specifically so there would be no traces of dialect in their speech. Even if you were to somehow acquire a Kansai accent, though, I guarantee you that it would be preferable to the Miami, Florida accent that you have right now."
The moral of the story is: get a basic working proficiency with the language before you start concerning yourself with what "flavor" of the language you're speaking. If you're still at the point where putting together a basic grammatical sentence is a struggle, whether or not your speech is "masculine" enough is really the least of your worries.