Can't recall the correct consonant when studying Japanese

Hi I've been studying Japanese on and off for years, and I've noticed something:

When I screw up and remember the pronunciation wrong its usually the consonant, not the vowel that I misplace.
For instance today I thought NI RA MU (to glower) was NI A MU, though other times I'll wonder if its NI GA MU.

The point is I don't have problems remembering the vowels. In fact, usually in a song I hear the vowels right--its the consonants that cause me the most trouble and seem less distinct to me. It might be me, but I think it might also be because Japanese sounds so "flat" without dramatic facial contortions for the consonants. I don't have trouble remembering the number syllables in a word either by the way.

So my question is, do others have the same problem of not being able to remember the right consonant?

That makes sense with music. When it comes to reading, I usually mistake the vowel.

Is it particular consonants, or particular situations (like when you only know the word from having heard it, not seen it written down, does that make it worse?)

It's not unusual to partially forget things (I went through a period where I kept thinking あらわす was あわらす). I find writing (physically or typing) helps me. Your mileage may vary, but I have seen research to suggest that combining things (such as listening with reading, transcription, reading aloud, sounding out a word as you write it) aids recall. Try varying the way you learn vocab and see if anything helps.

Maybe it is a western thing that we are wired to respond to rhyme from an early age? Keying onto the vowel seems natural, especially when learning new words (courtesy of Dr. Suess and other 1st grade primers). The vowels provide the flow, and the consonants the structure. Structure is harder, and requires repitition.