Don't worry about that too much its not that serious,
building vocabulary comes with time but i would suggest you to look japanese tv online, drama and funny tv shows as much as you can, starting by concentrating hundred percent on what they are saying and understand by looking at the context.
Soon you will find yourself hearing the same words in different type of material and the new words will come and stay naturally in your brain because you hear them again and again.