Listen more, speak more.
I'm not being glib here - you're suffering a quite common problem which comes from (I'm presuming) residing in an area where you don't have regular chances to interact with other Japanese speakers. Most of your study comes from books, rather than audio materials, so you're better at reading than listening. Production (writing/speaking) always lags behind comprehension (reading/listening), and you do need to specifically practice production, it does not come naturally.
It's hard to recommend materials as 'intermediate' covers a lot of bases; I'd suggest speaking to your teacher and asking for additional material/drills. Drills that involve practising writing/grammar production (i.e. forming grammatical sentences, conjugating verbs) may also help you with speaking.
I can't recommend one in particular, but you might also consider getting a tutor over Skype or similar, just for conversation practice.