Doing something you're not good at is never fun at first. Hell, it's damn hard at first. The only thing you can do is keep doing it until you're better (and then it becomes more fun!)
Anyway. Think of it this way: if you are truly dedicated to becoming proficient at Japanese,
YOU HAVE TO SPEAK IT. There are no ways around this.
The hardest thing is convincing yourself to go out and do it, and to let go of your ego that cares if you make a fool of yourself in front of others. Changes are definite that you will make mistakes, and chances are VERY good you'll embarrass yourself at some time or another, or offend someone, or make a situation awkward due to your lack of ability. Get over it.
You "find it easier" to learn from the books because they're safer. A book won't reject you if you make a mistake. A book also won't make you grow as a learner of the language. It's just a crutch.
If you want to get better, dedicate yourself to speaking or trying to speak at least 2-3 hours a day. Even just sitting around people and eavesdropping on their conversations is OK - just do it.
In my first experience in Japan, I had about one year of study under my belt. Once I got there, I cut class and hung out with Japanese friends for about 10 months. At that point, my speaking ability had surpassed many people who arrived with me who had 4+ years experience.
When I got home every day, my head hurt. I can't remember a single day during that 10 months that I wasn't tired from the mental strain of speaking/listening... but... then everything got easier. I just started understanding things without having to work at it - it was a glorious time for me.
From what you've written, you're looking for this type of experience. The only way I know how to get it is to do what I just did - you'll only learn as much of the language as the amount of pain, work, fatigue you're willing to endure.
My position in a nutshell:
be stubborn, just do it until you're good
P.S. Getting a girlfriend also helps tremendously