To read Japanese yes you need to know all those kanji. But if you do not intent on writing or reading much, I would focus on speaking and it's much easier. Of course doing both at the same time is the best way. It is a lot to remember, but Japanese people study throughout school, so don't expect to learn it in 1 year, or even two...you will continue to learn.