3 years starting from when? Studying in college? Studying on your own? Or living in Japan? I study very hard but I am not a particularly social person, so my speaking/listening skills lag behind my reading/writing. But I would say that after one year of living in Japan (and working as an English teacher), I'm somewhere between small talk and hold my own--I couldn't participate in a native-native conversation at normal speed, but I can talk about most things I want to say in somewhat broken japanese. If you're talking 3-5 years living in Japan, don't sell yourself short. If you're talking listening to podcasts overseas--well, I have no idea.
ETA: Before I came over I worked my way through 2/3 of the Genki 1 textbook on my own, but had never really spoken the language.