These are the things I find hardest:
So many words seem so similar (in sound), and non-distinctive, and difficult to remember. My acquisition of vocabulary seems painfully slow.
Obviously learning the kanji is a lot of work, but I'm disappointed that even when I know the core meanings of all the kanji involved, I often can't figure out the meaning of kanji compounds.
I find it very hard to remember kanji readings -- much harder than meanings. There are so many characters, so many readings, and, again, so many are duplicated/non-distinctive.