Yeah.
You said to him you're terrible at Japanese, right?
He meant, 'But still, I'm impressed, because you know kanji!'
If he said でも、漢字を知ってるからスゴイ!, it might have been easier for you to see he was referring to you, I guess.
We drop subject(who, I, You, etc.) very often(or in almost all cases) in a casual conversation, so if you are not sure who the other person is referring to, you could ask them 'Who? Me? or You?'
I sometimes do that myself, when I can't guess from their face expressions, tone of voice and whatnot...
Btw, I guess you're mixing up Sugoi with Hidoi...
Sugoi means, good, cool, amazing, awesome etc.
Q