Yeah, the derogatory sense is from the tone of the whole sentence such like まったく, これでも, よく言えるな, and especially, ~ときたら. この人 also sounds too stiff and formal for their mother, and the speaker is used this word in that nuance here, but it's relatively neutral, I think.