Wouldn't that affect your refund? Also wouldn't that be more problematic if you ever decided to return to Japan as it's a national obligation, not local?
For the record, I think it's better to bite the bullet and pay your obligations. Yes I know, I'm not answering your question.