Thanks for the replies!
To clarify: I have journalist visa. The Swedish companies I do freelance work for have no offices in Japan, or any presence whatsoever apart from myself. My salary is paid in Sweden, in Swedish kronor (the local currency), to my Swedish bank account.
The income I earn while in other countries is from the same two Swedish companies, yes.
As for my obligations in Sweden, everything is absolutely clear: I'm not supposed to pay any taxes there at all, as long as I don't spend more than 72 days per year there. (This is what I've been told multiple times by the Swedish tax agency.)
The "advice" I got from the Japanese tax authorities was actually (unfortunately?) verbally, at the time of my filing, back in March of this year. I went to the temporary filing place in Shinjuku believing my Swedish-source income was not taxable in Japan, only to be sternly corrected by the tax agent assigned to me. However, he then asked me to deduct all income earned while I've been in other countries from my total taxable income for the year (2013). So I did all the calculations right there, in front of him, until he was satisfied with the result.