got back from Tokyo yesterday, the weather is a bit chilly but layers will be fine. i wore t-shirt, sweatshirt, jeans and a windcheater most of the time and was fine.
if you go anywhere like the mountains it's freezing though
the shops, subway stations, trains etc are all heated (too hot for me, to be honest) and a thick winter coat was too much, when shopping in the city centre etc.
not sure how far south the places are you want to go in Japan are, but its a lot warmer than the winter I experienced in the US once (Virginia)