I hated having to ride on the sidewalk because I was more afraid of getting hit by a careless driver than running into a careless pedestrian. On the bright side, it made me a very cautious cyclist, though it doesn't seem to have the same effect on others. I imagine that likewise, bicyclists imposing themselves on the roads must be a huge nuisance to drivers.
Japan could certainly be more bike-friendly. It could also be more wheelchair-friendly.