Around where I am there are places way back in the mountains where there used to be temples long ago. They fell victim to declining populations around them as people shifted from backwoods rural living to move into town. Some were more like monasteries and no doubt were victims of their isolation....severe even for the times....and the forest has reclaimed them. I know of one place from finding a very small rural graveyard alongside the road and finding a small handwritten sign from somebody pleading with people not to forget the temple it was a part of....a temple which has vanished without a trace.
More commonly seen in very rural areas....back in the hills-n-hollers....are temples which don't have a resident priest, yet which are not "abandoned". They typically still have the altar and associated appurtenances in place and temple duties and rituals are performed by a priest attached to some other temple on an as-needed basis (typically associated with funerary rites).
Also, not everything that we might take for a temple is actually a temple. There are 観音堂 (Kannondou) and 薬師堂 (Yakushidou), for example, which some might mistake for temples. Despite not having residences or resident priests, they too aren't necessarily "abandoned", as there will be some sort of committee in charge of the place, probably in association with a priest from a temple.
Buddhism is alive and well in Japan, with practically everybody at least dying Buddhist, whether they were particularly devout, knowledgeable, or even mildly interested in it when they were alive. The devout take it quite seriously; for others it is just the socially expected manner in which you have your funeral service. Some are quite well informed of the precepts and teaching; others are utterly clueless yet will answer "Buddhist" if you ask what religion they are. Buddhism is a religion that doesn't make a terribly great number of demands on its adherents in Japan. According to my own understanding of the matter, since every sentient being has 仏性 (Buddha spirit) the dichotomy is not one of "Buddhist" or "non-Buddhist" but rather one of "People who know they're Buddhist" and "People who don't know they're Buddhist". In other words....you can't not be Buddhist, as all of the precepts and teachings apply to you anyway, whether you know it or not. The lack of proselytizing pressure probably arises from the idea of 輪廻 (cycle of rebirth), which means if you don't figure it out in this incarnation, maybe you'll figure it out in the next one, so there isn't quite the same urgency to "save" a sinner as one finds in some other religions.