When I was in Japan, I saw many eldery people walking up and down the temples. I went to Shikoku and also saw a few elder people walking to complete the pilgrimage.
I guess the only thing I want to say is that, don't let your age stop you! I have met many amazing travellers at 70+ years old, backpacking and travelling by themselves. I just met a 72 years old American visiting Xi'an, China all by herself and not knowing how to speak Chinese.
Good luck and enjoy Japan!