My non-native understanding is this; linguistically it can take on either meaning, since word order is rather flexible in Japanese. However, since the subject of the passage is the temple, and not the mountain per se, the most common interpretation would be that the "quiet" refers to the temple (which is located on the mountain).
静かな山の寺
quiet mountain temple
If you wanted to specify that the mountain was quiet while ignoring the volume level of the temple, you might say
静かな山の中にあるお寺