When to use "no ga" and "no wo" in nominalization?

hello, i am anjani from india. i am trying to learn japanese and this thing called "nominalization" is just going off my head actually my question is "how to know to use when to use "no ga" and "no wo" in nominalization?". eg) musuko ga piano wo hiite iru no wo mimashita. why have we used "no wo" here not "no ga". can we interchange "no wo" with "no ga"? if not why is that? please give examples if required.. i will be waiting for your replies.

The の (no) is the bit which does the nominalization. It turns a sentence ending in a verb into a phrase which can be grammatically treated like a noun. Which particle follows (it could be ga, wo, or something else) depends on the context.

X wo miru means "see/watch X", but X ga miru means "X sees". So in your example, if we want the sentence to mean "I saw my son playing the piano", we need to use wo.

Just to add to nekojita's explanation.

If you want to say you saw something, "no wo" is the way to go, it nominalizes the expression making it an object of your action "miru/see"

"no ga" would nominalize the former expression, and add an action of "seeing" to it ( in your case, its wrong)