It's more likely modifying clause (corresponding to relative clause in English grammar) even just for a single word. Thus, the construction is closer to "language that is written" than "written language", therefore it can have tense, aspect or mood.
e.g.
書かれた言語
language that is written
書きたくない言語
language that I don't want to write
書けなかった言語
language that I was not able to write
いる is used there because the -te iru form expresses an aspect of the verb (the present state resulting from the past action in this case), not because to connect ほれて to a noun. First, the form is chosen to express the meaning, and then it's used to modify the following noun.