Its an A is B type sentence.
koto makes hanasu(hanasimasu) into a noun so it can be used in this way, a verb can't 'be' anything.
So A is 'anata to hanasu koto' Speaking with you.
And B is 'tanosii', fun.
So Speaking with you wa fun desu,
Speaking with you = fun.
Speaking with you is fun.