It is understandable. It might be more complete (correct) to say, "I don't want to play against you", but I don't think anyone would consider "play you" as a mistake. At least not in the spoken context you posted above.
I like "play against" also. Just "play you" has another negative meaning sometimes. If I "play you" it can mean I trick , use , or manipulate you somehow for my personal gain.( kind of a criminal meaning"