Its pretty cold in November; also, there are winter storms that can make the area treacherous to swim. You can see surfers from time to time, but its really not swimming weather.
While swimming is really not a possibility, there are other, different reasons to go to the area at that time. I would take a bus down to Tanabe or Shirahama (its pretty reasonable price wise), and visit some of the onsens... they are unbelievable.
Second, if you're really into seeing the outdoors, I'd suggest that you try to walk an extended portion of the Kumano kodo pilgrimage trails. The vistas from that hike are fantastic, and quite enjoyable. Its a two day trip for most people for the Tanabe-Hongu portion (I think most arrive at Chikatsuyu for the night), but it would be a really worthwhile experience. Its also a fairly inexpensive trip, compared to the rest of japan. I'll try to dig up some photos of the area for you.
Something to consider.