Your understanding is correct. Incidentally, 明日は公園に来られませんか can be valid even if the speaker is not in the park now, but this implies that they will be in the park tomorrow. The destination of 来る is always the location where the speaker is now or will be at the time.