The basic rule is that に is used for "absolute" time such like 2021年, 6月, 16日, 水曜日, 8時, etc., while に is not used for "relative" time, which is decided by "now", such like 今日, 昨日, 明日, 今週, 先週, 来週, 今月, 先月, 来月, 今年, 去年 or 来年. 夜 belongs to the former ones, so に is attached to it.
However, there are many exceptions, and the particles also differ depending on the context, so it's a bit complicated, indeed.