It looks simple, but I don't think there's equivalent word to "sorry" in English to use in such a situation.
"Sorry" can be translated as "zannen", which is to express disappointment, and it does not fit in well in this situation, or "kinodoku" which somehow sounds rather patronizing if not condescending in this situation.