I don't think 別に is necessarily always rude alone and is actually no problem between good friends.
It also isn't necessarily uncommon to use 別に in everyday speech or a term reserved for formal conversation or anything like that. The closest difference in English is making it "aside from" or "apart from" instead of "besides" "other than" or "except for."
I hear "hoka" or "igai" for these more often but I'm not really sure if this is what you're asking....
If you had a more concrete example that would help a lot.