I think the downside is, a lot of people don't use their ISP accounts for this kind of stuff (or even at all, due to the fact many can't pick a good username and such, and people don't know how to use POP, or their web interface bites), which as mentioned, scares people away.
Is it possible to disable settings that sends email to those that have bounces from emails? I did that for my site once, but my user database is probably not as large as yours, and I had to do it manually. -_-