That's what a lot of men don't quite get these days.

That's what a lot of men don't quite get these days. New relationships or sex partners don't ever negate what you two had. While you're in your right to say no, I'd still ask why it bothers you so much.