Advice on this situation?

Ignore the people you lost over this, if they couldn't see your side of being pestered by some dirty old man then they aren't your friends and you're better off without them.