red flag

Describes a negative quality about a person in, usually in the context of dating, that makes a person less likely to be worthy of pursuing further

He hangs out with his ex on the weekends. That's definitely a red flag.