Out of pocket

Something that was uncalled for, rude or an inconvenience.

Bailey: your shirt is super unattractive, but it looks good on me.
Mary: That was out of pocket so shut up.