Much like the popular phrase "that's what she said", "that's what I told my first wife" can apply to male only references. The first known use was by Michael Thor Lengies Sept. 2012.
Perfect set ups:
"It's doesn't get any harder than this." "That's what I told my first wife."
Sarcastic:
"I always wear condoms." "That's what I told my first wife."
Out of no where:
"I've got a date at 9:00." "That's what I told my first wife."
Self degrading humor:
"I'm just not a one woman man." "Yeah, that's what I told my first wife."
Thats what he said, Thats what she said, Inuendo, Sexual,