In black culture, dissimilar to "say mothers", suggesting that one's father has abandoned them.
Person 1: You're a nigger Person 2: Say fathers Person 1: Ah you got me, that does make me more black.