Southern American English phrase used when someone says something that is crass or offensive, but you aren't mad at them because it was also somewhat true or funny. You yourself didn't say it because you're just not quite fucked up enough to do so, but the other person is. Instead of telling the other person "How dare you!" or "That was a terrible thing to say!" you can just blithely remark, "You ain't right."
Jim: "Maybe this country wouldn't be so fucking fat if people went outside more often and ate less McDonald's. A few more ladies could at least start eating less."
Bob: "You ain't right, Jim."