It's when a person fails to do something or says something that just doesn't work. When the result of something is totally unsatisfactory and/or potentially embarrassing.
It's like when a cake tastes like a dry piece of cornbread with super sugary icing on top: basically unpalatable.
Person 1: Did you see that guy trip over his pant leg just now?
Person 2: Yeah, that was a total fail cake. Sucks to be him.