rock in my shoe

The opposite of a feather in my cap. Something that is less than ideal; annoying; a hinderance

"Being the head of this project has proved to be a rock in my shoe."
"I always feel like a rock in my shoe at a party"
"This rock in my shoe has really been a rock in my shoe"