When someone or something is really frustrating and/or distressing you, it means that your tower is being, or as been, jumbled.
Similar to Rustle My Jimmies
That asshole at work really jumbles my tower
I'm getting so fucking jumbled right now
You're really jumbling my tower
My jimmies are jumbled