Conflation of the expressions drive me up a wall and driving me nuts. Basically when someone is annoying you so intensely it's almost like they're kicking you in the balls. Hence the reference to the word "nuts."
Co-worker: How's the project with Amy coming along?
Me: Holy moly ... she's really driving me up my nuts. I can't stand her constantly changing her mind about everything.