going on my nerves

When someone is making you feel bad because of his/her annoying behavior or talk.

Girl: Boy, stop flirting with me, you are going on my nerves.
Boy: Girl, stop ignoring me, you are going on my nerves.
Girl: I have not done anything, you did.