Vex

make (someone) feel annoyed, frustrated, or worried, especially with trivial matters.
"the memory of the conversation still vexed him"

It seems that the children's antics will continue to vex the nanny. Don't vex your mother; she's in a foul mood. I didn't think that such an inconsequential matter would vex you so.