imma let you go

I allow you to think that I don't want to hold you up, while in reality I'm the one that's ready to get off the phone.

Phone conversation...
A: It's been a while since I was in that area. Girl let me tell you...You got a minute?
B: Naw, Imma let you go!
A: I'm fine! I can take a break.
B: Naw, just call me later, we can talk then! (Won't answer the phone later...)