I think it means, to me, that the relationship is over, and the ex is being overly hateful about it and won't let it go. He is bothered by the overall hatefulness of her, and he wants it to be over. Their relationship is over, and I think that he's wanting to move on; people are in their business, all the time... It's time to move on, and his ex keeps bringing up his past. He never came to bother her, and he never came to ask her opinion anymore; they are done, so he has left, so there is no leaving to be done. Why can't she just leave him alone?
I think it means, to me, that the relationship is over, and the ex is being overly hateful about it and won't let it go. He is bothered by the overall hatefulness of her, and he wants it to be over. Their relationship is over, and I think that he's wanting to move on; people are in their business, all the time... It's time to move on, and his ex keeps bringing up his past. He never came to bother her, and he never came to ask her opinion anymore; they are done, so he has left, so there is no leaving to be done. Why can't she just leave him alone?
I personally relate to this.