To me, it sounds self explanatory. He was in a relationship with someone he loved, they fought often ("but every day I found new ways to hurt you and every day we took our sides") and finally ended their relationship. He's conflicted because he loves this person but he knows the longing he feels now is because the one he loved left and he doesn't see it as fair ("and it's just my soul responding to the love, to the love you took from me"). It wasn't that the love simply left, it was specifically the love this person took from him.
To me, it sounds self explanatory. He was in a relationship with someone he loved, they fought often ("but every day I found new ways to hurt you and every day we took our sides") and finally ended their relationship. He's conflicted because he loves this person but he knows the longing he feels now is because the one he loved left and he doesn't see it as fair ("and it's just my soul responding to the love, to the love you took from me"). It wasn't that the love simply left, it was specifically the love this person took from him.