Lyric discussion by greenzob 

Cover art for Sultan lyrics by What Made Milwaukee Famous

to me, this reminds me of regretful sentiments post-breakup. you need to get rid of your past to forget the relationship. you need to get rid of the memories or else "they're never going to leave you alone." probably a messy breakup in my opinion, since he mentions not wanting the truth to be told, and also talks about "are you in or are you out" which could allude to an on-off sort of relationship.