Lyric discussion by Prisoner.I.Was 

Cover art for Guilt lyrics by Long Blondes, The

I don't think adamz is right, Paddy_Kitten has sort of got it. Basically, a man is interested in the song's speaker, but she's worried about getting involved since she's already in a relationship. However, she does get involved, then does her best to avoid him afterwards. The fact that she sings 'Guilt has nothing to do with it' is ironic, since it really does.

This is a perfect pop song, I absolutely love it.