Lyric discussion by hellojello 

Cover art for Sweater Weather lyrics by Neighbourhood, The

I personally don't think this song is about infidelity. This is because he says

"One love, two mouths One love, one house No shirts, no blouse Just us, you find out"

This chorus I think is specifically talking about him and the woman he desires. Also he says in another verse,

"These hearts adore, everyone the other beat heart is for"

I think he's saying their hearts beat for each other.

To me this song is about him offering all that he has to this woman, even if it's not much, "Both your hands in the holes of my sweater"

My Interpretation