Why Won't They Talk to Me? Lyrics

Lyric discussion by thujone 

Cover art for Why Won't They Talk to Me? lyrics by Tame Impala

Isn't he singing "why won't YOU talk to me?" sometimes. In the last refrain it really seems like he's singing this.

If true, is it a breakup song? Did she leave him because he's not in the clique anymore? Or does "they" represent all the girls who've left him?