Lyric discussion by walterbyrd 

Cover art for Refugee lyrics by Tom Petty & the Heartbreakers

A refugee is somebody who does not have choices. A refugees has to go wherever he/she is accepted.

Tom is saying: you don't have to live like that. You have choices, you don't have to settle for a crappy relationship.

We did somethin' we both know it We don't talk too much about it Ain't no real big secret all the same Somehow we get around it Listen it don't really matter to me baby You believe what you want to believe You see you don't have to live like a refugee

Maybe the song is about two people, both in bad relationships, who cheat with each other?

Maybe there is some amount of guilt, and Tom is saying "screw it, we don't have to stay in bad relationships."

Song Meaning