Lyric discussion by treant 

Cover art for Sweet Talk, Sweet Talk lyrics by New Pornographers, The

I tend to interpret songs as chronicling personal relationships, and this one is no exception.

First stanza: "A mistake on the part of nature" could refer to the narrator's love interest, who is "so fabled, so fair" they don't seem to fit into everyday reality. I think "feeling Byzantine" could mean that the experience of the person visiting ("Just sit anywhere") feels like something from another time and place.

Second stanza: "Mistakes on the part of nature / The living proof of / What they're calling love / On certain sideways streets / Where things that don't match meet" - These lines may describe the unpredictable nature of attraction. Love seems to happen just by accident, as people that don't seem to match meet and are drawn together.

Third stanza: The other person seems like "a tall glass" (as in the expression "a tall drink of water"). They are "A blast from the past", which the narrator recalls fondly as being a simpler time -- although their partner is skeptical.

Chorus, part 1: "A mistake on the part of nature / It's forgiven, moved on" - Maybe the relationship didn't work out, so the narrator wants to write it off as a mistake and move forward with his or her life. However, he/she is avoiding the particular place he/she originally met the love interest in, because that would remind him/her too much of the past.

Chorus, part 2: not sure

Fifth stanza: "Amnesia becomes ambition / Ambition becomes a new sort of / Charming simplicity / Like always, Byzantine" - These lines seem to be musing on the cyclic nature of history, perhaps as applied to the narrator's personal experiences. As people forget past failures, they gain new ambitions for the future which may lead to the same problems (or maybe not).

My Interpretation