Lyric discussion by andrewisok 

Cover art for Peep-Hole lyrics by Guided by Voices

The word albatross is sometimes used to mean an encumbrance, or a wearisome burden. A poem talks about a shipmate on a boat shooting an albatross that was following a ship. For his punishment his crew mates made him wear it around his neck.

I think it's just about being in love with a girl.

I think its one of my favorite GBV songs.