I have the benefit of having read Lightfoot's biography so I know this song refers to a very brief affair he had in NYC, while married. I think he felt guilty about it so in the song had the woman ask the "the riddle" which in his mind justified his affair, i.e., who do the moments of our lives belong to but ourselves, what's wrong, what's right, should this liason be a secret I keep hidden? I don't believe this song made his performing setlist much in his later career, as he had changed and didn't want to be seen as a philanderer. Nonetheless, the song is hauntingly beautiful.
I have the benefit of having read Lightfoot's biography so I know this song refers to a very brief affair he had in NYC, while married. I think he felt guilty about it so in the song had the woman ask the "the riddle" which in his mind justified his affair, i.e., who do the moments of our lives belong to but ourselves, what's wrong, what's right, should this liason be a secret I keep hidden? I don't believe this song made his performing setlist much in his later career, as he had changed and didn't want to be seen as a philanderer. Nonetheless, the song is hauntingly beautiful.