I agree with most of what you have both said, but I think that maybe it hadn't been a relationship YET. When the song says "it's time to forget the road we never traveled along," I feel it implies that he knew they couldn't be together, but that didn't stop him from falling in love, so when it came time to make a decision, he knew that there would be no future, so he left with what little heart he had left to save himself from more pain later.
I agree with most of what you have both said, but I think that maybe it hadn't been a relationship YET. When the song says "it's time to forget the road we never traveled along," I feel it implies that he knew they couldn't be together, but that didn't stop him from falling in love, so when it came time to make a decision, he knew that there would be no future, so he left with what little heart he had left to save himself from more pain later.