I think the last stanza makes sense if you think of it in relation to the southern bird simile. When winter comes the birds will leave their nests. When the narrator sees them leave, he sees that as his best opportunity to leave too. Because he feels out of place already, he's almost too embarrassed to leave and get more judgments from those around him. But, again, with everyone leaving their nests for the winter no one will be there to criticize him as he finally leaves.
I think the last stanza makes sense if you think of it in relation to the southern bird simile. When winter comes the birds will leave their nests. When the narrator sees them leave, he sees that as his best opportunity to leave too. Because he feels out of place already, he's almost too embarrassed to leave and get more judgments from those around him. But, again, with everyone leaving their nests for the winter no one will be there to criticize him as he finally leaves.