Lyric discussion by RedNeckChris 

At this point, the main character Pink(AKA:Roger Waters)has succeeded in building a metaphorical wall to keep everyone out. The worms represent bad thoughts and bad ideas that keep creeping into his head. He no longer has any outside help keeping the bad thoughts at bay due to his self inflicted isolation behind the wall.

The image of the Nazi is just a symbol of the evil thoughts that we all entertain from time to time. Usually we're kept from acting on those thoughts by our own innate sense of right and wrong and, more importantly, by the influence of those we have chosen to let into our world. Pink has no one to act as that good influence because he is alone behind the wall.

He's starting to question the wisdom of erecting such an impenetrable barrier in the first place, and he's wondering if it is in fact too late to change his mind.

I would say most of your post is spot on. Only thing I would like you to consider is the role of the "worms". In a "normal" persons life there are positive role models (people we aspire to be like or at least come close to being like) and people who are negative role models ( people we despise and do not want to be like).

An error occured.