I think the song is telling men don't be afraid to be fashionable and handsome. Especially since this was in the 80s where fashion was highly androgynous, especially on the music scene following both the punk and funk scenes (Siouxsie, Prince, Madonna, etc.). I think the song is saying don't be afraid to present yourself as the 'freak' that you are, don't be afraid to wear what you want when you want, and people will say this and that but show off, don't lower your standards, be beautiful.
I think the song is telling men don't be afraid to be fashionable and handsome. Especially since this was in the 80s where fashion was highly androgynous, especially on the music scene following both the punk and funk scenes (Siouxsie, Prince, Madonna, etc.). I think the song is saying don't be afraid to present yourself as the 'freak' that you are, don't be afraid to wear what you want when you want, and people will say this and that but show off, don't lower your standards, be beautiful.