McCartney is using the term The Man as was used commonly in the 60's/70's, as a collective term for authority, which acts and expresses itself sometimes in your best interest and sometimes oppressively ( it's also used in the military for superior officers ), so regardless of what the authorities have to say about a certain thing, even if it's about love, eventually "people will find a way to go" anyway.
McCartney is using the term The Man as was used commonly in the 60's/70's, as a collective term for authority, which acts and expresses itself sometimes in your best interest and sometimes oppressively ( it's also used in the military for superior officers ), so regardless of what the authorities have to say about a certain thing, even if it's about love, eventually "people will find a way to go" anyway.