Phillips learned violin and piano as a child, and played reeds in his teens as a member of his brother's European band. He got into the music business as a publisher and director for the Edison-Bell Gramophone Company. In 1930 he began writing arrangements for Bert Ambrose, and joined Ambrose's ensemble in 1933, remaining there until 1937. Late in the 1930s Phillips played in the United States on radio and freelance in clubs. He served in the Royal Air Force during World War II, then put together his own quartet in 1946 and wrote several pieces for the BBC Symphony Orchestra. He led a Dixieland jazz band of his own formation from 1949, and his sidemen included George Shearing, Colin Bailey, Tommy Whittle, and Kenny Ball.
Phillips's first recordings under his own name were made in 1928, and he continued to record as a leader into the 1970s.