Willis was born in 1932 and raised solely by his mother. He relocated to Detroit in 1953. He had no real interest in music, he said, "But then I saw Sonny Boy Williamson II." Willis was "spellbound at the way he played. After the show I went home and practiced for hours. Every day after that I would practice until I got the sound I wanted." His daytime job was working in a used car lot.
His first professional appearance was at the Good Times Bar in Detroit, playing in Washboard Willie's backing group. He put together his first band in March 1956. For the following fifteen years he performed in numerous Detroit clubs, often boosting his earnings by photographing customers between his performances on stage.
He often performed with John Lee Hooker, Eddie Kirkland and Baby Boy Warren. Another club stalwart, Eddie "Guitar" Burns, accompanied him on Little Sonny's debut single, "I Gotta Find My Baby" (1958), released by Duke Records. It was co-written by Little Sonny's wife, Maggie. His follow-up release, "Love Shock", was for Excello Records. He received $25 for that track. He then established his own label, Speedway Records, and sold enough copies of his next single, "The Mix Up", to pay his production costs.