John Anthony Gillis was born in Detroit, Michigan, the youngest of ten children—and the seventh son—of Teresa (née Bandyk) and Gorman M. Gillis. His mother's family was Polish, while his father was of Scottish-Canadian descent. He was raised a Catholic, and his father and mother both worked for the Archdiocese of Detroit (as the Building Maintenance Superintendent and secretary in the Cardinal's office, respectively). Gillis became an altar boy, which landed him an uncredited role in the 1987 movie The Rosary Murders, filmed mainly at Holy Redeemer parish in southwest Detroit.
Gillis' early musical influences were inherited from his older brothers, and he learned to play the instruments they abandoned; he began playing the drums in the first grade after finding a kit in the attic. As a child, he was a fan of classical music, but in elementary school, he began listening to the Doors, Pink Floyd, and Led Zeppelin. As a "shorthaired [teenager] with braces," Gillis began listening to the blues and 1960s rock that would influence him in The White Stripes, with Son House and Blind Willie McTell being among his favorite blues guitarists. He has said in interviews that Son House's "Grinnin' In Your Face" is his favorite song of all time. As a drummer, his heroes include Gene Krupa, Stewart Copeland, and Crow Smith from Flat Duo Jets.
In 2005 on 60 Minutes, he told Mike Wallace that his life could have turned out differently. "I'd got accepted to a seminary in Wisconsin, and I was gonna become a priest, but at the last second I thought, 'I'll just go to public school.' I had just gotten a new amplifier in my bedroom, and I didn't think I was allowed to take it with me." Instead, he got accepted into the famed Cass Technical High School as a business major, and played the drums and trombone in the band. At 15, he began a three-year upholstery apprenticeship with a family friend, Brian Muldoon. He credits Muldoon with exposing him to punk music as they worked together in the shop. Muldoon goaded his young apprentice into forming a band: "He played drums", Gillis thought. "Well I guess I'll play guitar then." The two recorded an album, Makers of High Grade Suites, as the Upholsterers.
As a senior in high school, he met Megan White at the Memphis Smoke restaurant where she worked, and they frequented the coffee shops, local music venues, and record stores of the area. After a courtship, they got married on September 21, 1996. In a reversal of tradition, he took her last name.
After completing his apprenticeship, he started a one-man business of his own, Third Man Upholstery. The slogan of his business was "Your Furniture's Not Dead" and the color scheme was yellow and black—including a yellow van, a yellow-and-black uniform, and a yellow clipboard. Although Third Man Upholstery never lacked business, he claims it was unprofitable due to his complacency about money and his business practices that were perceived as unprofessional, including making bills out in crayon and writing poetry inside the furniture.