Bad stats: A regular series exploring slip-ups, snafus and salutary lessons from the world of statistics

“New research among 4.4 million Danes shows that sunbathers on average live six years longer. The Danish Cancer Society finds the figures exciting.” This surprising claim in a 2013 story in the Danish newspaper Politiken (tinyurl.com/4wnvrjzt) is an example of a type of statistical blunder that was first pointed out almost 200 years ago, but that refuses to die. It is kept alive by investigators with easy access to large databases, publicity offices of universities and journals, and eager journalists. It is an artefact that statisticians now call “immortal time bias”. The basic idea reflects a fact of life: most “stations” or “honours” take some time to reach or achieve. Some political offices have minimum age requirements; and one must have reached 80 to join the octogenarian club. On joining, Queen Elizabeth quoted Groucho Marx, who had already defined the core issue: “getting older is no problem: you just have to live long enough.” By definition, all octogenarians were “immortal” until age 80, but some who had feted them at their earlier birthdays died before they could join. This seems pretty obvious, but it lays a trap for researchers investigating the flip side: factors that influence longevity.1 For example, one cannot credit the 41-year difference in longevity between George Burns and Richard Burton to the fact that Burns, who lived to 100, had won an Oscar (at age 80), whereas Burton, who died at 59, had not. Over the years, failure to recognise immortal time bias has led some observers to identify various exclusive “clubs” whose members enjoy supposedly greater longevity: popes blessed with longer lives than mere priests; eminent orchestra conductors and musicians whose life expectances exceed national averages; Oscar winners who outlive performers who do not win; politicians whose careers peak later in life, and (thus) die at older ages than those who peak earlier. All very amusing. But in 2013 the same blunder was behind a study that identified a surprising new – and much larger – club: sunbathers. This was no laughing matter, as the claim threatened to have serious public health implications. Rather than find this “exciting”, as Politiken claimed, the Danish Cancer Society was worried. The message undermined the Society’s decades of public health work promoting a careful approach to sun exposure. Naturally, the sun-deprived Danish public warmly welcomed this epidemiological “news”. The public regarded the Society and the sceptical statisticians, who claimed the benefits of joining this “club” were a statistical artefact, as spoilsports. After all, the research article had appeared in the prestigious International Journal of Epidemiology (IJE).2 Nevertheless, the popular Danish fact-checking show Detektor devoted a programme to the claims and counter-claims. The authors of the research dismissed the arguments of biostatisticians Theis Lange and Niels Keiding of the University of Copenhagen. When asked why they had not included a trained statistician in their research group, the authors replied that “if one gave the same data set to ten different statisticians, one would get ten different conclusions”. Moreover, they insisted, “the numbers as such do not lie.” Determined to raise awareness of the immortal time bias issue in both the public’s perception and in the journal, Lange and Keiding wrote a letter to the editors of the IJE.3 The original authors responded with new analyses that toned down and indeed seemed to refute their claim.4 Coincidentally, the journal was just about to publish a tutorial that addressed these types of blunders and how to avoid them, and so the editors invited its authors to add a commentary on this specific case. The letter, the response, and the expanded tutorial1 were accompanied by an editorial5 acknowledging that the journal had indeed “fallen prey to” a common blunder. The editors, Jane Ferrie and Shaw Ebrahim, illustrated the problem of “immortal time bias” by using an example first pointed out in 1840: “Generals and bishops live longer than corporals and curates – but this is not necessarily because an elevated occupational status makes you live longer – it may simply be because you have to reach a certain age before it is possible to hold such positions. People become generals and bishops in middle age so their deaths arise after this point in time, whereas corporals and curates can die at any age above 20 or so.” So how had immortal time bias created the illusion that sunbathing leads to a longer life? It had worked its way in as a result of how the authors created their comparison groups. Using a large Danish health register, they identified keen sunbathers as those who at some point developed skin cancer (specifically, the less serious non-melanoma type). They then compared this group’s risk of dying from any cause over the age of 40 with those without such a diagnosis. The problem is that those with skin cancer had to have lived long enough to get their diagnosis – instantly creating a longevity bias. It created the (false) impression that those who love sunbathing live substantially longer. In statistical terms, the astoundingly low mortality rate ratio for the avid sunbathers of 0.52 amounted to a longevity difference of 6 years. The 6 years was never mentioned in the scientific article, but it was this derived statistic that caught (or was brought to) the attention of the Politiken journalist. Blunders involving immortal time continue to occur and to do damage. In addition to repeating the warnings,1 what general lessons might we draw concerning “findings” from non-experimental data? Authors cannot rely on journalists to read the entire article, or to appreciate the limitations and nuances that are typically addressed at the end. So abstracts or summaries of the articles should not focus on headline-grabbing “crude” – and often misleading – statistics. Figures should also avoid uncorrected comparisons, and the corresponding p-values. And, before authors publish, they can use simulated data where an “effect” is set to be null1 to identify data-analysis methods that might have created a spurious effect. How the immortal time bias created the illusion of sunbathers gaining 6 more years of life. This standard Kaplan-Meier survival plot (modeled after Figure 1 of the IJE article2, but using simulated data1) shows the percentage dead by various indicated ages : note the large difference in median longevity and the smoothness resulting from the Big Data. The ‘off the scale’ P value (<2×10−308) was one of the smallest ever recorded1,3. The large difference is because most non-melanoma skin cancers are diagnosed when people are in their 60s 70s and 80s (tinyurl.com/y4x7c7ca), by which ages considerable fractions of their peers have already died. One would obtain a similar artefact if one compared the longevity of those who lived to see their great-grandchildren with those who only lived to see their grandchildren. Reviewers and editors can and should damp down the overenthusiasm of authors, and look for other explanations, especially when results seem “too good to be true”. How plausible would it be if, in a seemingly well-run and very large randomised trial, the all-cause mortality rate was cut almost in half (ratio 0.52) just by manipulating sun exposure – or giving a flu vaccine?6 When the claims that are made, or are likely to be made, have large societal implications, one might argue that peer-reviewers have an obligation to be more proactive. Looking back on it recently, Lange recalled how “professional and honest the journalists had been in reporting the objections of the statisticians”. In their response4 to the letter from the statisticians, the authors had also welcomed the statisticians’ comments to the media as a “second (post-publication) round of revision”. Moreover, at the statisticians’ insistence, and under the headline “Scientists admit mistakes – you do not live longer because you sunbathe”, the Danish newspaper Politiken which had run the original story printed a short “correction”. It noted that “the researchers recognised that some of their analyses could not be used to conclude that sunbathing could have a life-extending effect” (tinyurl.com/2nw2ner7). Happily, unlike some of the manifestations of immortal time bias that underlie other off-the-scale claims of longevity benefits (see Julianne Moore’s 2015 Oscars speech: tinyurl.com/4a4u5uv2), “sunbathers live much longer” has not become an urban myth. Overall, Lange is pleased by the way it played out: “Yes, immortal time bias mistakes happen, but when all act professionally the damage can be fixed.” The author wishes to thank Theis Lange and Robert Matthews.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.