Commentary: Money and models: double-edged swords

Being richer rather than poorer is well known to be associated with a longer and healthier life. It is a small leap from this long-standing observation to the proposition that giving money to people could offer them this same advantage. But evaluation of this simple premise has been hampered by the non-random fashion in which wealth is typically distributed.1 Apart from lotteries, people rarely receive extra doses of wealth at random, and lotteries are problematic because those at risk of winning are the volunteer subset of the population who bought tickets, and therefore have uncertain generalizability.2 Moreover, if money is a treatment, then one also has to be specific about the timing and magnitude of the dosage. Whereas one 75-mg aspirin pill every day might reduce the risk of a heart attack, for example, 180 pills taken at once every 6 months would have a very different effect, despite being the same annual amount. Bruckner et al.3 are interested in exactly these two important questions: first, can one identify a causal effect of increased wealth on mortality through some fortuitous exogeneity, and secondly, could a large intermittent dose, such as a lottery win, actually have a harmful effect? The working hypothesis is that people might use a windfall to treat themselves to harmful products, such as intoxicants and fast cars, thereby elevating their risk of adverse events. The authors therefore turn their attention to accidental deaths, including those due to motor vehicles, poisoning and overdose. As a ‘natural experiment’ for the quasi-random assignment of income windfalls, they capitalize on the 1995 placement of a casino among a rural population of Cherokee Indians in Western North Carolina.3 The casino operator was obligated to make ongoing payments to the enrolled members of the tribe as a percentage of profits, and this amounted to something in excess of US $5000 per person each year. These payments raised the incomes of tribal members considerably, but they did so through one or two large instalments each year, and the authors speculate that this acute dosing of wealth supplements could prove to be detrimental. Bruckner et al.3 examine whether the months in which payments were received demonstrate excess deaths from accidents among the 57 000 residents of three counties in which the tribal members live. They compare the pattern of payments across the 204 months from January 1990 to December 2006 with the monthly incidence of accidental death in the Native American population up to the age of 55 years in these counties. Randomization is attractive as a means of identifying a causal effect because the potential outcome will be independent of the receipt of exposure, and therefore unconfounded in expectation. A so-called ‘natural experiment’ simply involves an independency that arises through some inadvertent mechanism, as opposed to intentional randomization designed and implemented by researchers.4 Unfortunately, the treatment schedule in this instance was not at all random. Annual payments commenced on 1 December 1995. Starting in 1998, this annual lump sum was divided into a two semi-annual sums, distributed every 1 December and 1 June. Therefore, Cherokee adults received cheques every December since the opening of the casino. After 3 years of annual sums, the payments were split and arrived the first day of every December and June. There were therefore 21 out of 204 months in which payments were received (12 Decembers and 9 Junes). With respect to the outcome measure, 1995 may have been a ‘random’ year in which to initiate the payment regimen: but, in every subsequent year, the monthly schedule was fixed with absolute certainty. The obvious problem here is that if excess accidental deaths occur in December and June for other reasons, the analytic design assures that these excesses will be attributed to the delivery of cheques from the casino. Unfortunately, the authors employ no control group (e.g. non-Cherokee in the same counties) with which to assess this concern or subtract out this underlying seasonality of events. Furthermore, some published data exist to suggest that for accidental deaths, these months are indeed known to be especially risky. For example, there is seasonality in blood alcohol levels among crash victims,5 with a noted December excess, and a similar finding among randomly screened college students whose drinking peaked at the ends of semesters (i.e. December and June).6 Another report showed that motor vehicle accident deaths in Canada peaked in summer, with lowest rates in March and April, whereas non-fatal crashes peaked in December and were also lowest in spring.7 Bruckner et al.3 were aware of this concern about seasonality, and for this reason they also specify a fixed-effects model to estimate the effect of payments conditioned on month and year. This model does not identify the causal effect of interest, however. Consider how we would interpret the effect estimate conditioned on month only. Fixing the month, we can find variation across years. Ten of the months are always untreated months, and therefore do not contribute to the effect estimate at all. But there is treatment variability across Decembers and across Junes, with 5 untreated versus 12 treated Decembers, and 8 untreated versus 9 treated Junes. There could still be confounding by year, if there were important secular trends, for example, but this could be checked against some control group (e.g. non-Cherokee) and differenced out if necessary.8 Regrettably, this is not what the authors do. Instead, they also condition on year. But holding constant both year and month with 11 + 16 dummy variables for the 12 months and 17 years invokes a positivity violation. There is no longer any treatment variability. If one specifies a month and year, the treatment status is determined with certainty: December 1991? Untreated. December 1997? Treated. And so on. The treatment indicator itself therefore carries no additional information. How then were the authors able to get the model to converge when the treatment indicator was a completely deterministic function of the year and month indicators? The answer is that treatment is not a linear function of these variables. The model therefore converges (although with debilitating imprecision) solely on the basis of model form, rather than on the basis of data (electronic Appendix available as Supplementary Data). To see this, imagine a non-parametric analysis of the same data. For example, try to estimate the effect by matching. Given a treated observation (e.g. June 1998), one must find an untreated observation that has the same month (June) and the same year (1998). Good luck with that. Indeed, it is this remarkable capacity of regression models to provide a spurious effect estimate despite a complete absence of overlap between treated and untreated units that motivated statisticians such as Rosenbaum and Rubin9 to militate for matched estimators in place of regression-based causal inference. The operating hypothesis here is that intermittent cash payments facilitate unhealthy behaviours, such as drunken driving. With respect to this possibility, there may be a much more convincing identification strategy than a regression model, and that is simply asking people how they spent the money. Impressively, these authors did exactly that, and as shown in Table 4 of the paper, the results are entirely supportive of their hypothesis. Males make up over three-quarters of the deaths, and over a third of the males interviewed acknowledged spending their windfall money on alcohol and/or drugs. One can only imagine that this self-reporting would, if anything, err on the low side. The interviewees were all 25 years of age or less, whereas two-thirds of the deaths occurred in older subjects. Nonetheless, the ethnographic data speak much more convincingly than the regression model. This is perhaps a good lesson about whom one should trust as a source of information about human behaviour: humans or regression models. Of course humans do not always tell the truth. But regression models? They don't even know the meaning of the word. This work was supported by funding from the Canada Research Chairs program.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.