[Commentary] A HARD NATURAL EXPERIMENT TO FATHOM: A TESTAMENT TO THE INCREASING DIFFICULTY OF CONDUCTING RELIABLE SURVEYS?

The paper by Mäkeläet al. [1] adds to the important tradition of studying ‘natural experiments’ in alcohol policy, a tradition that is especially strong in the Scandinavian countries [2]. Had the results followed the almost invariable pattern that reduced taxes lead to increased consumption, there would have been much less over which to discuss and puzzle. The authors are to be applauded for publishing these unexpectedly negative results. Arguably, studies such as this have unique value by forcing the field to question received wisdom and, I suggest, also traditional research methods. In discussing the results it is important to bear in mind the backdrop of a multitude of studies using many different designs, nearly all of which have indicated that alcohol behaves like other commodities, in so far as its consumption is responsive to price changes and hence manipulation of price is of major public health importance [3-6]. The main objective of this particular study was to study differential responsiveness to price changes among different subpopulations to confirm whether or not age, gender, income and the level of alcohol consumption influence substantially the degree of this responsiveness. However, the survey-based measures of drinking mainly failed to replicate the changes that were both expected from a reduction in taxation in Finland and Denmark (i.e. an increase in volume of consumption) and which were also observed in national sales data. The latter showed that a substantial reduction in Danish spirit taxes was associated with a 16% increase in spirits consumption (but 2% reduction in overall alcohol consumption as beer and wine sales decreased) and also that across-the-board reductions in Finnish liquor taxes resulted in a 10% increase in overall consumption. The panel design surveys used in Makela et al.'s study [1], however, found significant decreases in overall alcohol consumption in these two countries following the tax reductions, while the repeated cross-sectional surveys found no significant overall changes (although they had less power to detect these). A tempting conclusion at this point is that the panel survey design was not sufficiently sensitive to the population-level changes in consumption suggested by sales data, and hence questions of differential responsiveness by population subgroups could not be addressed. There are two broad areas of complexity which may have contributed to this discordance between survey and sales data. First, the natural experiment itself is not straightforward: (i) these are small neighbouring countries with a strong tradition of cross-border trade in alcohol (e.g. [7]); (ii) in Denmark only spirits taxes were reduced and this took place only three-quarters of the way through the first comparison year; (iii) changes to the cross-border allowances between Finland and Estonia were made in the middle of one of the second comparison years, 2004; (iv) there was a substantial relaxation of travellers' allowances introduced at the beginning of 2004 in Finland and the control country, Sweden. Secondly, there are a number of reasons for doubting the reliability and validity of the surveys conducted and the comparisons between them made to evaluate these changes: (i) as is increasingly common, response rates to the surveys, especially telephone surveys, were low (typically around 50%); (ii) research has shown repeatedly that data based on surveys drastically underestimate true consumption in populations, partly by under-sampling some of the heavier drinkers (e.g. [8]); (iii) the authors note the problem of regression to the mean in studies of changes in drinking patterns using a panel design, and while they attempted to adjust for these there may still have been differential effects operating across different age, gender and alcohol consumption groups; (iv) the authors note that differential dropout of subjects may occur in different subgroups of interest in panel studies which may be harder to assess precisely; (v) the results obtained from the panel and the cross-sectional surveys were mainly inconsistent with each other in relation to the size and direction of changes in consumption between 2003 and 2004, suggesting the presence of random errors and/or biases differentially effecting the two survey methods. In summary, I suggest that the sizes of actual changes in consumption that occurred, following what appeared to be dramatic changes in tax levels, may not have been substantial enough to enable fine-grained analyses of differential effects on populations subgroups, given the multiple difficulties of accurate measurement over time using population surveys. Future studies of natural experiments in tax policy changes may need to focus upon larger changes in consumption, perhaps over longer time-periods, with ‘cleaner’ comparisons possible between intervention and control countries, using cross-sectional surveys with larger sample sizes and sampling strategies which result in much higher response rates than 50%. Admittedly, this is setting the bar very high and at a standard that is increasingly hard to achieve in most countries, especially with telephone surveys (e.g. [9]). The use of measures of population rates of alcohol-related harm from morbidity and/or mortality data is also recommended (e.g. [10]). Overall, these mixed results underscore the value of employing multiple data sources to interpret changes in the drinking behaviour and the need for caution when interpreting observed changes over time assessed by population surveys alone. I am grateful to Scott Macdonald for helpful comments on a first draft of this commentary.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.