Reducing bias in trials from reactions to measurement: the MERIT study including developmental work and expert workshop
Bibliographic record
Abstract
Background Measurement can affect the people being measured; for example, asking people to complete a questionnaire can result in changes in behaviour (the ‘question–behaviour effect’). The usual methods of conduct and analysis of randomised controlled trials implicitly assume that the taking of measurements has no effect on research participants. Changes in measured behaviour and other outcomes due to measurement reactivity may therefore introduce bias in otherwise well-conducted randomised controlled trials, yielding incorrect estimates of intervention effects, including underestimates. Objectives The main objectives were (1) to promote awareness of how and where taking measurements can lead to bias and (2) to provide recommendations on how best to avoid or minimise bias due to measurement reactivity in randomised controlled trials of interventions to improve health. Methods We conducted (1) a series of systematic and rapid reviews, (2) a Delphi study and (3) an expert workshop. A protocol paper was published [Miles LM, Elbourne D, Farmer A, Gulliford M, Locock L, McCambridge J, et al. Bias due to MEasurement Reactions In Trials to improve health (MERIT): protocol for research to develop MRC guidance. Trials 2018;19:653]. An updated systematic review examined whether or not measuring participants had an effect on participants’ health-related behaviours relative to no-measurement controls. Three new rapid systematic reviews were conducted to identify (1) existing guidance on measurement reactivity, (2) existing systematic reviews of studies that have quantified the effects of measurement on outcomes relating to behaviour and affective outcomes and (3) experimental studies that have investigated the effects of exposure to objective measurements of behaviour on health-related behaviour. The views of 40 experts defined the scope of the recommendations in two waves of data collection during the Delphi procedure. A workshop aimed to produce a set of recommendations that were formed in discussion in groups. Results Systematic reviews – we identified a total of 43 studies that compared interview or questionnaire measurement with no measurement and these had an overall small effect (standardised mean difference 0.06, 95% confidence interval 0.02 to 0.09; n = 104,096, I 2 = 54%). The three rapid systematic reviews identified no existing guidance on measurement reactivity, but we did identify five systematic reviews that quantified the effects of measurement on outcomes (all focused on the question–behaviour effect, with all standardised mean differences in the range of 0.09—0.28) and 16 studies that examined reactive effects of objective measurement of behaviour, with most evidence of reactivity of small effect and short duration. Delphi procedure – substantial agreement was reached on the scope of the present recommendations. Workshop – 14 recommendations and three main aims were produced. The aims were to identify whether or not bias is likely to be a problem for a trial, to decide whether or not to collect further quantitative or qualitative data to inform decisions about if bias is likely to be a problem, and to identify how to design trials to minimise the likelihood of this bias. Limitation The main limitation was the shortage of high-quality evidence regarding the extent of measurement reactivity, with some notable exceptions, and the circumstances that are likely to bring it about. Conclusion We hope that these recommendations will be used to develop new trials that are less likely to be at risk of bias. Future work The greatest need is to increase the number of high-quality primary studies regarding the extent of measurement reactivity. Study registration The first systematic review in this study is registered as PROSPERO CRD42018102511. Funding Funded by the Medical Research Council UK and the National Institute for Health Research as part of the Medical Research Council–National Institute for Health Research Methodology Research Programme.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.754 | 0.882 |
| Meta-epidemiology (narrow) | 0.004 | 0.005 |
| Meta-epidemiology (broad) | 0.009 | 0.019 |
| Bibliometrics | 0.013 | 0.009 |
| Science and technology studies | 0.005 | 0.007 |
| Scholarly communication | 0.012 | 0.018 |
| Open science | 0.008 | 0.021 |
| Research integrity | 0.016 | 0.011 |
| Insufficient payload (model declined to judge) | 0.013 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".