Things I Have Learned about Meta-Analysis Since 1990: Reducing Bias in Search of “The Big Picture” / Ce que j’ai appris sur la méta-analyse depuis 1990 : réduire les partis pris en quête d’une vue d’ensemble
Bibliographic record
Abstract
This paper examines sources of potential bias in systematic reviews and meta-analyses which can distort their findings, leading to problems with interpretation and application by practitioners and policymakers. It follows from an article that was published in the Canadian Journal of Communication in 1990, “Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis,” which introduced meta-analysis as a means for estimating population parameters and summarizing quantitative research around instructional research questions. This paper begins by examining two cases where multiple meta-analyses disagree. It then goes on to describe substantive and methodological aspects of meta-analysis where various kinds of bias can influence the outcomes and suggests measures that can be taken to avoid them. The intention is to improve the reliability and accuracy of reviews so that practitioners can trust the results and use them more effectively. Cet article examine les sources des partis pris potentiels dans les synthèses systématiques et les méta-analyses qui peuvent déformer les conclusions, ce qui peut causer des problèmes d’interprétation et d’application par les praticiens et les responsables des politiques. Il fait suite à un article publié dans le Canadian Journal of Communication en 1990, intitulé « Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis », qui présentait la méta-analyse comme moyen d’estimer les paramètres relatifs à la population et de résumer la recherche quantitative sur les questions de recherche pédagogique. L’article commence avec l’examen de deux cas dans lesquels de nombreuses méta-analyses sont en désaccord. Il décrit ensuite les aspects substantifs et méthodologiques des méta-analyses dans lesquels divers types de partis pris peuvent influencer les résultats et suggère des mesures qui peuvent être adoptées pour éviter ces partis pris. L’intention est d’améliorer la fiabilité et l’exactitude des synthèses afin que les praticiens puissent compter sur les résultats et les utiliser plus efficacement.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.582 | 0.801 |
| Meta-epidemiology (narrow) | 0.003 | 0.004 |
| Meta-epidemiology (broad) | 0.014 | 0.014 |
| Bibliometrics | 0.021 | 0.030 |
| Science and technology studies | 0.005 | 0.019 |
| Scholarly communication | 0.022 | 0.028 |
| Open science | 0.007 | 0.011 |
| Research integrity | 0.015 | 0.024 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".