Development and validation of critical appraisal tool for individual participant data meta-analysis: protocol for a modified e-Delphi study
Bibliographic record
Abstract
INTRODUCTION: Individual participant data meta-analysis (IPD-MA) is regarded as the gold standard for evidence synthesis. However, diverse recommendations and guidance on its conduct exist, and there is no consensus-based tool for the critical appraisal of a completed IPD-MA. We aim to close this gap by systematically identifying quality items and developing and validating a critical appraisal checklist for IPD-MA. METHODS AND ANALYSIS: This study will comprise three phases, as follows:Phase 1: a systematic methodology review to identify potential checklist domains and items; this will be conducted according to the Cochrane methods for systematic reviews and reported following the Preferred Reporting Items for Systematic Reviews and Meta-analysis 2020 guidance. We will include studies that address methodological guides and essential statistical requirements for IPD-MA. We will use the proposed items to prepare a preliminary checklist for the e-Delphi study.Phase 2: at least two rounds of an e-Delphi survey will be conducted among panels with expertise in IPD-MA research, consensus development, healthcare providers, journal editors, healthcare policymakers, patients and public partners from diverse geographic locations with experience in IPD-MA. Participants will use Qualtrics software to rate items on a 5-point Likert scale. The Wilcoxon matched signed rank test will estimate response stability across rounds. Consensus on including an item will be achieved if ≥75% of the panel rates the item as 'strongly agree' or 'agree' and items will be excluded if ≥75% rates it as 'strongly disagree' or 'disagree'. A convenience sample of 10 reviewers with experience in conducting an IPD-MA will pilot-test the checklist to provide practical feedback that will be used to refine the checklist.Phase 3: critical appraisal checklist validation: to improve confidence in the tool's uptake, a subset of the e-Delphi participants and graduate students of epidemiology and biostatistics will conduct content validity and reliability testing, respectively, per the Consensus-based Standards for the Selection of Health Measurement Instruments. ETHICS AND DISSEMINATION: Ethics approval has been obtained from the Western University Health Science Research Ethics Board in Canada. The validated checklist will be published in a peer-reviewed open-access journal and shared across the networks of this study's steering committee, Cochrane IPD-MA group and the institutions' social media platforms.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.305 | 0.409 |
| Meta-epidemiology (narrow) | 0.005 | 0.005 |
| Meta-epidemiology (broad) | 0.007 | 0.011 |
| Bibliometrics | 0.010 | 0.009 |
| Science and technology studies | 0.004 | 0.006 |
| Scholarly communication | 0.006 | 0.007 |
| Open science | 0.005 | 0.006 |
| Research integrity | 0.006 | 0.009 |
| Insufficient payload (model declined to judge) | 0.075 | 0.017 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".