Outlier detection methods for meta-analyses of site-specific effect estimates from a multi-site network
Bibliographic record
Abstract
Introduction: Data privacy legislation in Canada prohibits patient-level administrative health data from crossing jurisdictional boundaries. Accordingly, multi-site research networks often conduct distributed analyses and pool site-specific effect estimates (EEs) using meta-analysis models. Rare outcomes and heterogeneity in site-specific EEs can produce potential outliers that may bias pooled EEs. Limited research has compared outlier detection methods and the impact of potential outliers on meta-analysis results. Purpose and Objectives: The research purpose was to examine outlier detection methods for meta-analyses of site-specific EEs from a multi-site network. The objectives were to: 1) compare outlier detection methods for random-effects meta-analysis (REM) models, and 2) apply these methods to site-specific EEs from systematically selected real-world meta-analyses. Methods: We compared studentized residual estimates (StdR), relative change in pooled EE variance (RCPEV), relative change in estimated between-site variance (RCEBV), and model-based mean-shift method (MMS) using computer simulation. EEs were simulated assuming a normal distribution. Accuracy, misclassification error (ME), and F-1 score were assessed using random-effects analysis of variance models. We systematically selected meta-analyses conducted by investigators from the Canadian Network for Observational Drug Effect Studies (CNODES), applied outlier detection methods, and assessed the impact of potential outliers on REM results. Results: StdR had the highest accuracy (median: 89.9%) and lowest ME (median: 10.2%). RCPEV was the most consistent in all metrics. For StdR, the number of sites explained 95.1% and 93.0% of the variation in accuracy and ME values. For RCEBV and MMS, between-site variance described the most variation in accuracy and ME values. StdR and RCPEV were most sensitive to detect potential outliers in re-analyses of 39 published CNODES meta-analyses. Heterogeneity in site-specific EEs was reduced to zero in two-thirds of the meta-analyses when potential outliers were removed, and the precision of pooled EEs increased. Conclusions: StdR and RCPEV outperformed RCEBV and MMS in outlier detection. The number of sites and between-site variance explained the most variation in performance metrics for all methods. Excluding potential outliers from published meta-analyses, substantially reduced heterogeneity in site-specific EEs and increased the precision of pooled EEs.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".