Bibliographic record
Abstract
Background: Systematic reviews (SRs) can be retrieved in several ways.One approach is a search filter designed to retrieve SRs in bibliographic databases such as PubMed.Another utilizes machine learning (ML) to sort articles into two mutually exclusive classes in online platforms such as DistillerSR.Objective: To test the performance of a binary ML classifier designed to identify SRs in DistillerSR against search filters designed to retrieve SRs in PubMed.Methods: Umbrella reviews will be identified, included SRs will be extracted to create a reference set.Each SR in the reference set will be verified as indexed in PubMed.Two SR search filters will be tested: a narrow SR filter and a broad SR filter.A binary ML classifier developed by DistillerSR to identify SRs will be used, informed by the reference set and a seed of non SR articles.The number of SRs from the reference set retrieved by the two search filters, and the number of SRs from the reference set correctly identified as a SR by the ML classifier will each be recorded discretely.Results: Relative recall, precision and F1 score will be calculated for each set.Recommendations will be made on when to apply a search filter or ML classifier to a search.Conclusions: Different approaches to limiting search results to specific study designs can influence the overall search strategy.The objectives of the project and resource requirements need to be considered when deciding on the approach.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.015 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.004 | 0.000 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".