Applications of AI in public health: Improving access to high-quality evidence syntheses
Bibliographic record
Abstract
Abstract Issue Using the best available evidence is crucial for effective public health practice. Health Evidence™ provides access to high-quality synthesis evidence relevant to public health. As the volume of peer-reviewed published literature grows, maintaining a database of this magnitude is increasingly resource-intensive. Artificial intelligence (AI) can be used to reduce the maintenance burden thereby ensuring decision makers can easily access high-quality synthesized public health evidence. These innovative strategies may be transferable to enhance efficient maintenance of large curated databases of published literature. Description of the problem In 2020, The Health Evidence™ team conducted extensive training and testing of a supervised machine learning application to explore the accuracy of AI-assisted reference de-duplication and relevance screening. Finding promising results, the team implemented these AI-assisted strategies in August 2020. To assess the impact on the overall screening burden and time saved, implementation data was analyzed between November 2020 to 2023. Results For these 3+ years, AI assisted de-duplication and relevance screening was applied to 394,903 search results. The AI assisted de-duplication application removed 31% (n = 123,903) of references as duplicates. From the remaining reference sets (n = 272,253), the AI assisted screening application removed 70% (n = 190,966) of references as not relevant. Quality assurance spot testing found minimal classification errors (n = 1). In total, AI assisted approaches reduced the need for manual screening by 80%, saving approximately 626 hours of manual screening time over three years (or approximately 17 hours/month). Lessons With the reality of limited public health resources, continued access to high-quality synthesis evidence is critical. Innovative strategies using AI-assisted applications improves the feasibility of maintaining a large database of quality-appraised public health synthesis evidence. Key messages • Access to high quality synthesis evidence is critical for evidence-informed decision making. • Artificial Intelligence can be used to efficiently identify evidence syntheses relevant for public health.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.434 | 0.763 |
| Meta-epidemiology (narrow) | 0.002 | 0.004 |
| Meta-epidemiology (broad) | 0.005 | 0.007 |
| Bibliometrics | 0.052 | 0.034 |
| Science and technology studies | 0.004 | 0.005 |
| Scholarly communication | 0.025 | 0.016 |
| Open science | 0.008 | 0.021 |
| Research integrity | 0.005 | 0.008 |
| Insufficient payload (model declined to judge) | 0.019 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".