Digital symptom checkers for COVID-19: a scoping review and pilot study of the ENTIRE quality appraisal tool for health informatics studies
Bibliographic record
Abstract
The COVID-19 pandemic has generated an explosion of digital health applications to support citizens, professionals, healthcare organizations and governments. This scoping review aims to describe the scope and type of studies on COVID-19 digital symptom checkers in the early pandemic, and to pilot the ENTIRE quality appraisal tool for health informatics studies. The search strategy included three concepts: symptom checkers, digital, and COVID-19 and was conducted in Medline, Web of Science and ABI/INFORM Collection. The ENTIRE tool was built adapting items from the Mixed methods appraisal tool, the STAtement on the Reporting of Evaluation studies in Health Informatics guidelines, and from the mobile Health on the Net Code. Of the 4379 unique publications, 29 studies met the inclusion criteria, reporting on 26 COVID-19 symptom checkers. Most studies (75%, 23/29) were observational, and 38% (11/29) had no clear evaluation criterion. Few studies reported the evaluation of usability, and results observed in high-quality studies were mitigated, indicating a low added value of COVID-19 symptom checkers. With the ENTIRE tool, 17/29 studies obtained a score of 6 or more out of 10 (high-quality). Studies on digital symptom checkers were not frequent, and mostly driven by academic institutions and healthcare organizations. The ENTIRE tool was useful to assess the quality of studies: all studies with a quality score below 6 had unclear results. This review confirms the need for further developing standard methods for evaluating the quality of studies on digital health interventions.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.398 | 0.597 |
| Meta-epidemiology (narrow) | 0.003 | 0.003 |
| Meta-epidemiology (broad) | 0.011 | 0.017 |
| Bibliometrics | 0.044 | 0.037 |
| Science and technology studies | 0.003 | 0.004 |
| Scholarly communication | 0.011 | 0.012 |
| Open science | 0.005 | 0.014 |
| Research integrity | 0.004 | 0.004 |
| Insufficient payload (model declined to judge) | 0.009 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".