Digital symptom checkers for COVID-19: a scoping review and pilot study of the ENTIRE quality appraisal tool for health informatics studies
Bibliographic record
Abstract
Background The COVID-19 pandemic has generated an explosion of digital health applications to support citizens, professionals, healthcare organizations and governments.This scoping review aims to describe the scope and type of studies on COVID-19 digital symptom checkers in the early pandemic, and to pilot the ENTIRE quality appraisal tool for health informatics studies. ResultsThe search strategy included three concepts: symptom checkers, digital, and COVID-19 and was conducted in Medline, Web of Science and ABI/INFORM Collection.The ENTIRE tool was built adapting items from the Mixed methods appraisal tool, the STAtement on the Reporting of Evaluation studies in Health Informatics guidelines, and from the mobile Health on the Net Code.Of the 4379 unique publications, 29 studies met the inclusion criteria, reporting on 26 COVID-19 symptom checkers.Most studies (75%, 23/29) were observational, and 38% (11/29) had no clear evaluation criterion.Few studies reported the evaluation of usability, and results observed in high-quality studies were mitigated, indicating a low added value of COVID-19 symptom checkers.With the ENTIRE tool, 17/29 studies obtained a score of 6 or more out of 10 (high-quality). ConclusionStudies on digital symptom checkers were not frequent, and mostly driven by academic institutions and healthcare organizations.The ENTIRE tool was useful to assess the quality of studies: all studies with a quality score below 6 had unclear results.This review confirms the need for further developing standard methods for evaluating the quality of studies on digital health interventions.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.005 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.005 | 0.001 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".