A comparative performance analysis of live clinical triage using rules-based triage protocols versus artificial intelligence-based automated virtual triage
Bibliographic record
Abstract
Objective: Compare the triage care referral accuracy of artificial intelligence (AI) based virtual triage (VT) to rules-based triage protocols (RBTP) live telephonic triage.Methods: Clinical vignettes were selected for a comparison of care referral accuracy of RBTPs with a widely utilized AI-based VT solution. Vignettes (149) included patient complaints, expected triage and urgency assessment. Triage levels were mapped to three triage categories (urgent care, non-emergent care and self-care). Each vignette was evaluated/completed using AI-based VT and RBTP triage modalities by a total of four physicians in series, with independent assessment for errors and inconsistencies. Triage assessment precision was analyzed by matching the expected triage assessment, sensitivity and F1 scores (harmonic mean of precision and recall).Results: Both modalities achieved > 70% triage accuracy, and safety performance was identical at 91%. AI-based VT was more accurate in care referral for emergency and non-emergency care and overtriaged to emergency care 50% less frequently than RBTP, but was less accurate than RBTP in self-care vignettes (neither statistically significant). Both modalities demonstrated decreased sensitivity as care urgency/acuity decreased, more pronounced in AI-based VT than RBTP. AI-based VT captured four times as much information and data as RBTP.Conclusions: AI-based VT and RBTP were comparable in care referral accuracy and disposition safety. While AI-based VT provides accurate and safe triage recommendations at a lower total cost, care organizations should assess how AI-based VT compares to a live clinical triage capability with respect to organizational priorities, budgetary considerations, characteristics of the patient/member population served, and the existing technological environment.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.009 | 0.055 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".