Evaluating diverse electronic consultation programs with a common framework
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
BACKGROUND: Electronic consultation is an emerging mode of specialty care delivery that allows primary care providers and their patients to obtain specialist expertise without an in-person visit. While studies of individual programs have demonstrated benefits related to timely access to specialty care, electronic consultation programs have not achieved widespread use in the United States. The lack of common evaluation metrics across health systems and concerns related to the generalizability of existing evaluation efforts may be hampering further growth. We sought to identify gaps in knowledge related to the implementation of electronic consultation programs and develop a set of shared evaluation measures to promote further diffusion. METHODS: Using a case study approach, we apply the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) and the Quadruple Aim frameworks of evaluation to examine electronic consultation implementation across diverse delivery systems. Data are from 4 early adopter healthcare delivery systems (San Francisco Health Network, Mayo Clinic, Veterans Administration, Champlain Local Health Integration Network) that represent varied organizational structures, care for different patient populations, and have well-established multi-specialty electronic consultation programs. Data sources include published and unpublished quantitative data from each electronic consultation database and qualitative data from systems' end-users. RESULTS: Organizational drivers of electronic consultation implementation were similar across the systems (challenges with timely and/or efficient access to specialty care), though unique system-level facilitators and barriers influenced reach, adoption and design. Effectiveness of implementation was consistent, with improved patient access to timely, perceived high-quality specialty expertise with few negative consequences, garnering high satisfaction among end-users. Data about patient-specific clinical outcomes are lacking, as are policies that provide guidance on the legal implications of electronic consultation and ideal remuneration strategies. CONCLUSION: A core set of effectiveness and implementation metrics rooted in the Quadruple Aim may promote data-driven improvements and further diffusion of successful electronic consultation programs.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.002 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it