Selecting, Scaling, and Measuring Value of Ambient Artificial Intelligence in a Non-Academic Health System: A Multi-Phase Pilot Study (Preprint)
Bibliographic record
Abstract
Abstract Background Most US health systems operate on a local or regional scale and face substantial financial and staffing pressures, which are intensified by challenges related to physician satisfaction and difficulties in recruitment and retention. Ambient artificial intelligence (AI) documentation solutions have the potential to reduce burdens and improve satisfaction, but vendor selection is often undermined by cognitive biases, unvalidated marketing claims, and limited real-world testing. Objective To address this, McLeod Health developed and implemented an objective, multiphase approach to evaluate and adopt an ambient AI solution across its multihospital system. Methods Our evaluation process began in spring 2024 with 4 leading vendors tested through live clinical simulations using 15 complex outpatient scripts, with organizational leaders serving as standardized patients. Ambient patient encounters were captured in real time, and AI-generated notes were scored by physicians, revenue cycle experts, and nonclinical reviewers for accuracy, billing quality, and readability. The top 2 vendors advanced to demonstrations of Epic workflow integration, with physician usability feedback guiding the final selection. In the third phase, the chosen vendor underwent a 90-day pilot across 5 ambulatory specialties beginning in October 2024, followed by system-wide implementation in January 2025. Key performance indicators included documentation time, coding, and financial trends, as well as patient and provider satisfaction. All statistical comparisons were 2-sided using a 95% CI. Results The 3-phase evaluation process resulted in careful vendor selection. The pilot showed a 35.4% decrease in pajama time ( P =.054, trend toward significance) and a 28.3% decrease in time in notes (n=23; P <.001). Coding patterns shifted toward higher-complexity visits, with a 3.8% increase in level 4 established patient visits ( P =.05), and established patient volumes increased by 8.5%, which was associated with a projected revenue gain of US $2629 per provider per month. Patient satisfaction improved significantly across multiple domains, with large gains in listening, trust, communication, and treatment information (all P <.001). These gains exceeded those of prior system-wide patient satisfaction initiatives. System-wide rollout has achieved 81% adoption, with more than 150,000 notes generated. Conclusions Our structured, multiphase evaluation process minimized vendor influence and cognitive bias during selection, validated results through real-world clinical testing, and enabled a system-wide rollout. This approach offers a practical framework for nonacademic health systems to objectively assess, implement, and scale ambient AI solutions while preserving fairness, transparency, and measurable value.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.047 | 0.065 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".