Value and Acceptability of a Novel Machine Learning Technology for Heart Failure Readmission Reduction: Qualitative Analysis of Clinical Roles and Workflows
Bibliographic record
Abstract
Background: Despite widespread adoption and demonstrated value in a range of industries, machine learning predictive algorithms are yet to be routinely used in frontline medical care. Significant health system and industry-based resources are allocated towards validating and refining predictive algorithms for a range of applications to ensure accuracy and reliability. For these algorithms to be useful and useable, further work is required to understand how and why they might fit into, and augment existing clinical workflows. Objective: This qualitative study assessed the value and usability of a novel machine learning technology to predict and explain the risk of 30-day hospital readmission in patients with heart failure (HF). It involved exploring opportunities for integration of the technology within existing clinical workflows, and investigating key roles that use current readmission risk scores and may use future scores. Methods: Semi-structured interviews (n=27) and targeted observations (n=3) were carried out with key stakeholders, including physicians, nurses, hospital administration, and non-clinical support staff. Participants were recruited from cardiology and general medicine units at an academic medical center within the Partners HealthCare system. Data was analyzed via inductive thematic and workflow analysis. Findings were validated via member checking across limited key roles (n=3). Results: Results highlighted a number of factors that were deemed necessary by staff for successful integration of a risk prediction tool into existing clinical workflow. These included, but were not limited to the following. Staff clearly stated that any new tool must be easily accessible from within the electronic health record, which dictates the majority of existing clinical workflow. Staff emphasized that information should be consistently accurate and that any display must be digestible efficiently, intuitively and quickly (ie, within <5 seconds). Additionally, staff discussed that outputs of the risk prediction tool must match their clinical intuition, experience and interactions with the patient. To be truly valuable, the tool must also provide added value over and above these factors: some staff indicated that provision of role-specific and actionable next steps based on the system output would provide novel value to their daily work. Using these considerations, a number of role groups were identified as potentially able to derive value from the proposed risk prediction tool, including case managers, attending RNs, responding clinicians, hospital administration staff, nursing directors and attending physicians. Acceptability and value varied by role, specialization and clinical context. For example, cardiology-trained clinicians reported feeling well-versed in providing good clinical care and minimizing preventable readmissions, and thus saw less value in the tool. General medicine staff, however, indicated that a HF-specific tool may be impractical for their day-to-day work given the range of clinical presentations seen by them. Conclusions: Findings resonate with existing literature around successful implementation and adoption of technologies in health care. Frontline clinicians are incredibly discerning around proposed changes to their existing workflow. Many HF readmission risk tools and initiatives have been trialled with mixed success; frontline staff demonstrated fatigue around piloting new initiatives. However, given the right conditions, staff reported some perceived value in machine learning-based tools to improve their daily work.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".