AI in the ED? Maybe

Figure: machine learning, AI, artificial intelligence, emergency medicine, ED, EM, hospitals, CT imaging, diagnostic accuracy, ICH, hemorrhage, workflow, triage, syncope managementMachine learning is not ready for prime time, but validated models will likely be critical for EM As emergency departments become ever more crowded and emergency physicians and other clinicians more and more overworked, can machine learning (ML) and artificial intelligence (AI) help ease the burden by speeding up triage, predicting the need for hospital admission or otherwise improving the diagnostic process? At this point, the answer from most experts and data-crunching studies seems to be a qualified “maybe.” A systematic review of studies involving the use of ML/AI versus usual care for diagnostic and prognostic prediction in the ED found that ML appeared to have better prediction performance than usual care for ED patients with a variety of clinical presentations and outcomes. (Acad Emerg Med. 2021;28[2]:184; https://bit.ly/3CvzPKa.) The authors found ML models outperformed usual care in all performance metrics in the seven diagnostic studies they reviewed. Six studies assessing in-hospital mortality showed the best-performing ML models had better discrimination (area under the receiver operating characteristic curve [AUROC]=0.74-0.94) than any clinical decision tool (AUROC=0.68-0.81). Four studies assessing hospitalization found that ML models had better discrimination (AUROC=0.80-0.83) than triage-based scores (AUROC=0.68-0.82). The idea is to use objective data to assess whether machine learning models are working better than regular clinical decision rules or clinician gestalt for diagnostic or prognostic purposes, said lead author Hashim Kareemi, MD, a fifth-year emergency medicine resident at the Ottawa Hospital of University of Ottawa. “That's the first hurdle they need to get over in order to justify any trial of ML in a real-world clinical setting,” he said. A study he and colleagues conducted looked only at models comparing machine learning to a clinician or another validated model, and on the whole, ML seemed to have better performance metrics in most of the comparisons. But Dr. Kareemi cautioned against overinterpreting these findings because the data in most of the studies they found were not high quality, were based on retrospective studies of registries, and had a lot of natural biases. “The people who are collecting the data are not looking at a specific question, so you get a lot of noise and not particularly robust data,” he said. Another major limitation was the lack of external validation. Many studies would train their ML model on a data set, and then test it in the same data set, Dr. Kareemi said. “Obviously, you shouldn't do that; that's kind of like cheating,” he said. “To better assess the accuracy and utility of ML in the ED or any other clinical setting, we need high-quality, prospectively collected data, and external validation. They need to be studied in actual clinical trials.” Other studies have also found ML to have no significant advantage over traditional diagnostic methods. One systematic review by British and Belgian researchers found ML performed no better than logistic regression in clinical prediction modeling for binary outcomes in 71 studies with a median sample size of 1250. (This review was not specifically focused on studies in the ED; J Clin Epidemiol. 2019;110:12.) Like Dr. Kareemi, these authors found noteworthy issues with data quality, observing potential bias in the validation procedures in 48 (68%) of the studies. And a recent study of an ML approach for predicting the need for hospital admission in patients with traumatic brain injury or skull fractures identified on CT imaging compared with the institution's decision rule developed using traditional statistical techniques found no clear advantages for ML in sensitivity or specificity. (J Neurotrauma. 2023 Apr 24; doi: 10.1089/neu.2022.0515.) Yet another study integrated an AI-based tool for detecting intracranial hemorrhage (ICH) on noncontrast CT images into ED workflow and found it yielded an overall diagnostic accuracy of 93.0 percent, with 87.2% sensitivity and 97.8% negative predictive value. (Radiol Artif Intell. 2022;4[2]:e210168.) Workflow was also streamlined based on measures such as communicating a critical finding (70 minutes [95% CI: 49, 85] vs. 63 minutes [95% CI: 55, 71]) and minimum communication time of acute ICH (73 minutes [95% CI: 49, 97] vs. 58 minutes [95% CI: 48, 68]). Importantly, however, the investigators noted that the tool had its limitations, yielding lower detection rates for specific subtypes of ICH, such as 69.2 percent (74 of 107) for subdural hemorrhage and 77.4 percent (24 of 31) for acute subarachnoid hemorrhage. Common false-positive findings included postoperative and postischemic defects, artifacts, and tumors. (These studies are too recent to have been included in either of the reviews.) Despite mixed findings in current data, Dr. Kareemi said he believed ML and AI will be important tools for emergency physicians in the future. “We have finite resources to deal with more and more patients who are increasingly complex, with a larger number of therapeutic options,” he said. “The job of the emergency physician is becoming infinitely more challenging. ML and AI restructures the information that we have and thinks about it in ways that our biological brains can't quite grasp. We are quite good at collecting data; we just need to create models and algorithms that can synthesize it in an effective way to help us support or verify our clinical decisions.” Applications in the ED The potential uses of ML and AI in the ED fall into several categories, said Shammi Ramlakhan, MD, an emergency physician and a professor at Sheffield Children's Hospital in Sheffield, England, who has written several journal articles on building and utilizing these models in the ED as well as in other applications, such as neural networks for novel image interpretation. “First, there are operational type models; those that help with predicting attendances and arrival times, admission likelihood, and to facilitate triage for both clinical and operational reasons,” he explained. The use of ML models in the ED can facilitate triage with more accuracy and efficiency, requiring only information routinely collected by the triage staff, said Sangil Lee, MD, a clinical associate professor of emergency medicine at the University of Iowa's Carver College of Medicine, who has also published on AI and ML in the emergency department. (Acute Med Surg. 2022;9[1]:e740; https://bit.ly/465sqz2.) “In addition to predicting the urgency of medical conditions, ML techniques can be applied to develop screening tools for disease-specific risk prediction,” he said. Models are also being used for clinical decision-making, said Dr. Ramlakhan, helping to risk-stratify for certain conditions, such as sepsis, MI in chest pain, and stroke, going beyond the traditional clinical decision rules. Some of these use traditional CDR variables and then embed additional data that can be obtained from an electronic health record, triage, or the patient's intake form, he said. Diagnostic imaging interpretation is a particularly promising area for AI in the ED, Dr. Lee said. “Deep learning models for medical imaging with high sensitivities could help clinicians quickly identify life-threatening pathologies,” he explained. Dr. Ramlakhan said another example might be the interpretation of ECGs. “We already have low-fidelity software that interprets ECGs, but these interpretations are often unhelpful clinically or for workflow,” he said. “At the next level, an AI trained on multiple ECGs can flag potentially critical ones for the physician to review. Similarly, as with imaging, AI might highlight an area that perhaps you should look at a bit more closely so that subtleties aren't missed.” A collaborative review by Dr. Lee and colleagues found that the use of AI could also enhance syncope management. (JACC: Advances. 2023;2[3]:100323; https://bit.ly/46g5vRE.) “Syncope is very heterogeneous, and we need additional tools to guide clinicians in risk stratification,” he said. “Our study used a type of AI called a deep learning neural network to predict the length of hospital stay for people who present to the ED with syncope. We found that it was very accurate in predicting those patients who would only need to stay for a day or two versus those who would have to stay for more than three or four days, which has implications for the types of resources these patients will need. We also hope that having this information could help clinicians better plan care for these patients.” Another area where ML and AI could be deployed in the ED would be in workflow and process improvement, such as with completion of electronic health records. “There might be a role for that sort of generative AI in some aspects of emergency practice, minimizing non-value-added tasks that take up physician time,” Dr. Lee said. Designing AI Trials Retrospective data analysis is all well and good, but a key question still needs an answer: Do ML models perform better than existing standards of care in a prospective, real-world clinical scenario? Dr. Kareemi, now pursuing a master's degree in clinical epidemiology at the University of Ottawa, is developing a thesis project around how to design an interventional clinical trial using a machine learning model in a way that is robust and methodologically sound. “How are we going to implement an ML model in a clinical scenario? And once we decide to do that, how do we integrate it into an electronic health record, get physicians to use it and trust it, and get patient buy-in and consent for an ML model being involved in clinical decisions? Those are just a few questions that have to be addressed before we can go ahead with trials,” he said. “We need a lot of high-quality data, ideally prospectively collected, to train the AI. These are not like typical clinical decision rules; they require lots of events, patients with a positive outcome. How we tackle that issue is a big matter of debate in the field.” The field is still far from maturity, but he said he believed that there is an important role for well-validated ML models in emergency medicine. “It will likely have to start out with low-stakes decisions,” Dr. Kareemi said. “ML models have a whole black box around them where we don't completely understand how they work or why they're making the decisions they make.” Patients and physicians may be more comfortable with that if the clinical scenario is low risk, he said. Does a patient need blood work right at triage, for example, or can he wait to be seen by a physician? “As long as the decision is not saying, ‘you get care and you don't,’ but ‘should you get care right now versus an hour or two down the line,’ I think that would potentially be OK for physicians and patients, but it would need to be borne out in studies that include patients' voices,” Dr. Kareemi said. Ethical Concerns Dr. Ramlakhan noted that the philosophical and ethical issues surrounding the adoption of AI require careful consideration. “We need to understand the inherent bias in some data sets, particularly those that amplify human bias,” he said. “There have been demonstrable instances where racial or gender bias has crept into an AI tool and become amplified or embedded into the models, and it's difficult to tease that out after the model is deployed.” He said that led him to the conclusion that all AI developers need to validate their models externally and specifically in the context and similar type of population to where it's going to be used. What about questions of AI ultimately replacing physicians? Dr. Ramlakhan said he believed those fears are unfounded. “There will always be a need for that human interaction, particularly in emergency medicine,” he said. “The benefit of this technology is that it can make things easier for both the doctor and the patient, assisting clinicians rather than replacing them. Once it fits into the workflow, it's not something that you have to put a lot of effort into using, and it can reduce the cognitive burden on the physician and remove some of those low-value interactions where you don't need a skilled practitioner.” Tasks like routine information gathering can be automated, making EPs more efficient and freeing up time for patient interactions, Dr. Ramlakhan said, adding that his vision for AI and ML is to improve the quality of actual clinical time that physicians have with patients. AI can also help keep physicians on the evidence-based path, he said, addressing the variability in how individual doctors assess patients. “Maybe it's inherent bias, maybe they're tired, maybe they read an article recently, all kinds of factors that can change the assessment subtly or not so subtly,” Dr. Ramlakhan said. “Regardless of what the condition or diagnosis is, adding AI to support decision-making can remove some of that variability by focusing, interpreting, and amalgamating relevant information to help facilitate evidence-based clinical management.” MS. SHAW is a freelance writer with more than 20 years of experience writing about health and medicine. She is also the author of Having Children After Cancer, the only guide for cancer survivors hoping to build their families after a cancer diagnosis. You can find her work at www.writergina.com. Follow her on Twitter @writergina. Share this article on Twitter and Facebook. Access the links in EMN by reading this on our website: www.EM-News.com. Comments? Write to us at [email protected].

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.