Invisibility, cloaks and daggers: Balancing clinical hazards in the age of artificial intelligence

In the paper by Sibbald et al.,1 the authors argue that to begin to actualize the use of artificial intelligence (AI) in the clinical setting, it needs to be 'invisible', or as they describe it, provide clear value, be easy to use and offer clinicians a clinically relevant tool that is willingly adopted and habituated over time by considering change management, human factors and implementation challenges inherent to the clinical setting. Their argument is that this is common practice within other industries, leveraging processes like design thinking to improve systems design and user experience. However, they lament that these approaches are not often applied to the healthcare setting. By applying codesign and design-thinking processes, Sibbald and colleagues argue that they may address some of the hurdles which have thus far limited the adoption of AI in the clinical environment. We support the authors in their call for better design in healthcare. They are certainly not alone,2-5 and we are amongst those who agree that it is high time that designers be brought more actively into the healthcare space to improve our systems. This paper features a reasonable approach to improve adoption, and we agree that the foundation of many successful technologies stems from a seamless user experience.6 However, we wonder if such user-centred approaches have been absent from healthcare for reasons beyond inconvenience or naivaté. We posit that some reasons such approaches are not widely employed in the medical setting may be due to the unique circumstances of patient care. As opposed to commercial app development, there exist some hazards to having an 'invisible' layer of AI technology within the clinical world. As the authors note within their paper, 'invisible' AI may lead to serious complications, and we worry that it could even possibly lead to worsened clinical outcomes, especially for traditionally marginalized and/or disadvantaged groups.7, 8 Thinking through the AI pipeline, there are three particular areas to consider: the data used to develop a tool, the tool's implementation and the outcome of interest when using that tool. Data used to develop many AI solutions in healthcare have been notably flawed.9 Notably, differential and biased care in the past often provides the basis for worse predicted outcomes in the future. Race-based measures can affect the likelihood of receiving a kidney transplant or other limited resources.10 While these predictions were based on data, that data has been inaccurate and reflective of systemic biases.9 Furthermore, data are often used in a different setting than which it was obtained. The scientific method requires hypothesis testing and developing data sets to answer a specific question. In this data-hungry AI world, many data repositories are used outside the scope for which they were originally intended. Financial and bank data are used to predict healthcare outcomes which may only reflect the higher risks associated with living in lower socioeconomic status. Thus, any intervention utilizing these data might be better served to address the root cause, rather than to use it to determine which medication is best afforded by a patient. Disentangling associations of race, gender, class, access, geography will be a great challenge for healthcare AI. After all, in many regards, machine-learning algorithms are just seeking to find patterns in data, but we will need savvy and sophisticated end-user experience to help discern, verify, act upon and revise these association- and pattern-detectors. The failure to do so will have massive implications for perpetuating bias and marginalization of certain segments of stakeholders. Beyond just the data which are used to make decisions, their seamless implementation is likely to rely on the willingness of important vendors (like Epic or Cerner) to grant access to their real-world data or allow it to be used in the clinical environment. Effectively, these systems are implemented at the level of health systems, and without increased collaboration or buy-in by these industry leaders to affect change, academics will continue to speak only to themselves within scholarly zones such as this.11 This barrier to implementation has continued to be the case despite the requests of clinicians and researchers. An unfortunate reality is that seamless integration will likely need to conform with particular vendors and their commercial needs, and they may not be cost efficient or incredibly useful in the beginning. Thus, invisible AI will likely require balancing commercial needs with human outcomes we seek to improve—a recipe for conflicts of interest and clinician and patient suspicion. Implementation scientists seeking to move the mark in this area could explore industry−academic funding pathways and broker relationships with electronic health record vendors to allow for more experimentation and rapid prototyping. However, without buy-in of these parties, it will be virtually impossible to scale the use of AI to drive change in healthcare. Lastly, we must perceive and be specific about the outcome of interest which we hope to gain from any AI tool, whether to classify findings, automate processes or offer decision support. How 'invisible' the tool can be will likely be determined by this outcome. In the case of the electrocardiogram (ECG), the tool enhances the classification and interpretation of a graph by honing in on features which are common to certain conditions. It does not help the clinician order tests or activate the catheterization lab automatically, though these may be reasonable outcomes of a future tool. In the latter setting, we would need to determine a threshold level of certainty that the AI tool can make an accurate prediction, otherwise, there can be real harm associated with utilizing this tool in the wrong clinical setting due to excessive resource expenditure or unnecessary catheterization procedures. Essentially, in order for AI implementations to be 'invisible', the outcome of their use will likely need to be of relatively low risk to patients or the healthcare system. Those that have a higher risk, whether to resources or medicolegally, will likely require more up-front identification and reporting of how the tool is being used to prevent providers from simply using it without an appropriate understanding of the potential pitfalls. Thus, how 'invisible' an AI tool can depends on how much we trust the data it uses, in what form or setting we utilize the tool and what the outcome of interest for using the tool is and how much harm it may produce. We cannot approach AI in medicine as a monolithic data infrastructure and ecosystem, but rather a continuum of decisions, processes and interventions which must be evaluated for how much of a human response is needed—or how invisible an AI tool can be. As the authors state, opening the black box alone will not make AI implementations easier. But making the black box openable in a way that is useful to the end user may offer a check to the possibility of runaway effects.12-14 To address these issues, we might look to another design thinking solution: user experience. We don't necessarily need to know everything about a tool for it to work. Instead, the best result likely lies in balancing utility with discoverability and understanding. If the human goal is to better understand bed availability, that may not come with significant risks if the computer is incorrect and a user might reasonably assume that this prediction is based on the nature of the patient currently in the room or how long it has been since they left or are awaiting a cleaning team. However, if the goal is to provide a life-saving therapy to one patient versus another, the ability to understand which risk factors constituted that choice, and where the data came from, the ability to drill-down, is a must (Figure 1). In the case of the ECG, the ability to drill down and evaluate the primary data comes directly with the interpretation. Perhaps the next step for medical AI is not to change the status quo altogether, but to begin to incrementally add to it the layers of impact necessary to engender trust. Ultimately, when it comes to AI in medicine, regardless of how easy to use the tools and the reasonable nature of its founding theory and evidence, governance and surveillance are necessary over time to make sure that it isn't unexpectedly widening care disparities. We have seen chatbots rapidly begin to use racist language and major sepsis models unable to provide the same level of prediction in different hospital locations. But the only way we knew about the problem and could address it was because we had humans directing it and guiding it. Invisibility has its place, but sometimes we need to see the mechanics of what lies under the hood when things go wrong. Thank you to Dr. Sandra Monteiro and Dr. Mathew Mercuri for inviting us to write this commentary. And to the original authors of the piece upon which we commented for spurring an interesting discussion (Sibbald M, Zwaan L, Yilmaz Y, Lal S). Dr. Rose reports receiving grant funding from the McCoy Family Center for Ethics in Society and the Human-Centered Artificial Intelligence Center of Stanford University as well as consulting fees for Caption, Inc. Dr. Chan reports honoraria from McMaster University for her education research work with the McMaster Education Research, Innovation, and Theory (MERIT) group and administrative stipend for her role of associate dean via the McMaster Faculty of Health Sciences Office of Continuing Professional Development. She also discloses that she has received various unrelated research grants, teaching honoraria and speakership fees from academic institutions (Baylor University/Texas Children's Hospital, Catholic University of Korea, Taiwan Veteran's General Hospital, Prince of Songkla University, Harvard Medical School, International Association of Medical Sciences Educators, Ontario College of Family Physicians, Northern Ontario School of Medicine, University of British Columbia, University of Northern British Columbia, Holland Bloorview), nonprofit organizations (PSI Foundation), physician organizations (Association of American Medical Colleges, Canadian Association of Emergency Physicians, Society of Academic Emergency Medicine, the Royal College of Physicians and Surgeons of Canada, Medical Council of Canada), and governmental sources (Government of Ontario, Virtual Learning Strategy eCampus Ontario program). artificial intelligence, bias, delivery of healthcare, health analytics, machine learning, technology, user-centred design Research data are not shared.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.