Racial and Ethnic Biases in Computational Approaches to Psychopathology

Computational methods are a promising approach for the study, assessment, and treatment of mental illness.1–4 Natural language processing, automatic speech recognition, and facial recognition technology have each been used to systematically and automatically identify abnormalities in speech,5–9 language,10–18 and facial expressivity8,9,19–23 that characterize psychosis (see recent review14 for additional background on these methods). Because computational methods automatically extract measures of language, behavior, and expressivity, they have the potential to save time on extensive expert training and eliminate costs on expensive apparatus often necessary for measurement in these domains.24,25 Further, because the methods are amenable to naturalistic data sources, they can limit participant and patient burden, and can be used in contexts where the employment of cutting-edge assessment and treatment modalities have historically been severely limited or unavailable.26 Taken together, automatic measures could serve as a resource multiplier for clinicians and researchers alike, allowing them to rigorously assess marginalized individuals who may otherwise have fallen through the cracks, helping to reduce existing disparities in mental health outcomes.27 Indeed, half of the surveyed psychiatrists think that machine learning will significantly transform their jobs in the near future.3,28 However, it is not all good news. A common assumption is that computational methods avoid the harmful biases inherent in human raters—biases that can confound research findings and exacerbate structural inequity in clinical applications.29 However, recent research suggests that in fact computational methods can reify and magnify, rather than reduce, existing health disparities.3,30–39 Here, we review evidence that harmful racial biases may exist in computational methods already in use in studying psychosis and argue that a proactive approach to addressing these issues is urgently needed—especially as relates to racial identity.1,3,40 In initiating this discussion, we rely on macro-level sociodemographic groups (eg, “Black”) that are currently widely used and discussed in the United States. This allows us to point out problems and discuss solutions within the current real-world research-policy-practice interface. However, this macro-level approach can also be problematic. It can reify existing biases, promote a false notion of a uniform construct, and ignore the significant heterogeneity that exists within racial and ethnic groupings both within41,42 and across different cultures.43 Macro-level analyses are therefore not the final word on the topic, but rather provide a foundation for more detailed analyses to engage with the full range of human experiences. There are several potential issues with the automated methods currently used in the field. These methods have been shown to perform worse on racial minorities in other fields they have been applied in. In gender detection, the automated facial analysis systems that underlie facial emotional expressivity measures show error rates of only 0.9% on lighter-skinned men, but error rates of over 30% on darker-skinned women.33 Similarly, automated technologies currently being used for vocal analysis make twice as many errors on speech from Black individuals than White.44 It is highly likely that these higher error rates have been carried forward in studies detecting schizophrenia through automated vocal and facial emotion analyses. In our own work, we have found patterns consistent with racial bias in automated coherence models. These rely on machine learning to identify thought disorder in individuals with psychosis or at clinical high-risk for psychosis.45 Given a patient’s language sample, these models automatically assign a score intended to reflect the sample’s semantic cohesion. These scores have been argued to differentiate individuals with formal psychotic disorders from healthy controls with accuracies reaching 100%.10,12,13 However, past work has broadly compared psychosis/at risk for psychosis groups to healthy controls. In our work, we examined performance across racial groups. When analyzed by racial identity, these automated methods rated narrative speech samples from Black speakers as less coherent than those of White speakers—regardless of case vs healthy control status. Supposedly objective algorithms tended to rate entirely asymptomatic Black participants as having speech patterns consistent with thought disorder.2 This is problematic, as the field already has a bias toward misdiagnosing Black participants with schizophrenia at disproportionately high rates.46,47 Further, as this work was conducted in a clinical high-risk context, a next step translational or precision medicine version of the study (aiming to predict conversion and inform treatment) could yield false positives—which might lead to inappropriate treatment and additional stigma. This is not merely a theoretical possibility; researchers are already highlighting the limitations of screening instruments which predict psychosis-risk more accurately for White than for Black participants.48 Finally, these effects are unlikely to be limited to language. Automated facial emotion detection systems are more likely to rate Black individuals as expressing negative emotions—regardless of reported emotions30–32 and acoustic analysis systems make more errors on speech by Black individuals.44 Consequently, clinical studies using these techniques may also misdiagnose Black participants at disproportionately high rates—due to erroneous measures of expressivity and vocal productions. How can such issues arise? State-of-the-art machine learning algorithms are tuned based on a set of training data before being deployed. Unfortunately, these methods have often been primarily trained on samples from White individuals; this can distort algorithm performance on data from other social groups.49 This is particularly salient in the context of language, which is pervasively used to communicate and reinforce systems of racial oppression.50 For example, a recent study of personal narratives (a prototypical language sample for automated analysis) by one of our authors found that Black participants were more likely to focus on topics of danger and adversity, while White participants were more likely to focus on personal growth.51 Another issue in machine learning is that algorithms are often trained to mimic human annotations (eg, emotion detection systems are trained on human annotations of whether a particular individual looks happy, angry, etc.). If human annotations are biased,52 algorithms will detect and magnify these biases in their assessments, hard-wiring the very biases we hoped to avoid. Given these issues, how should the field proceed? A key insight from previous work is that diagnosing and eliminating these biases should directly guide development of computational methods, rather than follow it. When ethical issues are instead treated as tangential, there can be severe negative consequences. For instance, Cambridge Analytica was able to exploit personality assessment methods to influence politics in the United States and the United Kingdom.53 Similarly, Psychopathy Checklist-Revised (PCL-R) psychopathy scores are still widely used by prosecutors and parole boards, despite concerns about their reliability and validity.54 In this regard, psychopathology research is in a strong position: as computational methods are still relatively new, concerted efforts to address these issues now could help avoid serious problems. These critical challenges warrant large-scale and thorough investigations of the relationship between social identities—such as race—and automated algorithm performance in the study of mental illness. Collaborative, multidisciplinary studies applying a range of algorithms to large, diverse samples could better elucidate the presence and source of biases, fueling work aimed at improving upon existing methods. Papers that apply computational methods should treat generalizability across social groups as a central, explicit evaluation criterion. Work applying automated methods should always report relationships (or lack thereof) with key sociodemographic factors (ie, racial and gender identity, education, SES, etc.) and should diagnose why any identified relationships are observed. Such analyses should be guided by findings from previous social scientific studies of social differences or biases in human judgments. For example, if an automated system including language makes different predictions based on social group membership, social group differences in language use established by sociolinguistic research can inform the analysis of the algorithm as well as interventions applied to mitigate bias. Although these steps may delay implementation of computational methods, it will be most effective for psychosis researchers to address potential issues of bias early in the design process of computational methods before they are implemented in practice. By analogy, in complex systems design, early design defects are relatively simple to correct if they are addressed early. However, the cost dramatically increases if these defects are not addressed until after large-scale implementation.55 Computational methods that automatically identify abnormalities in speech, language, facial expressivity, and other aspects of human behavior could be transformative for the field. However, by overfocusing on the successes of these models, we run a very real risk of worsening existing health disparities. As a result, it is critical that we prioritize generalizability across social groups, to ensure that psychiatric computational methods do not become another domain that perpetuates existing systemic biases. The authors have declared that there are no conflicts of interest in relation to the subject of this study. This work was supported by the National Institutes of Health (grant numbers NIH R21 MH119677, T32 NS047987) and the Canadian Institutes of Health Research (grant number CIHR DFS-152268).

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.