Abstract B002: Fairness by Design: End-to-End Bias Evaluation for LLM-Generated Data

Abstract Background: Because LLMs are often pre-trained on data that reflects systemic inequities, they are prone to reinforcing existing biases, so it is critical to assess their performance in order to prevent downstream harms. We propose an approach that comprehensively evaluates data quality, and apply it in assessing subgroup bias in the context of extracting clinical variables. Methods: A LLM was developed to extract initial and metastatic (met) breast cancer (BC) dx and date from the electronic health record (EHR). To assess quality, we leveraged three tools: variable metrics, verification checks and replication. For variable metrics, a test set of 772 patients (pts) underwent duplicate abstraction. We calculated recall and precision for both the abstractor and LLM for correctly identifying dx and date (within 30 days). For verification checks and replication, a LLM-curated BC cohort (N=935,240) was compared to a human abstracted cohort (N=56,861) from the Flatiron Health Research Database. Verification checks assessed how often pts received definitive surgery or radiation after met dx (clinically unexpected). Replication assessed overall survival (OS) using the Kaplan-meier method comparing OS estimates from LLM-identified vs. abstraction-identified BC dx date. To assess bias, we stratified performance by race/ethnicity (Latinx, White, Black) and age (<50, 50-64, 65-75, 75+). Results: For initial dx, LLM performance differed by race/ethnicity with higher recall among Black (+4.7%, CI:2.7%, 6.7%) and higher precision among Latinx (+1.2%, CI:0.2%, 2.3%) compared to White pts. We saw lower recall among pts 75+ (-7.1%, CI:-13.9%, -0.4%) compared to pts 50-64. We saw similar trends in abstractor performance except for pts 75+, with higher recall among Black (+1.2%, CI:0.2%, 2.2%) and higher precision among Latinx (+2.3%, CI:0.9%, 3.7%) compared to White pts. Performance of extracting both dx status and associated date did not vary across race/ethnicity or age groups. Verification checks showed no variation by race/ethnicity or age, except among pts 75+ among whom we saw a higher rate of surgery after met dx (21%) compared to those with expected sequence (15%, SMD=0.22). Finally, OS from initial and met dx aligned between abstracted- and LLM-generated datasets, except for the Latinx group, where confidence intervals did not overlap for 5-year OS (LLM:0.71 CI:0.70-0.71; abstracted:0.73 CI:0.71-0.75), but absolute differences were small. Discussion: Using Flatiron’s quality assessment framework enabled a nuanced evaluation of bias in EHR data curation. We observed small but present subgroup differences across both abstraction and LLM curation approaches, which may reflect clinical or documentation complexity. LLMs showed lower recall for patients 75+, warranting validation on a larger test set. As LLMs become increasingly central to RWD, proactively assessing and addressing bias is critical. These insights can inform improvements to the modeling process (i.e., new prompts) to enhance curation fairness. Citation Format: Melissa Estevez, Olive Mbah, Asad Sheikh, Patrick Ward, Evan Vietorisz, Megan Hildner, Lauren Dyson, Nisha Singh, Aaron B. Cohen. Fairness by Design: End-to-End Bias Evaluation for LLM-Generated Data [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B002.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.