Abstract B006: Using large language models for scalable extraction of real-world progression events across multiple cancer types

Abstract Background: Accurate identification of cancer progression events from electronic health records (EHRs) can help enable promising oncology applications such as predicting disease trajectory, assessing treatment efficacy, and generating real-world evidence. These use cases require both large-scale and high-quality data but manual abstraction of real-world progression (rwP) is time-intensive, difficult to scale, and inherently challenging given the varied and unstructured ways it can be documented across cancer types. Large language models (LLMs) offer a scalable alternative, but their accuracy relative to expert human abstractors is unclear. We evaluated the ability of LLMs to extract rwP events and dates across 7 cancer types and assessed how using LLM-extracted data impacted real-world progression-free survival (rwPFS) estimates compared to using human-abstracted data. Methods: We applied LLM-based extraction techniques to unstructured EHR text for 7 cancer types from the Flatiron Health Research Database: bladder (N=377), breast (N=1000), colorectal (N=564), hepatocellular (N=217), renal cell (N= 229), non–small cell lung (N=1000), and small cell lung (N=955). Prompt engineering strategies including zero-shot, few-shot, and chain-of-thought were tested to optimize performance. We measured agreement between the LLM and abstractor on the presence of rwP (Y/N) and first rwP date (± 30 days) in the first-line (1L) setting. To contextualize the LLM’s ability to extract rwP relative to an expert human abstractor, we evaluated the difference in F1 scores between both curation approaches using a duplicate human-abstracted reference dataset. We also compared rwPFS calculated from LLM-curated data versus human abstractor-curated data for 1000 patients in each cancer type, indexed to 1L start date. Results: Across all cancer types, agreement between the LLM and abstractor on the presence of at least 1 rwP event ranged from 86%-90% while first rwP date agreement ranged from 80%-92%. The difference in F1 score between the LLM and human abstraction was within 3-8 points across cancer types. A comparison of rwPFS between the LLM and human abstractors showed <1 month difference in median rwPFS and overlapping 95% confidence intervals across all cancer types. Discussion: LLMs extracted rwP with high performance, achieving F1 scores similar to expert human abstraction. Across 7 distinct cancers, agreement with human-abstracted data aligned with published inter-abstractor reliability benchmarks, and rwPFS estimates were nearly identical across curation approaches, demonstrating both the generalizability and validity of the approach. These results highlight the potential of LLMs to extract high-quality clinical endpoints at scale, helping to advance research, enhance applications such as predictive algorithms, and ultimately supporting more personalized and effective cancer care. Citation Format: Aaron B. Cohen, Konstantin Krismer, Kelly Magee, James Gippetti, Aaron Dolor, Tori Williams, Erin Fidyk, Hank Kim, Qianyu Yuan, Melissa Estevez. Using large language models for scalable extraction of real-world progression events across multiple cancer types [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B006.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.