Abstract B012: Prompting Large Language Models to Predict Adverse Events during Cancer Treatment

Abstract Background: Large language models (LLMs) excel on standardized oncology exams; however, their broader clinical utility remains unclear. LLMs are easy to use through “prompting.” For example, a doctor or patient can provide a clinical note and ask about the probability of an adverse event (AE). Current AE prediction relies on machine learning using tabular data, which requires substantial engineering to adapt to specific tasks and settings, making them costly and less generalizable. We compared prompting LLMs against tabular ML models to predict AEs during systemic cancer therapy. Materials an. Methods: Patients with aerodigestive cancers at Princess Margaret Cancer Centre who received their first systemic therapy from 2008 to 2015 formed the development set, and from 2016 to 2018 formed the test set. We evaluated different prompting strategies with open-source LLMs using the de-identified consult and most recent pre-treatment note from each patient to predict the risk of clinical, symptom, and laboratory AEs. An ensemble of ML models was trained on tabular electronic health record data for comparison. We measured performance with the area under the receiver-operating characteristic curve (AUC). Using an established schema, an oncologist reviewed the text-based justifications from 20 random LLM predictions. Results The cohort included 6,381 patients. Notes had a median token length of 1,737 (range 137-7,795). The LLM Qwen 2.5 14B achieved the best AUC across 14 of 19 AEs in the development set. The larger 14B model outperformed the 7B model on all targets (p = 4e-5). Among prompting strategies, no benefit was observed with the oncologist versus AI model persona (p = 0.21), chain-of-thought reasoning (p = 0.23), or concatenating tabular data to notes (p = 0.42). In the test cohort, LLMs and tabular ML showed equivalent performance for some AEs, such as death within 30 days (LLM AUC: 0.73 [95% CI 0.66, 0.80], versus [v.] ML: 0.74 [0.67, 0.81], p = 0.89) and hyperbilirubinemia (0.79 [0.72, 0.86] v. 0.78 [0.70, 0.85], p = 0.77). For other AEs, performance was numerically similar, such as death in one year (0.72 [0.70, 0.74] v. 0.76 [0.73, 0.78], p = 0.02) and anemia (0.78 [0.75, 0.80] v. 0.82 [0.8, 0.84], p = 0.01). LLMs performed worse for symptom-related AEs, such as pain (0.48 [0.44, 0.53] v. 0.69 [0.65, 0.74], p = 1e-11) and tiredness (0.49 [0.45, 0.52] v. 0.69 [0.65, 0.72], p = 2e-14). The oncologist deemed LLM justifications satisfactory across all dimensions for at least 90% of predictions, except that 20% had factual consistency errors. Conclusion: Prompting LLMs performed similarly to engineered tabular ML models for predicting several AEs, despite using only raw text from notes. Better performance with larger models suggests the gap between LLMs and ML models may continue to narrow. This work lays the foundation for using LLMs as general-purpose clinical decision-support tools for cancer care. Citation Format: Wayne Isaac T. Uy, Galileo Arturo Gonzalez Conchas, Jiang Chen He, Muammar Kabir, Baijiang Yuan, Geoffrey Liu, Sharon Narine, Melanie Powis, Benjamin Grant, Mattea Welch, Tran Truong, Robert Grant. Prompting Large Language Models to Predict Adverse Events during Cancer Treatment [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B012.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.