Abstract B015: Large language models to predict cancer risk from free-text clinical notes
Bibliographic record
Abstract
Abstract Background: Accurate predictions of future cancer risk can increase early detection through selecting high risk individuals for screening. Existing risk prediction tools have limited predictive performance and are underutilized due to workflow disruption and reliance on structured data. We investigated whether large language models (LLMs) can predict cancer risk directly from routine free-text clinical notes recorded by primary care physicians. Methods: The dataset used for this study was from individuals aged over 18 years living in Ontario (2010–2016), obtained from ICES. ICES is an independent, non-profit research institute whose legal status under Ontario’s health information privacy law allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement. The development dataset consisted of 1,080 individuals selected from a cohort of 109,378 patients in Southern Ontario using stratified sampling. The testing dataset included 1,080 individuals from 135,894 patients in Toronto. A pipeline was developed using source-available LLMs to estimate cancer risk. Prompt and sampling methods were optimized on the development dataset. Prompts were designed to elicit probabilistic cancer risk estimates from progress notes. The study employed a fixed exclusion window of 365 days prior to cancer diagnosis and evaluated performance across lookahead windows of up to ten years. Lung cancer diagnosis was used as a case example to optimize the system, which was tested against the current screening eligibility guidelines based on age and pack-years of smoking. We evaluated generalizability to other geographic regions (1,080 individuals from 16,130 patients in Northern Ontario) and adapted the prompting pipeline for other cancer types (200 individuals from Southern Ontario). Results: The best-performing model was a Mistral Small model using a multiple prompting strategy (chain-of-thought, emotion, persona). The model achieved an area under the receiver operating characteristic curve (AUROC) of 0.787 (95% CI 0.749 - 0.824) in the testing dataset in Toronto for lung cancer prediction over a 10 year lookahead period. At the equivalent specificity to current lung cancer screening eligibility (0.941, 95% CI 0.920-0.961), predictions from the system had increased sensitivity (0.421 [95% CI 0.362-0.479] compared to 0.207 [95% CI 0.158-0.254], p<0.05). The system generalized to Northern Ontario (AUROC 0.807, 95% CI 0.771-0.844). When extended to other cancer types, the system achieved AUROC of 0.780 (95% CI 0.702–0.841) for pancreatic cancer and 0.729 (95% CI: 0.728–0.730) for prediction of any cancer diagnosis. Conclusions: LLMs can predict future cancer risk directly from routine clinical progress notes, offering a scalable, privacy-preserving, and generalizable alternative to structured-data-based models. This approach could be readily integrated into clinical workflows to improve cancer screening and early detection. Citation Format: Daniel Mau, Karl Everett, Ning Liu, Jason Chai-Onn, Liisa Jaakkimainen, Anna Dodd, Spring Holter, Steven Gallinger, Rahul G. Krishnan, Kelvin Chan, Robert Grant. Large language models to predict cancer risk from free-text clinical notes [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B015.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.015 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.005 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".