Abstract B019: Clinician-AI evaluation of prognostic information extraction in head and neck cancer using an on-premises LLM

Abstract Introduction: Head and neck (H&N) cancers involve diverse anatomical sites, histologies, risk factors, and outcomes. Prognostic data like TNM staging, p16/HPV status, smoking, alcohol use, and comorbidities, are often buried in unstructured electronic health record (EHR) notes. Manual extraction is labor-intensive, limiting efficiency in tumor board preparation, trial screening, and personalized care. Traditional rule-based natural language processing struggles with long, domain-specific clinical text. While LLMs show promise, proprietary cloud models like GPT-4 raise privacy concerns as data must leave secure hospital systems. In Canada, such transfers pose legal risks. Prior work has focused on public or simulated data; granular extraction from real oncology notes remains underexplored. We address this by introducing a de-identified H&N dataset and deploying an on-premises open-source LLM (Llama3.3-70B). We hypothesize that this local model can match expert abstraction accuracy while preserving privacy. Methods: We conducted a retrospective study at a tertiary center with ethics approval, including 1,360 pathology and consultation notes from 882 biopsy-confirmed patients (2010–2023). The LLM was deployed locally on secure hospital hardware via Ollama with no external access, running on 2 NVIDIA RTX 5000 GPUs. Thirty prognostic fields were extracted using Python and LangChain with direct and inference prompts. Fifty cases were selected through stratified sampling for clinician review. Three blinded oncologists independently rated model outputs as “Agree” or “Disagree.” Disagreements were resolved by majority vote. We calculated accuracy, precision, recall, and F1 scores. Results: Each note was processed in 30–50 seconds, versus 5–30 minutes manually, yielding up to 20× efficiency gains. Across 4,500 evaluations, the model achieved 97.7% accuracy, 97.1% precision, 99.9% recall, and 98.5% F1. Of 30 fields, 20 achieved perfect scores (100%), including lesion site, p16 status, smoking history, treatment recommendations, imaging interpretations, resection margins, Eastern Cooperative Oncology Group (ECOG) Performance Status/Karnofsky Performance Scale, prior treatments, and follow-up. Eight fields maintained >95% performance, including tumor type (93.2% accuracy) and TNM classification (97.7%). The Charlson score showed lower precision (67.4%) due to overestimation but retained 100% recall. Conclusion: A locally deployed LLM extracted prognostic data from oncology notes with near-perfect accuracy across 28 of 30 fields while maintaining privacy. Charlson score overestimation, e.g., hypertension or anxiety, misclassified as severe, highlighted reasoning limitations. Improved prompting and adherence to validated criteria may reduce false positives. This pilot Canadian study demonstrates that privacy-preserving LLMS can streamline tumor board prep, support AI-assisted decision-making, and scale oncology data extraction in compliance with strict data regulations. Citation Format: Yujing Zou, Laya Raifiee. Sevyeri, Farhood Farahnak, George Shenouda, Marie Duclos, Tomás Yokoo Teodoro. de Souza, Khalil Sultanem, Parsa Bagherzadeh, Farhad Maleki, Shirin A. Enger. Clinician-AI evaluation of prognostic information extraction in head and neck cancer using an on-premises LLM [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B019.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.