Abstract A050: Multimodal integration of H&E slides and matched targeted DNA sequencing data for enhanced cancer subtype identification

Abstract Determining a cancer’s site of origin is essential for providing effective patient treatment, but it can be difficult if the cancer initially presents as poorly differentiated, as metastatic, or as a Cancer of Unknown Primary (CUP). Previous machine learning models have used clinical, pathological or genomic data to infer cancer types, but there has been limited work to integrate these data modalities for tumor type inference. We hypothesize that a multimodal approach enhances inference of histologic subtype. To develop this approach, we harness two pre-existing unimodal deep learning models and retrain each on a pan-cancer cohort of 40,888 tumor samples, where each sample had both hematoxylin and eosin (H&E) whole-slide images (WSIs) and matched targeted DNA sequencing data (MSK-IMPACT) available. For the H&E WSIs, we use transformer model AEON (Adaptive Embedding Ontology Network), and for the MSK-IMPACT data we use hyper-parameter ensemble model GDD-ENS (Genome-Derived-Diagnosis Ensemble), to infer 110 distinct cancer subtypes as defined by their OncoTree code. ODEN (Oncotree-Diagnosis ENsemble), our multi-modal approach, combines the final probability output layers of both models using a weighted average corresponding to model training set accuracy, where inferred type represents the highest probability subtype after re-weighting. Both models achieved high weighted macro average area under the receiver operating characteristic curve (AUROC) values on a held-out test set of 11,158 samples (AEON .988, GDD-ENS .963), and good top-1 (and top-3) accuracy scores of 68.8 (89.1) for AEON, and 64.5 (81.8) for GDD-ENS. 50% of samples were correctly inferred by both models, 20% by AEON only, and 16% by GDD-ENS only, indicating the models are complementary and multi-modal integration could improve performance. ODEN AUROC was slightly higher at .990, but accuracy greatly improved to reach 77.8 (92.3). On a subtype-specific basis, most had greater than or equal precision in ODEN when compared to GDD-ENS (99/110), or AEON (90/110). Next, ODEN was applied to a set of 5,531 samples with underspecified labels, e.g., BRCANOS or SARCNOS. 88% of ODEN inferences were in the correct organ system for the underspecified subtype (80% in AEON, 78% GDD-ENS) with highest recall within the core GI (96%) and genitourinary (93%) systems. We also evaluated 1,749 CUP samples, and found that ODEN inferences spanned 100 different subtypes, most commonly LUAD (n = 216) or PAAD (n = 210). The ODEN-inferred subtypes showed similar genomic and prognostic trends when compared to true metastatic samples of each subtype. Overall, ODEN is a multimodal tumor type inference model that improves upon prior models trained on fewer types, samples and modalities. As H&E assessment is common clinical practice and DNA sequencing data is routinely collected for all patients in our institution (and rapidly expanding to others), widespread practical integration of ODEN is clinically feasible and could enable multimodal patient-specific subtype inference for diagnostically challenging cases. Citation Format: Madison Darmofal, Kevin Boehm, Andrew Aukerman, Arfath Pasha, Armaan Kohli, Raymond Lim, Tom Pollard, Darin Moore, Michael Berger, Nikolaus Schultz, Sohrab P. Shah, Francisco Sanchez-Vega. Multimodal integration of H&E slides and matched targeted DNA sequencing data for enhanced cancer subtype identification [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr A050.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.