Abstract B053: Empowering AI-driven prediction of the tumor microenvironment from histopathology images via molecular annotation

Abstract The tumor microenvironment (TME) actively contribute to tumor development and treatment response. The interplay between tumor cells, immune cells, fibroblasts and blood vessels contribute to immune escape and drug resistance. Prior to treatment, higher tumor infiltrating lymphocytes correlate with better survival, while greater stromal content is linked to poor survival. Studying the composition and dynamics of the TME is essential for improving patient stratification, however a scalable tool for addressing this question is still lacking. Spatially resolved omics technologies allow for charting tissue architecture at the individual cell level, though large-scale studies remain challenging due to high expenses. In contrast, hematoxylin and eosin (H&E) slides are a cost-effective modality that provide rich morphological information for studying spatial biology. However, their use relies on pathologist interpretation. A key area of research in digital pathology has been automating cell (type) identification, predicting nuclei location and cell type in H&E slides. Existing deep learning models are limited by the quantity and diversity of training data, which requires pathologists to carefully annotate the location and identity of large volume of cells. To date, the largest dataset comprises approximately 200,000 cells, annotated with four cell types across 19 cancer types. We propose a novel approach of automated “molecular annotation”, where cell types and location on H&E slides are annotated with the aid of spatial proteomics, in place of pathologist annotation. Specifically, samples were profiled with both modalities. From the spatial omics modality, pixels were first segmented into cells, followed by cell clustering and cluster annotation based on molecular features of the cells. The location and identity of all cells on the tissue were then identified. These information were subsequently transferred to the H&E image by alignment at single-cell resolution, forming a dataset annotated with molecular ground truth. With a spatial omics dataset of two spatial proteomics slides from colorectal cancer patients, we obtained 160,000 annotated cells including 50,000 immune cells, 25,000 tumor cells, 12,600 stroma, 7,900 endothelial cells, among others. The size of this dataset is close to the largest annotated dataset publicly available to date. We then use this dataset to benchmark existing state-of-the-art deep-learning based cell type predictions models, as well as to fine-tune existing models for predicting cells in the colorectal tumor microenvironment. This proof-of-concept study aims to demonstrate the feasibility of molecular annotation approach. By including more spatial omics data, this approach can boost the performance of existing pre-trained models and enhance generalizability to specific tumor types. It opens up the opportunity to harness millions of cells for deep learning models to predict cell types on H&E slides, make AI models a cost-effective option for studying the TME. Citation Format: Siao-Han Wong, Benedikt Brors, Sonja Loges. Empowering AI-driven prediction of the tumor microenvironment from histopathology images via molecular annotation [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr B053.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.