Abstract A049: Development of a tri-modal contrast learning model integrating pathology-text and CT-text for clinical oncology tasks
Bibliographic record
Abstract
Abstract This study aims to develop a path-image-text tri-modal representation learning framework (Tri-MCR) without paired data by integrating pathology-text and CT-text models to improve the performance of clinical cancer tasks. Aiming at the challenge of scarce multi-modal data pairing in the cancer field, we project the pre-trained pathology-text and CT-text models into a shared semantic space by using text (i.e., Electronic Health Record) as an intermediate modality and optimize the cross-modal alignment by using a semantic enhancement strategy. Tri-MCR projects pre-trained pathology-text and CT-text models into a shared semantic space, leveraging text as an intermediary modality. Cross-modal alignment is optimized via a semantic enhancement strategy involving two key components: (1) Dynamic Semantic Enhancement: Gaussian noise injection and cross-modal attention mechanisms are employed to dynamically aggregate text-guided pathology and radiology features, thereby enhancing the semantic integrity of the embeddings. (2) Dual-Alignment Strategy: Cross-modal contrastive loss enforces semantic consistency between pathology-text and CT-text representations within the shared space, while intra-modal contrastive loss mitigates representation shifts between pathology and CT modalities. Furthermore, an Interpretability Mapping technique visualizes pathology-CT-text semantic associations through a cross-modal similarity matrix, offering biological insights for clinical decision-making. In the validation of the colorectal cancer single-center cohort (N=1015) of the ChangKang (Healthy Bowel) project, Tri-MCR significantly outperforms the baseline model in tumor biomarker prediction and mortality risk prediction tasks. This method provides a new idea for efficient representation learning of tumor multimodal data, provides an interpretable multimodal analysis tool for tumor precision diagnosis and treatment, and reduces the dependence on large-scale paired data, which has important clinical application value. In the future, further model multi-task performance evaluation and reliability verification in multi-center and multi-cancer cohorts are considered. Citation Format: Zhicheng Du, Hui-Yan Luo, Lijin Lian, Vijay Kumar. Pandey, Jiansong Ji, Peiwu Qin. Development of a tri-modal contrast learning model integrating pathology-text and CT-text for clinical oncology tasks [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr A049.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.004 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.002 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.004 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".