Multi-view deep learning of highly multiplexed imaging data improves association of cell states with clinical outcomes
Bibliographic record
Abstract
Abstract Highly multiplexed imaging technologies can quantify the expression of 10s-100s of proteins and their modifications at sub-cellular resolution. Applications to tissue specimens across a broad array of pathologies have enabled patient subtyping in terms of novel cell states and spatial cell communities and linked their prevalence and overall architecture to patient outcomes. While highly multiplexed imaging analysis workflows typically summarize each cell in terms of its post-segmentation mean expression, additional cellular information can be quantified including cell morphology, sub-cellular expression patterns, and spatial cellular context, ultimately giving a multi-modal view of each cell. While deep latent variable models such as variational autoencoders (VAEs) are well established for other multi-modal single-cell assays, their ability to integrate these multiple views of a cell from highly multiplexed imaging data remains largely unknown. Here, we explore the abilities of multi-modal VAEs to learn unified latent cellular representations from multiple views of each single-cell quantified from highly multiplexed imaging, including mean expression, morphology, sub-cellular protein co-localization, and spatial cellular context, while conditioning on technical and batch specific effects. We first discuss strategies for training and hyperparameter optimization of such models across a set of highly multiplexed imaging datasets of breast and melanoma cancers. We next quantify the relevance of the integrated multi-modal latent space in predicting patient-specific clinical outcomes and demonstrate competitive performance across a range of outcomes compared to existing baselines. Then, we explore the ability of the cellular representations to learn cellular phenotypes that align with known cell types and find that the expression-specific latent representation can identify clusters more closely aligned with known lymphocyte and epithelial subpopulations compared to established workflows. Finally, we test the ability of each representation to perform consistent batch integration across different cohorts and discuss the considerations on taking known background variables into account.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".