Haiku: Bridging Spatial Biology, Histology, and Clinical Context via Tri-modal Alignment
Linking spatial biology and clinical histology via Haiku
Haiku is a tri-modal contrastive learning foundation model that unifies H&E histology, multiplexed immunofluorescence (mIF), and clinical text into a shared embedding space. Trained on 26.7 million patches, it achieves SOTA cross-modal retrieval (Recall@50 up to 0.611) and enhances clinical tasks like survival prediction (C-index 0.737).
TL;DR
Researchers have developed Haiku, a foundation model that finally connects the "three pillars" of pathology: H&E morphology, spatial proteomics (mIF), and clinical metadata. By aligning these into a shared latent space, Haiku allows users to retrieve molecular maps from simple H&E images and perform "counterfactual" experiments—such as asking, "How would this specific tissue niche change if the patient survived?"—without traditional lab staining.
Background: The Multimodal Gap in Pathology
In modern oncology, we often have three types of data:
- H&E Slides: Cheap, ubiquitous, but molecularly "opaque."
- mIF/Spatial Omics: Highly detailed molecular maps, but expensive and rare.
- Clinical Metadata: The patient's "story" (survival, stage, treatment response).
Until now, AI models mostly looked at these in isolation. You either had an H&E model or a proteomics model. Haiku breaks these silos by treating them as three different "languages" describing the same biological reality.
Methodology: The Core of Tri-modal Alignment
Haiku uses a Contrastive Learning framework (similar to OpenAI's CLIP) but extended to three modalities.
1. The Encoders
- Histology (H&E): Uses the MUSK Vision Transformer.
- Spatial Proteomics (mIF): Uses a VirTues architecture, uniquely augmented with ESM-3 protein embeddings to help the model understand the "biological meaning" of different protein channels.
- Text (Clinical/Semantic): Uses BiomedBERT to process structured clinical templates and biomarker descriptions.
2. The Shared Space
The model was trained on 26.7 million patches from over 1,600 patients. The goal is simple: for a single piece of tissue, the H&E embedding, the mIF embedding, and the clinical text embedding should all be represented by nearly identical vectors in a 512-dimensional space.
Figure 1: The Haiku architecture aligns three distinct modalities using modality-specific projection heads and contrastive loss.
Experimental Results: Retrieval and Prediction
Haiku doesn't just "cluster" data; it enables Cross-Modal Reasoning.
High-Fidelity Retrieval
Haiku achieves a Recall@50 of 0.611 for H&E-to-mIF retrieval. In plain English, if you give Haiku an H&E patch, it can accurately find the corresponding molecular profile from a library of millions. This outperforms naive RGB-stacking baselines which show near-zero performance.
Clinical SOTA
Because it learns from molecular-level detail, its "vision" of H&E is more acute.
- Survival Prediction (CRC): Achieved a C-index of 0.737, a significant jump over unimodal baselines (+7.91% relative improvement).
- Treatment Response: In melanoma cohorts, it reached an AUROC of 0.756, proving that its learned features are highly "transferable" to real-world clinical outcomes.
Figure 2: Qualitative and quantitative evidence of Haiku's superior cross-modality retrieval and zero-shot capabilities.
The "Counterfactual" Paradigm: In Silico Biology
One of the most exciting features is the Counterfactual Prediction framework.
The Workflow:
- Take an H&E slide from a patient who died (Original).
- Keep the image the same but change the text description to "Alive" (Counterfactual).
- See which mIF patches Haiku retrieves now.
In a lung cancer case study, when Haiku "imagined" a survival outcome, the retrieved molecular profiles showed increased CD8 T-cells and reduced PD-L1—patterns widely recognized in literature as markers of favorable prognosis. This allows researchers to generate biological hypotheses about which molecular components in a specific tissue niche might be driving survival.
Critical Insight & Conclusion
Haiku proves the Platonic Representation Hypothesis in medicine: as we train models on more "views" of biology (morphology, proteomics, clinical context), they converge on a single, more accurate truth of the disease.
Limitations: Currently, the model works at the patch level (256x256 pixels). To truly revolutionize clinical workflows, the next step is integrating this into Whole Slide Image (WSI) architectures to capture the macro-architecture of the tumor microenvironment.
Summary: Haiku isn't just a classifier; it’s a bridge. It transforms clinical images into molecular discovery tools, grounding its "hallucinations" in real, evidence-backed mIF data.