CoLT: Accelerating MLLM Reasoning with Chain of Latent Thoughts

TL;DR

The "Reasoning Era" of AI, epitomized by models like DeepSeek-R1 and OpenAI o1, relies on massive text-based Chain-of-Thought (CoT). However, CoLT (Chain of Latent Thoughts) demonstrates that models don't need to "talk to themselves" in text to think. By replacing hundreds of text tokens with just 3 latent thought vectors, CoLT achieves a 10.1x speedup while actually surpassing text-based reasoning accuracy on complex multi-modal benchmarks.

The Bottleneck of "Thinking out Loud"

Standard Multi-modal Large Language Models (MLLMs) are currently trapped in a "textual reasoning bottleneck." While generating explicit reasoning steps helps solve complex math or visual logic problems, it introduces three fatal flaws:

1. Extreme Latency: Generated chains can span thousands of tokens, making real-time inference impossible.
2. Deterministic Collapse: Once a word is chosen, the model's internal probability distribution collapses, limiting alternative reasoning paths.
3. Error Propagation: A single early mistake in the text chain often leads to a "hallucination spiral."

Previous "Latent Reasoning" attempts often failed because unconstrained continuous vectors are prone to semantic drift, making them "meaningless noise" to the model.

Methodology: The Three Pillars of Latent Supervision

CoLT's breakthrough lies in how it regulates the latent space during training without adding overhead during inference. The authors introduce a lightweight External Decoder (e.g., a 0.6B Qwen model) to provide three specific types of training signals:

1. External Decoder Supervision

• Forward Mode: The model is forced to ensure its latent thought $h_k$ contains enough information to reconstruct the next textual reasoning step.
• Backward Mode: It aligns the latent states with the hidden states produced when the decoder "reads" the ground-truth reasoning text, anchoring the latent space to human logic.

2. Internal Step-to-Step Consistency

To ensure the "thought" evolves logically, an internal MLP profection head predicts the next latent state from the current one, enforcing a coherent transition across the chain.

Figure 1: CoLT architecture vs. traditional paradigms. Note how the external decoder is removed at inference time for maximum efficiency.

Experimental Performance: Faster AND Smarter

The results across 8 benchmarks (SeedBench, ChartQA, etc.) are striking. CoLT doesn't just match the "Slow Thinking" of text CoT; it beats it.

• Efficiency: On MMStar, CoLT reduces generation time from 7.24s (Text CoT) to just 0.32s.
• Accuracy: CoLT achieves a +3.4% average improvement over Text CoT. In data-heavy tasks like ChartQA, the gain is a massive +9.6%.
• Robustness: Unlike text CoT, which breaks when input text has typos or images are noisy, CoLT’s continuous vectors are more resilient to perturbations.

Table 1: Performance comparison. CoLT (bottom row) consistently outperforms both text-based and visual-latent baselines.

Visualizing the "Invisible" Thought

Does the model really "think"? By using the forward decoder to project the 3 latent steps back into text, the authors show that each step focuses on a distinct logical phase. For instance, in a pattern-matching task, Step 1 identifies the rows, Step 2 analyzes the green slices, and Step 3 computes the final missing piece.

Figure 2: Interpreting latent thoughts by decoding them back into text. Each color represents a different latent step's contribution.

Critical Analysis & Conclusion

CoLT represents a significant shift from "Scaling Laws via Token Length" to "Scaling Laws via Latent Depth."

Strengths:

• Eliminates the need for expensive "Latent Visual Reasoning" annotations.
• Achieves massive speedups without sacrificing (and even improving) accuracy.

Limitations:

• The optimal number of steps ($K=3$) was determined empirically. Very complex "System 2" tasks might still require more steps or an adaptive mechanism.

Future Work: The most exciting path forward is Adaptive Latency—allowing the model to decide whether a problem requires 1 "thought" or 100, dynamically adjusting its reasoning time based on difficulty.