[COLM 2025] The Illusion of Superposition? Why Your LLM Isn't Actually Thinking in Parallel
The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
This paper investigates "latent superposition" in Language Models (LMs)—the hypothesized ability to process multiple reasoning paths simultaneously in continuous space. Through a principled analysis of training-free (Soft Thinking), fine-tuned (Coconut), and from-scratch models, the authors demonstrate that superposition is remarkably brittle and often a "thinking illusion" in standard LLMs.
TL;DR
The community has long hypothesized that "Latent Chain-of-Thought" (reasoning in continuous vector space) allows models to perform superposition—maintaining multiple reasoning paths at once. This paper performs a "reality check" and finds that for most LLMs, this is an illusion. Unless a model is tiny and trained from scratch, it either ignores the latent "thoughts" or collapses them into a single discrete token almost immediately.
Context: This is a critical "negative results" and interpretability paper that challenges the scaling laws of latent reasoning, identifying why current fine-tuning recipes for "thinking" models might be failing to produce a qualitative leap in logic.
The "Superposition" Dream vs. Reality
In discrete CoT (like OpenAI's o1), the model must commit to one word at a time. Latent CoT suggests we can keep the "thought" continuous. Theoretically, if a model isn't sure if the answer is "Apple" or "Banana," it could carry a vector that is 50% of both (Superposition).
However, the authors found that pretraining is a curse for superposition. Because LLMs are trained to predict a single discrete next token, they develop a "commitment bias." Even if you feed them a soft mixture of tokens, the internal layers quickly "crush" that mixture back into a single interpretation.
Methodology: Peeking into the "Black Box" of Thoughts
The authors investigated three types of models:
- Soft Thinking (Training-free): Mixing embeddings at the input.
- Coconut (Fine-tuned): Tuning a model to output continuous states.
- From-Scratch: Training a small GPT-2 style model specifically for latent reasoning.
They utilized Logit Lens (projecting hidden states back to the vocabulary at every layer) and Entity Probing to see if "Step 1" thoughts actually contained "Step 1" information.
Figure 1: Comparison of Coconut (Recurrent Latent) and Soft Thinking (Convex Combination).
Key Finding 1: The Entropy Collapse
In off-the-shelf models like QwQ-32B, the entropy—a measure of uncertainty/diversity in the representation—drops to near zero at the final layers. This means the model "decides" on a specific token early on and ignores the superposed information.
Figure 2: Entropy profiles for Soft Thinking vs. Discrete CoT are nearly identical, proving no extra "parallel exploration" is happening.
Key Finding 2: The Shortcut Scandal
The authors analyzed Coconut, a state-of-the-art latent reasoning method. Using entity-level probing, they discovered something shocking: the latent tokens were largely useless.
- In a 5-hop reasoning task, the Coconut model already "knew" the final target answer at Step 0.
- The model achieves 96.6% accuracy even if you delete the "thinking" tokens.
- It wasn't reasoning step-by-step; it was finding the answer instantly and using the latent steps as an "echo chamber."
Figure 3: Coconut (Left) predicts the target immediately. Discrete CoT (Right) actually moves through the hops.
Why is Superposition so Brittle?
The authors identify two main executioners of superposition:
- Pretraining Bias: Next-token prediction optimizes for separation, not mixing. In Figure 5 of the paper, they show that randomly initialized models keep entropy high, but pretrained models collapse it.
- Capacity Overload: Paradoxically, larger models are worse at superposition. When a model has many layers (8+), it is "smart" enough to find a direct shortcut from the question to the answer, bypassing the need to maintain complex parallel reasoning paths in its latent space.
Deep Insight & Future Outlook
This work suggests we cannot simply "fine-tune" our way into better latent reasoning if we use the same old LLM backbones. Total superposition likely requires:
- Bottlenecked Architectures: Forcing the model to use latent states because it doesn't have enough "instant horsepower" to solve the problem in one pass.
- New Objectives: Moving beyond next-token prediction to objectives that reward maintaining uncertainty.
Conclusion: Latent reasoning's true power might not be "Parallel Thinking" (superposition) but rather "Flexible Thinking"—the ability to express concepts that don't fit into human words. However, as it stands, the "superposition" in today's models is mostly an illusion.