Formalizing Latent Thoughts: Why Your LLM Isn't "Thinking" the Way You Think It Is
Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs
This paper introduces an axiomatic evaluation framework for "latent thoughts" in Large Language Models (LLMs). By formalizing four functional axioms—Causality, Minimality, Separability, and Stability—the authors audit various continuous reasoning methods (like Soft Thinking and Latent Thinking) independently of downstream benchmark scores, revealing critical representational failures.
TL;DR
As LLM research shifts from explicit Chain-of-Thought (CoT) text to continuous "latent thoughts" (vectors), we’ve lacked a way to measure if these vectors actually mean anything. This paper introduces a rigorous 4-axiom framework to audit these thoughts. The verdict? Current "latent thinking" methods often fail to encode more information than the initial prompt and suffer a "representational collapse" where they can't tell two similar questions apart, even if they get the answers right.
The "Accuracy Paradox" in LLM Reasoning
In the quest for faster and more powerful reasoning, researchers have started replacing readable "Thinking..." text with compressed hidden vectors. Usually, if the model’s accuracy goes up, we assume the "thought" was good.
However, the authors of this paper argue that accuracy is a lying metric. A model might guess the right answer based on task-level patterns while its internal "thought" representation is a mess. We have been missing a way to evaluate the representation as an object, independent of whether the final answer happened to be "A" or "B".
The Four Axioms of Functional Thought
The authors propose that for a latent vector to be a "thought," it must behave according to four functional requirements:
- Causality: If you swap the original reasoning text for the vector in the model's brain, the output should stay the same.
- Minimality: The thought should compress the input by throwing away useless noise while keeping everything needed for the answer.
- Separability: The representation must be distinct enough to tell different questions apart.
- Stability: The thought should be robust to surface-level changes (like paraphrasing) and reflect the model's internal uncertainty.
Methodology: Auditing the "Black Box"
The team audited several popular methods, including:
- Soft Thinking (ST): Using weighted averages of token embeddings.
- Latent Thinking (LT): Iteratively updating hidden states.
- Last Input Token (LIT): The standard hidden state used for prediction.
They tested these across 23 reasoning tasks (Big Bench Extra Hard) on models like Llama-3.3-70B and DeepSeek-R1-Distill. To measure these axioms without retraining the models, they used a "frozen" backbone model as a probe to see what information could be extracted from these latent vectors.
Key Findings: The Representational Collapse
The results provide a sobering "reality check" for the field of continuous reasoning.
1. The Within-Task Identity Crisis
Current latent representations are great at identifying what kind of task they are doing (e.g., "this is a math problem"), but they are terrible at identifying the specific question. In "Same-Task" discrimination tests, most methods were essentially guessing at random.
2. Thinking More, Knowing Less?
Perhaps the most shocking finding is that increasing "thinking steps" often makes the representation worse. While we hope that more iterations lead to deeper thoughts, the evidence shows that it often leads to "drift," where the vectors lose touch with the specific input question.
3. The Input Embedding Floor
In almost every test, the simple Input Embedding (IE)—the vector formed just by looking at the prompt—was as good as or better than the complex "thinking" vectors produced after 128 steps of computation.
Figure: The "Separability" collapse across different models. Note how almost all candidates (except the direct Output Embedding) fail to move the needle on the Within-Task axis.
Why is this happening?
The authors suggest a structural collapse. In the high-dimensional space of an LLM, the directions that differentiate one question from another within a task are very narrow. As the model "thinks," these specific details are drowned out by task-level noise or "superposition," where different reasoning paths become indistinguishable.
Critical Insight & Future Outlook
This paper serves as a vital "guardrail" for the AI community. It proves that we cannot trust benchmark scores alone to validate internal reasoning mechanisms.
The Takeaway: If we want models that truly "think" in a latent space, we need to train them specifically to satisfy these axioms—forcing them to maintain identity (Separability) and causal power (Causality) rather than just hoping these properties emerge from RLHF or distillation.
Limitations: The study focuses on reasoning tasks; it’s possible that for creative writing or conversational tasks, these axioms (like Minimality) might need to be relaxed. Furthermore, the audit only covers "training-free" extraction. The next frontier will be using these four axioms as loss functions to train a new generation of models that "think" with mathematical rigour.
Code and Framework available at: https://fard-lab.github.io/formalize-thoughts