[ICLR 2025] How LLMs Compute Verbal Confidence: It’s Cached, Not Just-in-Time

TL;DR

When you ask an AI, "How confident are you in this answer?", does it rethink the whole problem (Just-in-Time), or does it already have a "gut feeling" stored away (Cached Retrieval)? This paper provides definitive mechanistic evidence that LLMs like Gemma 3 and Qwen 2.5 automatically compute confidence while they are writing the answer, cache that signal at a specific hidden position (the newline token), and simply read it out when prompted. This confirms that LLMs possess a "second-order" metacognitive capacity—they evaluate the quality of their thoughts in parallel with generating them.

The "Just-in-Time" vs. "Cached" Debate

In human neuroscience, there is a classic debate: do we know we are confident because the decision was "easy" (First-order), or does a separate part of the brain monitor the decision-making process (Second-order)?

For LLMs, the question translates to:

1. Just-in-Time (JIT): The model sees the prompt "Confidence: " and looks back at the question and answer to calculate a score from scratch.
2. Cached Retrieval: The model realizes it's struggling (or succeeding) during generation. It "writes down" this internal state in its hidden activations (the residual stream) and later retrieves it.

The authors find that the Cached Retrieval hypothesis is the winner.

Methodology: Peering into the Residual Stream

To prove this, the team used a series of "surgical" interventions on the model's internal activations:

1. Activation Steering & Patching

By extracting a "confidence vector" (the difference between 'Almost Certain' and 'No Chance' activations), researchers could "steer" the model. Injecting a "high confidence" vector at the Post-Answer-Newline (PANL) token—the invisible token right after the answer—caused the model to report higher confidence later, even if the answer was wrong.

2. The Pathway of Information

Through Attention Blocking, the authors mapped the "circuit" of confidence:

• Phase 1: Confidence is gathered from answer tokens.
• Phase 2: It is cached at the PANL token (Layers 21–25).
• Phase 3: It is retrieved by the final colon (":") token (Layers 30–35) for verbalization.

Figure 1: The information flow from Answer -> PANL -> Confidence Output. Blocking the JIT path (red) had no effect, while blocking the Cache path (purple) disrupted the model.

Key Finding: It's More Than Just Fluency

One of the most profound "Why" questions in this paper is: Is verbal confidence just a proxy for how "smoothly" the tokens were generated? (i.e., log-probabilities).

The researchers used Variance Partitioning to answer this. If verbal confidence were just log-probs, then a probe trained on internal activations shouldn't be able to tell us anything new. However, the internal PANL activations explained significantly more variance in the final confidence score than the log-probs did.

This means the model is performing a "Second-Order" evaluation: it knows more about its own correctness than the surface-level token probabilities suggest.

Figure 2: Linear Probing results showing that confidence information is decodable at the PANL token much earlier than at the final reporting site.

Why This Matters for the Future

1. Calibration without Probabilities: Since we now know where the confidence lives (the PANL token), we can extract better uncertainty estimates from "black-box" models by probing their internal states, even if we don't have access to their output log-probs.
2. Error Detection: Because this is a second-order signal, LLMs might be capable of "knowing they made a mistake" immediately after the last token of an answer is generated, but before they start explaining themselves.
3. Against Post-hoc Rationalization: This study proves that confidence isn't just "made up" to satisfy the prompt; it's a reflection of an actual internal state computed during the "thought process."

Critical Analysis & Limitations

While the evidence for cached retrieval is strong, its generality is still being tested. The authors focused on TriviaQA (factual recall). How does this work for multi-step math or creative writing? Additionally, while the model computes this signal, it isn't always perfectly calibrated. The "internal" knowledge of the model is often more accurate than what it actually "says" in the chat box—a gap that future Research can bridge using the steering methods introduced here.

Conclusion

The "gut feeling" of an LLM is a real, measurable, and localized phenomenon. By identifying the PANL token as the "metacognitive hub" for confidence, this paper opens new doors for making AI more honest and self-aware.