What would a fair evaluation of recurrent latent reasoning for small models need to measure?

Why final accuracy alone is a misleading yardstick

If you only compare final scores, you might miss what latent reasoning is actually good at. The whole point of latent reasoning is to get the benefits of chain-of-thought (CoT) without generating long, token-by-token rationales—so a fair evaluation must measure inference cost and latency, not just accuracy. In one study, Thinking States—a method that generates thinking tokens while the input is being processed—matched CoT on 2-hop question answering while improving latency, meaning it got the same accuracy but faster [2]. That speed advantage is the core selling point, so a fair test must quantify it directly, not just report the final number.

Accuracy also hides generalization failures. The same study showed that Thinking States successfully extrapolated to longer sequences than seen during training on state-tracking tasks, while CoT did not [2]. That means a fair evaluation should include out-of-distribution tests—longer inputs, harder reasoning steps—because a model that only memorizes training patterns will look good on the test set but fail in the real world.

The hidden reasoning trap: small models already have latent ability

A fair evaluation must also account for the fact that small models can already reason—just not under standard decoding. One study found that small models (GPT-2) can generate high-quality reasoning paths during sampling, even without chain-of-thought prompting, but these paths are latent because they have low probability under standard decoding [1]. This means that if you test a small model with a standard greedy decoding, you might conclude it can't reason, when in fact it can—just rarely. A fair test should therefore include sampling-based evaluation or explicitly probe for latent reasoning, not just rely on the most likely output.

This also complicates the comparison between latent reasoning and knowledge distillation. Another study showed that fine-tuning a small model on teacher-generated CoT outputs can dramatically improve reasoning—for example, T5 XXL's accuracy on GSM8K jumped from 8.11% to 21.99% when fine-tuned on PaLM 540B's CoT [3]. But if the small model already has latent reasoning, then part of that improvement might come from activating its own ability, not just from the teacher's knowledge. A fair evaluation should separate these two effects—for instance, by comparing a distilled model against a self-trained model that uses its own latent paths, as SERT does [1].

One task is not enough: measure across reasoning types

Latent reasoning methods can excel on some tasks and fail on others, so a fair evaluation must cover a range of reasoning types—arithmetic, commonsense, symbolic, and state-tracking. In the studies here, Thinking States narrowed the gap to CoT on math problems and matched it on 2-hop QA, but it was on state-tracking that it actually beat CoT [2]. Meanwhile, the distillation studies focused on arithmetic (GSM8K) and commonsense reasoning, showing large gains from teacher-generated CoT [3]. If you only tested math, you'd miss the state-tracking advantage; if you only tested commonsense, you'd miss the latency benefit. A fair evaluation should include at least one task where latent reasoning is expected to shine (like state-tracking) and one where CoT is strong (like math), to see where the method truly stands.

The studies also differ in their training setups—some use a teacher model (GPT-3.5, PaLM 540B) to generate reasoning paths [1][3], while others use self-supervision from natural language [2]. This means a fair evaluation must also control for the training signal: are you testing the method's ability to learn from a teacher, or its ability to generate and use its own latent thoughts? The SERT study explicitly shows that self-training on filtered, self-generated paths can improve reasoning without a teacher's CoT [1], so a fair test should include a self-training baseline to isolate the contribution of latent reasoning versus distillation.

About These Sources

This answer is built on 3 studies (2 peer-reviewed, 1 preprint) — published from 2023 to 2026, 2 from 2024 or later, collectively cited 89 times — selected as the most relevant from 3 studies that passed quality screening, drawn from 20 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions