What would a fair evaluation of looped language models for tool calling need to measure?

What should a fair evaluation measure beyond final accuracy?

A fair evaluation must go beyond simple success rates because looped models trade compute for accuracy. In a controlled comparison, looped models improved multi-step tool use but gains were smaller and model-dependent for isolated API calls [4]. So a fair test should report accuracy separately for single-turn vs. multi-turn tasks, and also measure compute cost per successful call. The scaling law study found that looping a block four times is equivalent to a 580M non-looped model but costs the training compute of a 1B model [7] — that's a 72% compute overhead for the same validation loss. In evaluation, you'd want to know if that extra compute buys enough accuracy to be worth it.

Another key metric is internal reliability: can the model's hidden state predict which tool it will call before it emits the call? One study showed that reading the tool choice from internal state recovers 61-82% accuracy on a benchmark while base generation only achieves 2-10% [1]. That means the model often 'knows' the right tool internally but fails to output it. A fair evaluation should measure this gap, because it indicates whether errors are due to representation or output wiring. The same study found that when the model is unsure between two tools, failures are 21 times more frequent [1] — so measuring confidence or uncertainty could flag risky calls.

Finally, a fair evaluation should include both synthetic and real-API benchmarks. The tool-calling study used a 15-tool synthetic benchmark and a real-API airline benchmark, and found that internal steering worked at 83-100% on the synthetic but 77-94% on the real one [1]. That difference shows that synthetic benchmarks may overestimate reliability. So a fair evaluation would report results on both types, and ideally include a stateful environment like the one in the visual tool-calling benchmark, which found that even the best model succeeds less than 50% of the time on multi-turn visual tasks [5].

How should compute and model scale be accounted for?

A fair evaluation must control for compute and parameter count, because looped models reuse weights and can appear cheaper than they are. The iso-depth scaling law study introduced a 'recurrence-equivalence exponent' of 0.46, meaning that looping a block four times is not equivalent to four unique blocks — it's only about half as effective in terms of validation loss [7]. This means that when comparing a looped model to a non-looped one, you must match not just parameter count but also training compute and inference FLOPs. The LoopMoE paper did exactly that, matching total parameters, per-token FLOPs, and active sublayer ratios, and found that looped MoE outperformed vanilla MoE on 8 of 9 benchmarks [8]. That's the kind of controlled comparison that makes results trustworthy.

Scale also matters: the tool-calling study found that internal steering works well on 4B+ instruction-tuned models (83-100% accuracy) but less reliably on smaller models [1]. The visual tool-calling benchmark found a 'planning-to-precision crossover': smaller models fail at deciding what to do, while larger models fail at perceiving what they see [5]. So a fair evaluation should report results across a range of model sizes, and ideally separate failures by type (planning vs. perception vs. execution). Otherwise, you might conclude a model is bad at tool calling when it's actually just bad at vision, or vice versa.

Compute efficiency also includes inference-time choices. The looped tool-calling study found that adaptive inference — allocating more computation only when needed — achieves a better compute-performance trade-off than fixed recurrent depth [4]. So a fair evaluation should measure not just accuracy but also the compute cost per successful task, and consider whether adaptive strategies are allowed. The DSPy framework shows that optimizing pipelines can improve performance by 25-65% over standard few-shot prompting [3], which suggests that the evaluation protocol itself can dramatically affect results. A fair evaluation should specify whether the model is allowed to be optimized for the benchmark, or if it's a zero-shot test.

Why should internal reliability and intervention success be part of the evaluation?

A fair evaluation should measure not just what the model outputs, but whether its internal state is reliable. The proto-introspection study found that a probe on hidden states can predict eventual success on math problems with AUROC 0.797, which is better than using shortcuts alone (0.731) [2]. That means the model's internal state carries information about task quality that isn't fully reflected in the output. For tool calling, this could mean measuring whether the model's internal representation of the tool choice matches the actual call. The tool-calling study showed that base models already carry the right tool internally before they can emit it, with 61-82% accuracy on a benchmark while generation only gets 2-10% [1]. So a fair evaluation would report this 'internal-to-output gap' as a diagnostic.

However, internal reliability does not guarantee that interventions can improve outcomes. The proto-introspection study tested several interventions (branching, steering, LoRA) and found that none produced a validated capability gain [2]. This is a crucial caveat: even if we can read the model's internal state, we may not be able to use it to fix errors. A fair evaluation should therefore distinguish between 'readable' and 'usable' internal signals. The tool-calling study also found that multi-turn agent loops are less stable for interventions, with gains or losses of up to 30 percentage points and no consistent direction [1]. So a fair evaluation should test interventions across both single-turn and multi-turn settings, and report variability, not just average effects.

Finally, a fair evaluation should consider the risk of overreliance on tools. The alignment study found that LLMs often overuse tools, and their framework reduced unnecessary tool usage while maintaining performance [6]. That suggests that a fair evaluation should measure not just success rate but also tool efficiency — how many calls were necessary to complete the task. A model that makes 10 calls when 2 would do is less efficient, even if it succeeds. So metrics like 'calls per successful task' or 'unnecessary tool usage rate' should be part of the evaluation.

About These Sources

This answer is built on 8 studies (2 peer-reviewed, 6 preprints) — published from 2023 to 2026, 7 from 2024 or later, collectively cited 539 times — selected as the most relevant from 9 studies that passed quality screening, drawn from 35 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions