[ArXiv 2025] DUEL-EVOLVE: Scaling Test-Time Compute via LLM Self-Preferences

TL;DR

DUEL-EVOLVE is a novel framework that allows Large Language Models to improve their own outputs at test time without needing a reward model or ground-truth labels. By framing solution refinement as an evolutionary process guided by Bayesian Pairwise Preferences, it achieves massive gains (+20% accuracy on MathBench) simply by "dueling" candidates against each other.

Core Position: This work marks a shift from "LLM-as-a-Scorer" (scalar) to "LLM-as-a-Ranker" (pairwise) in the taxonomy of test-time optimization, proving that relative comparison is a significantly more robust signal for self-improvement.

The "Scalar" Trap: Why Self-Correction Often Fails

Most current efforts to scale LLM performance at inference time (like Best-of-N or iterative refinement) rely on scalar rewards. Whether it's a separate Reward Model (RM) or the LLM giving itself a score from 1-10, the signal is often:

• Sparse: In math, it's usually just "right" or "wrong" at the very end.
• Inconsistent: LLMs are notoriously bad at being calibrated "thermometers" of their own quality.
• Brittle: Optimization against a noisy scalar reward often lead to "reward hacking" or stagnation.

The authors of DUEL-EVOLVE realize that while an LLM might struggle to give a "9/10" score consistently, it is remarkably good at looking at two solutions and saying, "Candidate A is better because it handled the edge case in step 3."

Methodology: Bayesian Dueling Bandits

DUEL-EVOLVE turns the search for an optimal solution into a Dueling Bandits problem.

1. The Bradley-Terry Aggregator

Each candidate $y_i$ is assigned a latent utility $het{a}_i$. When the LLM (acting as the Judge) compares two candidates, the result is modeled via the Bradley-Terry likelihood: $P(y_i\succ y_j)=\sigma (het{a}_i-het{a}_j)$ This allows the system to build a global ranking from local, noisy comparisons.

2. Double Thompson Sampling (DTS)

With a limited budget of LLM calls, you can't compare everything. DUEL-EVOLVE uses DTS to:

• Explore: Compare candidates where the quality is still uncertain.
• Exploit: Focus on comparing the top-tier candidates to find the absolute "winner."

3. Evolutionary Generation

High-scoring candidates aren't just winners; they are parents. The LLM is prompted with a few top-ranked solutions (and their estimated scores) to "evolve" a new, superior candidate.

Figure 1: The DUEL-EVOLVE loop: Selection via DTS, Quality Estimation via Bradley-Terry, and Evolution via In-Context Learning.

Performance: Breaking the SOTA

The results are particularly striking in domains where rewards are historically hard to define.

MathBench (Reasoning)

In a 150-generation run, DUEL-EVOLVE surged to 94% accuracy. For context, a standard Zero-shot CoT approach sits at 57.3%. Most of the gains occurred early, reaching 90% within just 10 generations.

Figure 2: Accuracy scaling over generations. Note the rapid early convergence across different difficulty levels.

LiveCodeBench (Coding)

Coding is the ultimate test of discrete optimization. DUEL-EVOLVE reached 37.4% accuracy, outperforming "Feedback Descent" (a single-path refinement method) by a wide margin. This suggests that maintaining a population of solutions is strictly better than "hill-climbing" a single answer.

Critical Insight: The "Judge" is the Bottleneck

While DUEL-EVOLVE is powerful, its Achilles' heel is Judge Bias. If the LLM judge prefers "confident-sounding" answers over "correct" ones, the evolutionary loop will optimize for arrogance rather than accuracy.

However, in mathematical and symbolic domains, the logic is often self-evident once written down, making the pairwise preference signal much stronger than a blind scalar score.

Conclusion

DUEL-EVOLVE proves that we can scale test-time compute effectively without external oversight. By combining the probabilistic rigor of Bayesian bandits with the semantic creativity of LLMs, it provides a blueprint for models that can truly "think" and "iterate" their way to a solution.

What's next? Integrating this with "evolving memory" (allowing the model to keep notes on what failed) and ensembling judges could lead to even more resilient self-improvement loops.