TEMPO: Scaling Test-time Training for Large Reasoning Models

TEMPO: Breaking the Self-Training Plateau for Large Reasoning Models

TEMPO: Scaling Test-time Training for Large Reasoning Models

Summary

Problem

Method

Results

Takeaways

Abstract

TEMPO (Test-time Expectation-Maximization Policy Optimization) is a novel test-time training (TTT) framework that enables Large Reasoning Models (LRMs) to continuously improve on unlabeled test data. By framing TTT as an EM-style process with alternating critic recalibration and policy refinement, it achieves state-of-the-art results, such as boosting Qwen3-14B on AIME 2024 from 42.3% to 65.8% accuracy.

TL;DR

Current Large Reasoning Models (LRMs) often stop improving or lose their "creative" diversity during test-time training because they essentially "grade their own homework" without a teacher. TEMPO fixes this by introducing an alternating Expectation-Maximization (EM) framework. By periodically recalibrating a Critic model on a small set of labeled data, it provides a stable, grounded reward signal that allows LRMs like Qwen3 and OLMO3 to keep learning from unlabeled test questions without hitting a performance ceiling or collapsing into repetitive answers.

Problem & Motivation: The Self-Reinforcement Trap

In the quest to make AI more like human thinkers, Test-Time Training (TTT) is a holy grail. It allows a model to adapt its parameters to new, difficult problems during inference. However, existing methods like TTRL (Majority Voting) or entropy-based rewards face a grim reality: the self-reinforcement loop.

If a model is asked a complex math problem it doesn't quite understand, and it consistently generates the same wrong answer, a majority-voting reward tells the model it is doing great. This "echo chamber" effect causes two major issues:

Performance Plateaus: The model can't learn anything beyond its initial biased guesses.
Diversity Collapse: In its rush to maximize its own (wrong) reward, the model forgets how to explore alternative reasoning paths, killing its Pass@K performance.

The authors of TEMPO realized that these failures stem from a structural deficiency: prior methods only perform "Policy Refinement" (The M-Step) while ignoring "Reward Calibration" (The E-Step).

Methodology: The EM Engine of TEMPO

TEMPO treats the "correctness" of a test-time answer as a latent variable. To optimize this, it uses a two-stage alternating process:

1. The E-Step (Critic Recalibration)

Instead of trusting the model's own confidence, TEMPO uses a separate Critic model. This critic isn't static; it is periodically updated on a labeled dataset ( $D_{L}$ ). This ensures the critic's definition of a "good answer" remains tied to reality (ground truth), prevented it from drifting as the policy model changes.

2. The M-Step (Policy Refinement)

The model (Actor) generates reasoning trajectories for unlabeled test questions ( $D_{u}$ ). The now-recalibrated Critic provides quality-aware scores. These scores are used to calculate "Advantages" — reinforcing tokens that lead to high-value outcomes and penalizing those that diverge.

TEMPO Framework Architecture

Scalability and Results

The most striking result of TEMPO is its Scalability. While traditional self-rewarding RL methods (like TTRL represented in grey in Figure 1) see their gains vanish or plateau after a few steps, TEMPO (the blue curve) shows a sustained upward trajectory.

AIME 2024 Benchmarks: Qwen3-14B jumped from a baseline of 42.3% to a staggering 65.8%.
AIME 2025 Benchmarks: OLMO3-7B improved from 26.3% to 37.0%.

Preserving Diversity

Perhaps more important than raw accuracy is the preservation of reasoning diversity. In the chart below, we see that while TTRL (grey) improves average accuracy at the cost of diversity (Pass@16), TEMPO manages to improve both. This means the model isn't just getting better at guessing one answer; it's getting better at finding multiple ways to solve a problem.

Diversity Comparison

Critical Analysis & Conclusion

Takeaway

TEMPO proves that the secret to continuous learning isn't just "more compute," but grounded compute. By formalizing TTT as an EM process, the authors have provided a roadmap for how models can safely explore unlabeled problem spaces without losing their logical "north star."

Limitations

Overhead: Running both an Actor and a Critic during test-time increases VRAM requirements.
Dependency: It still requires a small, high-quality labeled dataset for the E-step calibration.

Future Outlook

The success of TEMPO on STEM and Logic puzzles (BBH, ZebraLogic) suggests this framework is general-purpose. We can expect future "Real-time RL" systems—like those used in coding assistants or autonomous agents—to adopt similar periodic recalibration steps to ensure they don't hallucinate progress.

Performance Scaling Ablation Figure 5: Even after a model converges on supervised training, TEMPO continues to unlock new capabilities by training on novel problems.

Find Similar Papers

Try Our Examples

Find other recent papers that use Expectation-Maximization (EM) or Variational Inference to improve reinforcement learning for LLM reasoning tasks.
Which paper first proposed the concept of Test-Time Training (TTT) for deep learning models, and how has the definition evolved from Computer Vision to Large Language Models?
Look for studies investigating how to maintain output diversity in Large Language Models during on-the-fly reinforcement learning or parameter-efficient fine-tuning.

Contents

TEMPO: Breaking the Self-Training Plateau for Large Reasoning Models

1. TL;DR

2. Problem & Motivation: The Self-Reinforcement Trap

3. Methodology: The EM Engine of TEMPO

3.1. 1. The E-Step (Critic Recalibration)

3.2. 2. The M-Step (Policy Refinement)

4. Scalability and Results

4.1. Preserving Diversity

5. Critical Analysis & Conclusion

5.1. Takeaway

5.2. Limitations

5.3. Future Outlook