FASTER: Decoupling Performance Gains from Denoising Costs in RL

TL;DR

FASTER (Value-Guided Sampling for Fast RL) is a plug-and-play framework that unlocks the benefits of "Best-of-N" action sampling for diffusion policies without the $Nimes$ compute penalty. By training a lightweight critic to evaluate initial noise seeds instead of finished actions, FASTER achieves state-of-the-art performance in complex manipulation tasks while reducing inference FLOPs by up to 8x for large-scale Vision-Language-Action (VLA) models.

The Bottleneck: The "Tax" of Expressivity

In modern Reinforcement Learning (RL), diffusion policies are the gold standard for representing multi-modal, complex action distributions. To push performance further, researchers often use test-time scaling: sampling $N$ candidates and picking the one with the highest Q-value.

The catch? For a diffusion model with $T$ denoising steps, this requires $NimesT$ forward passes. When using a 3.3 billion parameter VLA model, this "tax" becomes an existential threat to real-time robotics, leading to latencies that make fluid movement impossible.

The Insight: Noise is Destiny

The core intuition behind FASTER is that the "goodness" of a sample is already latent in its starting noise $ϵ$. Instead of waiting for $T$ steps to see the final action $a$, why not predict the value of $a$ directly from $(s,ϵ)$?

The authors frame this as a Filtering MDP:

• State: The environment state + the current set of noise candidates.
• Action: Which candidates to keep and which to discard.
• Reward: The final Q-value of the resulting denoised action.

In its most practical form, FASTER collapses this MDP into a single step: it uses a noise-level critic to pick the best seed at $t=T$ (pure noise) and only denoises that one winner.

Methodology: Learning the Shortcut

The training process is elegantly simple. The noise-level critic $Q_{dn}(s,ϵ)$ is supervised by the standard action-level critic $Q_a(s,a)$. Crucially, the noise-level critic only needs to learn the ranking of seeds, which is a much lighter task than generating the actions themselves.

${\mathcal{L}}_{reg}=||Q_{dn}(s,{ϵ}_{i^{\ast }})-extsg[Q_a(s,a_{i^{\ast }})]|{|}_2^2$

This allows the use of a tiny critic (e.g., 20M parameters) to guide a massive actor (3.3B parameters).

Experimental Results: Speed Without Sacrifice

FASTER was tested against heavyweights like EXPO and IDQL across Robomimic and LIBERO benchmarks.

1. VLA Scaling: On a 3.3B parameter model, FASTER-EXPO reached comparable success rates to the baseline but slashed training update time from 11.6s to 2.5s and inference FLOPs by 8x.
2. Sample Efficiency: FASTER-EXPO consistently outperformed traditional methods like RLPD and QSM, proving that test-time scaling—when made efficient—is a superior path to high performance.

Why Distillation Isn't the Answer

A common alternative is to distill the $N$ samples back into a single-step policy. However, the authors show that FASTER significantly outperforms distillation. This is because distillation creates a "moving target" problem, where the policy struggles to track a non-stationary Q-function. FASTER avoids this by maintaining the richness of the original diffusion model and simply acting as an "intelligent filter."

Conclusion & Future Outlook

FASTER changes the coordinate system of generative RL. It shifts the focus from "how to denoise faster" to "how to choose what to denoise." While it currently relies on the existence of initial noise seeds, its success suggests a broader principle: In iterative generation, selection should happen as early as possible.

As we move toward even larger foundation models for robotics, "intelligent budgets" for test-time compute, like those provided by FASTER, will be the difference between a research demo and a production-ready robot.