Sol-RL: Scaling Diffusion RL to the "Speed of Light" with FP4 Exploration
FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling
The paper introduces Sol-RL (Speed-of-light RL), a novel two-stage reinforcement learning framework that accelerates the alignment of text-to-image diffusion models. Categorized as a post-training optimization method, it integrates NVFP4 quantization for high-throughput rollout scaling, achieving up to 4.64× faster convergence and reaching new SOTA alignment levels on models like FLUX.1-12B and SD3.5-L.
TL;DR
Post-training alignment via Reinforcement Learning (RL) has become the de-facto standard for making diffusion models follow human preferences. However, "scaling rollouts"—generating dozens of images just to pick one for training—is prohibitively expensive. Sol-RL solves this by using a hardware-accelerated FP4 exploration stage to find the best/worst candidates at 4x speed, then switching to BF16 for the actual training, achieving up to 4.64x faster convergence without losing any image quality.
The "Group" Bottleneck in Diffusion RL
Recent breakthroughs like GRPO (Group Relative Policy Optimization) have shown that the best way to train models is to generate a group of outputs for every prompt and optimize the model based on their relative quality.
The logic is simple: the more candidates you explore, the better the gradient signal. But for massive models like FLUX.1 (12B), generating 96 candidates per prompt is a computational nightmare. In most setups, the GPU spends 80% of its time just "sampling" and only 20% actually "learning." Moving to low-precision (FP4) seems like an obvious fix, but there’s a catch: Quantization Noise. Training a high-precision model on "noisy" FP4 images corrupts the model, leading to visual artifacts and unstable training.
Methodology: The "Decoupled" Two-Stage Framework
The researchers from NVIDIA, HKU, and MIT noticed a hidden property: FP4 images might look slightly "off," but their relative ranking remains nearly identical to BF16. If image A is better than image B in BF16, it is almost certainly better in FP4 too.
Stage 1: Massive FP4 Exploration
Using the NVIDIA Blackwell architecture's NVFP4 capabilities, Sol-RL generates 96 candidates using a quantized model and very few denoising steps (e.g., 6 steps). This is strictly used to identify the winners and losers (Top-K/Bottom-K).
Stage 2: Selective BF16 Regeneration
Once the best/worst seeds are identified, the model "goes back in time" and regenerates only those 24 contrastive samples in full BF16 precision.
Figure 1: The two-stage pipeline separating high-throughput exploration from high-fidelity training.
Why It Works: Statistical Synergy
The paper provides a beautiful theoretical justification using Extreme Value Theory (EVT). Essentially, while FP4 introduces a constant "quantization disturbance" (), the reward margin between the best and worst samples grows with the group size at a rate of . As you scale , the signal-to-noise ratio improves, and the "correct" ranking emerges despite the low precision.
Experimental Results: Faster, Better, Cheaper
The framework was tested on FLUX.1-dev, SANA-1.5, and SD3.5-L.
- Convergence: Sol-RL reaches the same reward level as standard methods in less than half the time (up to 4.64x faster on certain tasks).
- Efficiency: Isolated "Rollout Time" was slashed by 2.4x on SD3.5-Large.
- Fidelity: Unlike naive FP4 training, Sol-RL maintains 99% of the BF16 alignment performance because the actual gradient updates are performed on high-precision samples.
Figure 2: Learning curves showing Sol-RL consistently outperforming DiffusionNFT across multiple reward metrics (HPSv2, ImageReward).
Critical Insight & Future Outlook
Sol-RL represents a shift in Algorithm-Hardware Co-design. It acknowledges that while training must be precise, exploration can be "messy" but fast.
Takeaways for Practitioners:
- Don't train on quantized rollouts: Directly using 4-bit samples as labels for a 16-bit model is a recipe for failure due to distribution shifts.
- Scale the Group, not the Steps: You get more benefit from ranking 96 cheap samples than 8 expensive ones.
- Hardware Matters: This method specifically leverages NVFP4, which is a game-changer for next-gen GPU clusters.
Limitations: The current approach requires a second "regeneration" pass. While this is cheaper than a full high-precision group rollout, it still adds some latency compared to a theoretical "single-pass perfect quantization" (which doesn't yet exist).
Conclusion
Sol-RL effectively breaks the efficiency-integrity dilemma in Diffusion RL. By treating low-precision inference as a "ranking proxy" rather than a "ground truth," it unlocks the ability to scale human-preference alignment to the world's largest generative models at a fraction of the traditional cost.