How does DiffusionART stabilize and scale reinforcement of diffusion models?

Two families of diffusion RL trade efficiency for stability

Online diffusion RL evaluates complete generated samples with black-box rewards, so output-level feedback must be converted into updates across many denoising states. The paper frames this as a credit-assignment problem and splits existing methods by where they apply the update.

Reverse-process methods such as Flow-GRPO optimize sampled denoising transitions using group-based endpoint rewards. They train on states the generator actually visits, but they need an ODE-to-SDE conversion for tractable likelihoods, and each selected transition is pushed along its own transition-level sampling noise while weighted by a trajectory-level advantage that the transition only partly determines. The paper identifies this mismatch as poor credit assignment that limits efficiency.

Forward-process methods such as DiffusionNFT noise the generated clean samples and regress velocities with reward-weighted positive and negative objectives. They need no transition likelihoods and are efficient, but the paper reports they are unstable without two stabilizers: joint positive-negative training and an EMA rollout policy whose lag must be tuned per task. Neither family is both efficient and stable without per-task tuning, which is the gap the paper targets.

NFT objectives carry a hidden consistency force

The paper starts from a puzzle: the positive and negative branches of NFT-style training share the same ideal reward-improving direction, so either branch alone should suffice, yet each alone collapses. The explanation is that NFT's targets, built from forward-noised endpoints, implicitly assume the model's sampling velocity equals the ideal forward velocity induced by its own endpoints. In practice the two differ.

That gap creates a non-reward-improving forward-reverse consistency force. The positive branch pulls each state's velocity toward the ideal forward velocity and the negative branch pushes it away. Joint positive-negative training can cancel this force only partially, because reward centering balances rewards across samples rather than at each individual state. The residual can destabilize training and slow reward optimization.

The paper also reinterprets EMA. Its lagged reference supplies a restoration force that stabilizes NFT, not off-policy rollouts as such, but the lag must still be tuned per task. This distinction matters because it suggests stability can be obtained by a restoration mechanism that does not require an EMA schedule.

Aggregating rewards first removes the force by construction

DiffusionART borrows supervised velocity regression from forward-process training but applies updates to states and actions visited during reverse-process rollouts. For each prompt, it collects K endpoints with rewards, states, and the model's velocity predictions, then aggregates that prompt's samples and rewards into a single shared reward-improving direction G(c).

Each visited state's action is regressed toward the frozen target formed by its own prediction plus G(c). Because one direction is shared across the prompt's states rather than derived per endpoint, the consistency force is removed by construction rather than cancelled approximately. The method requires no differentiable reward model, so it works with scalar black-box rewards.

A second update is added with a trust-region mask. The mask detects excessive drift and outward updates using a fixed drift threshold of 10^-5 and a factor that emphasizes high-noise states, and it constrains only this second update. The paper describes the second update as drift-adaptive restoration that replaces per-task EMA tuning.

Video training reaches baseline rewards with far fewer GPU hours

The headline comparison uses HunyuanVideo-13B with equally weighted visual and motion quality rewards. DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours respectively, while its mean reward over the final 50 training steps is 13.018 versus 10.105 and 8.510, improvements of 28.8% and 53.0%.

Under the main text's matched TA:VQ:MQ weights of 7:2:1, DiffusionART reaches a training VQ of 6 and MQ of 3 after 29 and 46 steps, levels the other methods do not reach within their recorded trajectories. For a VQ of 5.5 it needs 24 steps and 63.4 GPU hours, versus 273 steps and 1,401.4 GPU hours for DiffusionNFT with EMA, which the paper reports as 11.4 times fewer steps and 22.1 times fewer GPU hours. Evaluation VQ and MQ reach 8.970 and 4.224, 31.4% and 57.4% above the strongest baseline.

These are training-reward and evaluation comparisons within the paper's own setup, not claims about arbitrary video models. The paper also reports that DiffusionART keeps text-alignment reward comparable with the strongest baseline while improving visual and motion quality.

Image gains hold, but the mask matters mainly for video

On SD3.5-M, DiffusionART avoids the severe collapse of on-policy DiffusionNFT without per-task EMA tuning and reaches training efficiency comparable to DiffusionNFT with EMA. It reaches Flow-GRPO's step-500 reward with 10.0, 5.2, and 9.6 times fewer training steps on GenEval, OCR, and PickScore respectively, and behaves similarly under single- and multi-reward training.

Scaling to FLUX.1-dev at 12B, DiffusionART learns fastest among the compared methods while DiffusionNFT shows pronounced reward fluctuations because its hyperparameters were not retuned for that model.

The trust-region mask is not equally important everywhere. Removing only the mask on image tasks leaves training effective and can even improve slightly faster, with a no-mask FLUX.1-dev run reaching a training GenEval score of 0.971 by step 180. For video with VQ:MQ = 1:1 weights, the no-mask run drops around step 200 before recovering while the masked run improves steadily. Diagnostics on HunyuanVideo report a maximum first-update gradient norm of 1.96 with masking versus 3.33 without over the shared first 300 steps, suggesting the mask also keeps later first updates well behaved.

Small rollout groups and pairwise rewards stay effective

DiffusionART is reported to be insensitive to group size K, and smaller K substantially reduces compute. Under the same 400-step budget with pairwise feedback, K=2 reaches a final VQ of 12.235, no lower than 12.092 for K=8, while using only 260.8 GPU hours versus 883.3, a 3.4-fold reduction.

DiffusionART-Pairwise extends the method to online pairwise preference feedback by splitting each prompt's K endpoints into disjoint pairs with winner, loser, or tie labels, replacing scalar rewards in the otherwise unchanged covariance and training pipeline. The paper reports pairwise training performance close to scalar feedback when scalar reward is used as the evaluation metric.

The paper also compares against ReFL, which backpropagates through a frozen reward model and VAE. DiffusionART improves both rewards with fewer optimizer updates while using reward values alone, though ReFL uses fewer samples per update, K=1 versus K=8.

Choose DiffusionART when tuning budget is the constraint

The paper's distinctive claim is not simply that DiffusionART is faster, but that it removes the need to tune an EMA lag per task while keeping forward-process efficiency. That trade-off is what separates it from DiffusionNFT, whose stability depends on a tuned EMA, and from Flow-GRPO, whose reverse-process credit assignment limits efficiency.

The evidence also sets a practical priority: the trust-region mask earns its place mainly in video generation, where removing it produced a mid-training drop, whereas image tasks trained effectively without it. And when compute is tight, the K=2 pairwise result indicates that one pair per prompt can match K=8 final VQ at roughly a quarter of the rollout cost in the reported setting.

These conclusions are bounded by the paper's own configurations: LoRA rank-32 adapters at 512x512 for SD3.5-M, full fine-tuning of HunyuanVideo-13B at 480x480 with 16 sampling steps, and specific reward weightings. The reported speedups and stability are measured within those setups, so the priority they support is a starting hypothesis for comparable image and video post-training, not a guarantee across different reward models or sampling budgets.

Sources used in this answer