AR-CoPO: Solving the Determinism Trap in Autoregressive Video RLHF

Executive Summary

TL;DR: AR-CoPO (AutoRegressive Contrastive Policy Optimization) is an alignment framework that brings RLHF to the challenging world of low-latency, streaming AR video generators. By ditching ineffective stochastic SDE exploration in favor of a chunk-level contrastive "forking" mechanism, it achieves superior human preference alignment without the typical "reward hacking" that destroys video motion.

Strategic Positioning: This work bridges the gap between high-efficiency distilled video models (like Self-Forcing) and controllable reinforcement learning. It moves beyond the limitations of early SDE-based GRPO methods that struggle with the near-deterministic nature of few-step solvers.

The Problem: The "Determinism Trap" in Few-Step Models

Modern video generation is moving toward streaming autoregressive (AR) architectures for low latency. To make these practical, they are often distilled into few-step solvers (e.g., Consistency Models).

However, this creates a fundamental conflict for RLHF:

1. Exploration Failure: Standard RL methods like SDE-GRPO rely on injecting noise at every step to explore. In few-step distilled models, the output is almost entirely determined by the initial noise. Adding noise at intermediate steps changes almost nothing (see Figure 2), leaving the RL agent with no signal to learn from.
2. Streaming Complexity: Calculating rewards for a full long-form video is slow, and assigning "credit" to which specific frame caused a bad reward is a nightmare.

Fig. 1: Demonstrating that in few-step AR models, only the initial noise (Row 2) causes meaningful variation, while intermediate solver noise (Rows 3-5) is ignored.

Methodology: Chunk-Level Forking & Contrastive Optimization

AR-CoPO introduces three key innovations to solve these issues:

1. The Forking Mechanism

Instead of optimizing the whole video at once, AR-CoPO selects a pivot chunk.

• It generates a "Shared Context" (pre-pivot).
• It forks at the pivot chunk, creating $G$ different candidates using perturbed initial noise.
• It completes the video for each branch using shared future noise. This ensures that the difference in final rewards can be blamed entirely on the decisions made at the pivot chunk.

2. Contrastive Policy Optimization (CoPO)

Since standard policy gradients fail due to low stochasticity, AR-CoPO uses a "surrogate" distribution. It measures the distance between the current policy's prediction and the "neighbor" candidates in the latent space. If a neighbor has a high reward, the model is "pulled" toward that neighbor's latent representation.

3. Semi-On-Policy Exploitation

Pure on-policy exploration often leads to reward hacking (e.g., the model finds a "shortcut" to high text-alignment by making the video static, killing motion quality). AR-CoPO uses a replay buffer of reference rollouts to keep the model grounded in high-quality generation while it explores.

Fig. 2: The AR-CoPO workflow: Context -> Chunk-level Forking -> Sequence Reward -> Local Update.

Experimental Results: True Alignment vs. Hacking

The results on the Self-Forcing baseline demonstrate that AR-CoPO is more robust than previous SDE-based approaches.

• SDE-GRPO vs. AR-CoPO: As shown in the training curves, SDE-based methods fail to improve the reward at all, while AR-CoPO shows steady gains.
• Averting Motion Collapse: Pure on-policy optimization for Text Alignment (TA) usually results in a catastrophic drop in Motion Quality (MQ) from 1.68 to 0.25. AR-CoPO’s semi-on-policy strategy maintains a high MQ of 1.86 while still improving the overall alignment.

Table 1: AR-CoPO (merged) achieves the best balance between human preference (VideoAlign) and general quality (VBench).

Critical Insight: The "Dual-Benchmark" Requirement

The authors make a crucial point for future AI Research: Never trust a single reward score. They observed that models can "hack" reward models (e.g., getting a high Text-Alignment score by producing artifacts). By requiring a model to improve on both the in-domain reward (VideoAlign) and an independent out-of-domain benchmark (VBench), AR-CoPO proves it is achieving genuine quality improvement.

Conclusion

AR-CoPO provides a scalable, stable blueprint for aligning the next generation of real-time video AI. By acknowledging the deterministic nature of few-step models and using contrastive chunk-level updates, it effectively bridges the gap between raw generative power and human-centric control.

Future Outlook: The "forking" logic is likely applicable beyond video—any autoregressive task (like long-form audio or multi-turn dialogue) facing credit assignment challenges could benefit from this contrastive pivot-chunk approach.