[arXiv 2026] Astrolabe: Steering Distilled Video Models Toward Human Preference

TL;DR

Astrolabe is an efficient online Reinforcement Learning (RL) framework specifically tailored for distilled autoregressive (AR) video models. By utilizing a trajectory-free forward-process RL formulation and a specialized streaming training scheme with a rolling KV-cache, it aligns real-time video generation with human aesthetics without the massive memory overhead typical of RL.

Positioning: This work moves beyond simple distribution matching (distillation) to human-centric alignment (RLHF for video), serving as a "post-training" booster for the current generation of efficient streaming video models.

Problem & Motivation: The Gap in Streaming Video

While current video diffusion models (like SVD or Sora) produce stunning results, they are slow. To solve this, the community has turned to distilled AR models (e.g., Causal Forcing, Self-Forcing) which enable streaming generation via KV-caching. However, these models have two major flaws:

1. Preference Mismatch: They mimic the teacher's average distribution but don't optimize for what humans actually like (e.g., avoiding motion blur, maintaining geometric consistency).
2. Optimization Bottleneck: Standard RL (like PPO or GRPO) requires storing the entire sampling trajectory or unrolling reverse diffusion steps, which consumes massive VRAM—often exceeding the limits of even high-end H200 GPUs for long videos.

Methodology: High-Efficiency Alignment

The authors define Astrolabe through three core innovations:

1. Forward-Process RL (Trajectory-Free)

Instead of backpropagating through the complex denoising solver steps (Reverse-Process), Astrolabe uses Negative-Aware Fine-Tuning (NFT). It contrasts "positive" samples (high reward) with "negative" ones directly at the inference endpoint. This creates an implicit policy direction $v^{+}$ and $v^{-}$ to update the velocity field $v_{h}eta$ without needing to store intermediate states.

Fig 1: Astrolabe facilitates alignment across short, long, and multi-prompt video generation.

2. Streaming Training Scheme

To handle long videos (30s+), Astrolabe employs a Rolling KV-Cache with Frame Sinks.

• Frame Sinks: Permanently store a few anchor frames to maintain global semantic context.
• Detached Context: During training, the history (KV-cache of previous frames) is "detached" from the gradient graph. Only the current local "active window" receives updates. This keeps memory usage constant regardless of video length.

3. Multi-Reward & Selective Regularization

To prevent "Reward Hacking" (where the model finds a loophole in the reward function, like producing static but high-quality images), Astrolabe uses:

• Composite Reward: A mix of Visual Quality (HPSv3), Motion Quality (VideoAlign), and Text Alignment.
• Uncertainty-Aware KL Penalty: Unlike standard RL which penalizes all deviations from the base model, Astrolabe only applies a KL penalty when different reward models disagree, allowing for more creative exploration in "safe" regions.

Experiments & Results

The framework was tested on several baseline architectures including Self-Forcing and Causal-Forcing.

SOTA Comparison

As shown in the table below, adding Astrolabe to existing distilled models significantly boosts the Human Preference Score (HPSv3) and Motion Quality (MQ) without decreasing the generation throughput.

Table 1: Performance gains over various Distilled AR Models.

Ablation Insights

The ablation study confirms that "Clip + Detach" (their streaming strategy) is the only method that avoids Out-Of-Memory (OOM) errors while providing a 2x memory reduction compared to full backpropagation.

Table 4: Strategy comparison showing the efficiency of the Streaming Training Scheme.

Critical Analysis & Conclusion

Takeaway: Astrolabe represents a significant step in making long-form, high-quality video generation viable on consumer-grade (or at least reasonable data center) hardware. By shifting optimization to the forward process and using streaming windows, it solves the "memory wall" of video RL.

Limitations:

• Reward Reliance: The model is only as good as the reward functions (HPSv3, etc.). If these models have biases (e.g., preferring certain colors/styles), Astrolabe will inherit them.
• Capacity Ceiling: RL can align existing knowledge but cannot "teach" a model new physics that wasn't present in the base distilled architecture.

Future Work: The integration of even more robust, physics-aware reward models could further refine its ability to generate complex interactions and causal sequences.