DeScore: Decoupling Reasoning and Scoring for SOTA Video Reward Modeling

TL;DR

To build a better Video Reward Model (RM), you need more than just a preference score; you need to know why one video is better than another. DeScore introduces a "Think-then-Score" paradigm that first generates a Chain-of-Thought (CoT) analysis and then uses a dedicated module to regress a scalar reward. This approach solves the optimization instability of generative models and the "shortcut learning" issues of discriminative ones, achieving SOTA results with 76% less data.

The Dilemma: Accuracy vs. Stability

In the current video generation landscape (Sora, Kling, Wan-2.1), the Reward Model is the "judge" that guides post-training and test-time scaling. However, developers faced a choice between two flawed paradigms:

1. Discriminative RMs: They treat the video as a black box and output a number. Because they don't "reason," they often find shortcuts (e.g., favoring bright colors over actual motion consistency) and require millions of samples to generalize.
2. Generative RMs: They use CoT to explain their judgment. While interpretable, they treat the reward as just another token. This couples logic and scoring, leading to high-variance gradients—if the model is 100 tokens into a "thought," it's hard to tell if a bad final score was due to a logical error or a bad scoring token (the credit assignment problem).

Methodology: The DeScore Architecture

DeScore bridges this gap using a decoupled framework. It employs a Qwen3-VL-8B backbone that first outputs a detailed reasoning sequence (analyzing subject, dynamics, camera, etc.). Instead of ending with a "score token," it appends a learnable [Reward] query token.

The hidden state of this [Reward] token aggregates the visual features and the reasoning tokens into a d-dimensional vector, which is then projected by a regression head into a continuous scalar.

The Two-Stage Training Recipe

• Stage 1: Discriminative Cold Start: The model is trained on human preference pairs using the Bradley-Terry (BT) loss. A key innovation here is Random Masking—the CoT is randomly dropped during training, forcing the scoring head to look at the actual video and the text, ensuring it doesn't get lazy and only read its own "thoughts."
• Stage 2: Dual-Objective RL: DeScore uses Group Relative Policy Optimization (GRPO) to improve the quality of the "Thinking" process, while simultaneously using an auxiliary BT loss to keep the "Scoring" head calibrated. This prevents "reward drift" where the model gets better at talking but worse at judging.

Experimental Results: Efficiency and Generalization

The results prove that "Thinking" makes the model smarter on less data. DeScore achieves better generalization (Out-of-Distribution performance) than VideoAlign while using only a fraction of the training data.

Better Video Generation

When DeScore was used as the reward signal for training Wan-2.1-1.3B using Flow-DPO or GRPO, the resulting videos showed marked improvements in subject consistency and dynamic degree. This proves DeScore is an effective "teacher" for generative models.

Critical Insights: Why it Works

The mathematical secret of DeScore lies in its gradient behavior. The paper theoretically proves that GRPO gradient variance scales as $\Omega (T)$, where $T$ is the sequence length. By moving the score to a separate regression head optimized by BT loss, DeScore provides a deterministic "push-and-pull" force that is far more stable than the stochastic exploration of policy gradients used in categorical token prediction.

Limitations

Currently, DeScore focuses heavily on semantic alignment (does the video match the prompt?). The authors admit it might be less sensitive to purely visual artifacts or "hallucinations" in motion that aren't explicitly described in text. Future iterations plan to incorporate multi-dimensional rewards (aesthetic, motion, etc.) to address this.

Conclusion

DeScore demonstrates that the future of reward modeling isn't just "black-box scores" or "pure-text responses." By decoupling Thinking (symbolic, interpretable) from Scoring (continuous, stable), we create an RM that is both human-readable and mathematically robust. This is a vital step toward the next generation of high-fidelity, controllable video AI.