Why a single reward score can't capture what people actually prefer
Human preference in video-audio generation isn't one thing—it's a bundle of things: audio quality, visual fidelity, synchronization, and overall coherence with the text prompt. Older approaches tried to combine separate metrics for each dimension, but that doesn't work well. A 2026 study on joint video-audio generation found that optimizing against these separate metrics encourages 'reward hacking'—the model produces content that scores high on each metric but looks incoherent or unfaithful to human viewers [4]. In other words, the reward model was rewarding the wrong thing because it couldn't see the whole picture.
The same study built a large human-preference dataset (VAPref-10K, with 9K prompts and 10.3K paired comparisons) and found that a chain-of-thought reward model that reasons about multiple quality dimensions predicts human preferences better than metric-based baselines, both in-domain and out-of-domain [4]. This suggests that to evaluate a reward model, you need to test it against human judgments on holistic coherence, not just individual metrics—which is harder to set up and measure.
The problem of delayed feedback and unreliable progress signals
In long-horizon tasks, you don't get feedback until the very end, after many steps. That makes it hard to know which step caused a good or bad outcome. A 2026 paper on video editing agents noted that agents receive 'final-product feedback only after many interdependent decisions,' and that editing quality is subjective and not meaningfully calibrated across different requests [5]. They proposed a method that converts same-task rankings into credit assignments over segments, but the core difficulty remains: you have to infer credit for each step from a single final judgment.
Progress-based reward models try to solve this by estimating how far along the task you are at each step, but they can be unreliable. A 2026 study on robot manipulation found that existing progress models can assign high rewards to 'visually plausible but physically incorrect states'—a false positive that is especially harmful in long tasks like cloth folding [3]. Their solution, a confidence-gated reward model, only rewards confident forward progress and suppressed uncertain matches, which improved success rates significantly on long-horizon tasks [3]. This shows that evaluating a reward model isn't just about whether it matches human preferences at the end, but whether its intermediate signals are trustworthy—and that's much harder to verify.
What this means for evaluation: you need human judgment, not just metrics
Because preferences are subjective and multi-dimensional, evaluating a reward model requires comparing its predictions to actual human judgments—not just checking if it improves a metric. A 2026 study on video-to-audio generation introduced a human-preference-aligned scoring system (AudioScore) that assesses semantic consistency, temporal alignment, and perceptual quality, and used it to generate preference pairs for training [2]. They found that models optimized with this human-aligned reward outperformed those optimized with a generic policy optimization method, suggesting that the reward model's alignment with human preference is what matters [2].
Another 2026 study on joint video-audio generation built a benchmark (VA-Judger-Bench) with both in-domain and out-of-domain comparisons to evaluate whether reward models truly align with human preferences [4]. They found that their reward model outperformed metric baselines in predicting human preferences, and using its rewards for post-training improved generation quality [4]. The takeaway: to evaluate a reward model for long-horizon video-audio tasks, you need to test it against human preferences on holistic coherence, not just individual metrics—and that's harder to set up and measure.
About These Sources
This answer is built on 5 studies (1 peer-reviewed, 4 preprints) — published from 2025 to 2026, 5 from 2024 or later — selected as the most relevant from 10 studies that passed quality screening, drawn from 56 papers retrieved from a database of over 500 million.
Sources used in this answer
SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation
In robot manipulation, a stage-aware video-based reward model outperformed baselines on validation and real rollouts, and when integrated into behavior cloning, achieved 83% success on folding T-shirts from flattened state versus 8% for vanilla cloning, showing that progress-based rewards can be effective but require careful stage modeling.
V2A-DPO: Omni-Preference Optimization for Video-To-Audio Generation
A direct preference optimization framework for video-to-audio generation, using a human-preference-aligned scoring system (AudioScore), outperformed models optimized with denoising diffusion policy optimization and pre-trained baselines, achieving state-of-the-art performance on VGGSound.
RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
A confidence-gated progress reward model, trained on general-purpose videos, improved success rates in reinforcement learning across 9 simulated and 4 real-world manipulation tasks, with large gains on long-horizon tasks like cloth folding, by suppressing uncertain progress estimates that cause false-positive rewards.
VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
A chain-of-thought omni-reward model for joint video-audio generation, trained on a large human-preference dataset (VAPref-10K), outperformed metric baselines in predicting human preferences on in-domain and out-of-domain benchmarks, and using its rewards for post-training improved generation quality.
Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation
A group-relative preference backpropagation method for long-horizon video editing agents, which redistributes credit over semantic segments, improved editing behavior and rendered products over baselines, supporting task-local preference reduction as a practical approach to learning from subjective, delayed outcomes.