What does 'reliable' actually mean for a creative team?
For a creative team, reliability means the reward model's scores consistently match what a human would prefer, not just what a metric says is good. The papers show that current metric-based rewards (like audio quality or visual fidelity) often fail to capture the overall coherence between text, video, and audio that humans care about [2]. In fact, optimizing against these metrics can lead to 'reward hacking'—generating content that scores high but looks or sounds wrong to people [2].
A more reliable approach is to train reward models directly on human preferences. The VA-Judger paper built a large-scale dataset of 10,300 human preference comparisons and showed that their model outperformed metric baselines in predicting human preferences [2]. This suggests that reliability improves when the reward model is trained on what humans actually prefer, not on hand-crafted metrics.
How much improvement in generation quality can you expect?
When reward models are well-aligned with human preferences, they can lead to significant quality gains. MJ-Video, a fine-grained video reward model, improved overall preference judgment by 17.58% and fine-grained judgment by 15.87% over existing models [1]. In plain terms, that means the model was much better at picking the video a human would prefer, and when used for fine-tuning, it improved alignment in video generation [1].
Similarly, V2A-DPO, which uses human-preference-aligned scoring for video-to-audio generation, achieved state-of-the-art performance on multiple metrics, beating both pre-trained baselines and models optimized with other methods [3]. So, when the reward model is well-designed, the payoff is real—but it depends on the model being trained on high-quality human feedback.
The catch: reliability drops outside the training zone
The biggest risk is that a reward model works well on the data it was trained on but fails on new, unseen prompts or styles. The VA-Judger paper specifically tested both in-domain and out-of-domain scenarios, and while their model did better than baselines, the fact that they needed a separate benchmark for out-of-domain evaluation shows this is a real concern [2]. Creative teams often work with novel ideas, so a reward model that only works on 'typical' prompts may not be trustworthy for their unique needs.
Another issue is noise in human labels. The Atari study, though not about video-audio, found that human preference labels can be noisy, and they studied the effects of that noise on reward learning [4]. This is a reminder that even human feedback isn't perfect—so the reward model's reliability is limited by the quality of the human annotations it's trained on.
About These Sources
This answer is built on 4 studies (2 peer-reviewed, 2 preprints) — published from 2022 to 2026, 3 from 2024 or later — selected as the most relevant from 4 studies that passed quality screening, drawn from 35 papers retrieved from a database of over 500 million.
Sources used in this answer
MJ-Video: Benchmarking and Rewarding Video Generation with Fine-Grained Video Preference
MJ-Video, a mixture-of-experts video reward model, improved overall and fine-grained preference judgment by 17.58% and 15.87% respectively on the MJ-Bench-Video benchmark, and improved video generation alignment via preference fine-tuning.
VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
VA-Judger, a chain-of-thought omni-reward model trained on 10,300 human preference comparisons, outperformed metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations, and using its rewards for post-training improved generation quality.
V2A-DPO: Omni-Preference Optimization for Video-To-Audio Generation
V2A-DPO, a direct preference optimization framework with a human-aligned scoring system, achieved state-of-the-art performance on multiple metrics for video-to-audio generation, surpassing pre-trained baselines and models optimized with DDPO.
Reward learning from human preferences and demonstrations in Atari
In Atari games, combining human demonstrations and trajectory preferences to learn a reward model beat imitation learning in 7 of 9 games and achieved superhuman performance in 2 games, but also highlighted reward hacking problems and the impact of noise in human labels.