Why human ratings are the gold standard—and how to collect them
When you ask 'how good does this video look?', the only truly reliable answer comes from people. Subjective quality assessment means collecting human opinions, typically as Mean Opinion Scores (MOS) from a panel of viewers. A 2023 study on HDR videos gathered over 20,000 human judgments from 310 videos under two different lighting conditions, showing that human ratings capture perceptual differences that simple technical metrics miss [1]. For AI-generated content, human ratings are equally essential—VBench, a 2024 benchmark, built a dataset of human preference annotations specifically to validate that its automated metrics align with what people actually perceive [5].
The catch is that human ratings are expensive and slow. That's why researchers build automated models that try to predict human scores. But the key is calibration: any automated metric should be tested against human ratings before you trust it. VBench does this for each of its 16 quality dimensions, ensuring that a high automated score actually corresponds to what a human would say [5].
Don't ask 'is it good?'—ask 'what's wrong with it?'
Subjective quality isn't a single number. For video generation, you need to separate technical quality (noise, blur), motion quality (smoothness, jitter), and semantic quality (does it match the prompt and tell a coherent story?). VBench dissects video generation quality into 16 specific dimensions, including subject identity consistency, motion smoothness, temporal flickering, and spatial relationships [5]. This granular approach lets you see exactly where a model fails—e.g., a video might have perfect frames but jittery motion, or vice versa.
A 2025 model called AIGVEval takes a similar approach, decomposing quality into technical, motion, and semantic dimensions, and then uses a large language model to reason about them [2]. This multi-dimensional view is crucial for semantic completion because a video can be technically flawless but semantically wrong—e.g., a cat that turns into a dog mid-clip. By separating dimensions, you can pinpoint whether the semantic completion failed or the rendering did.
Automated models are getting better, but they still can't fully replace humans
You might want an automated way to evaluate quality without running human studies. The good news: multimodal large language models (MLLMs) are emerging as powerful tools for this. A 2025 perspective argues that MLLMs can jointly encode vision, language, and even sound to assess scene composition, motion dynamics, and narrative consistency—overcoming the limitations of hand-engineered metrics and CNN-based methods [3]. AIGVEval, which uses an LLM as the quality regression module, achieved second place in the NTIRE 2025 AI-Generated Video Quality Assessment challenge, showing that LLM-based approaches can compete with top methods [2].
But there's a clear caveat: even the best automated models don't perfectly match human perception. A 2025 study on AI-generated images found that off-the-shelf quality models performed poorly, and even GPT-4V (a vision-enabled LLM) struggled to produce alignment scores that correlate with human judgments [4]. However, fine-tuning learning-based models on task-specific data led to 'significant gains' [4]. So the practical advice is: use automated models as a screening tool, but validate them against human ratings for your specific use case—especially if semantic alignment is critical.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2025, 4 from 2024 or later, 1 in Q1 journals, collectively cited 170 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 41 papers retrieved from a database of over 500 million.
Sources used in this answer
A Study of Subjective and Objective Quality Assessment of HDR Videos
Created the first public HDR video quality database with 310 videos and over 20,000 human judgments under two illumination conditions, and showed that a modified VMAF model (HDRMAX) improved quality prediction on both HDR and SDR videos.
Towards Holistic Visual Quality Assessment of AI-Generated Videos: A LLM-Based Multi-Dimensional Evaluation Model
Proposed AIGVEval, a multi-dimensional evaluation model that decomposes AI-generated video quality into technical, motion, and semantic dimensions and uses a large language model with LoRA fine-tuning; it achieved second place in the NTIRE 2025 AI-Generated Video Quality Assessment challenge.
A Perspective on Quality Evaluation for AI-Generated Videos
Argues that multimodal large language models (MLLMs) are becoming the cornerstone of next-generation video quality assessment because they can jointly encode vision, language, sound, and depth to assess scene composition, motion dynamics, and narrative consistency.
Subjective and Objective Quality Assessments of AI-Generated Images for Language E-Learning
In a study of AI-generated images for language e-learning, subjective ratings showed higher perceptual quality correlated with stronger semantic alignment, but off-the-shelf quality models performed poorly, and GPT-4V struggled to produce alignment scores that correlate with humans; fine-tuning learning-based models led to significant improvements.
VBench: Comprehensive Benchmark Suite for Video Generative Models
Introduced VBench, a benchmark suite that dissects video generation quality into 16 dimensions (e.g., subject identity, motion smoothness, temporal flickering) and provides human preference annotations to validate each dimension's alignment with human perception.