Why subjective quality isn't one number—and how to capture it
Subjective quality is what a human viewer perceives, but it's rarely a single score. In a 2024 study of colonoscopy videos, researchers asked human observers to rate videos on five distinct distortion attributes—temporal-spatial visibility, brightness, specular reflection, stability, and utility—plus an overall quality score [1]. This shows that a single 'quality' rating can hide important differences: a video might be bright but unstable, or clear but with distracting reflections. For agentic video understanding, you need to evaluate each dimension that matters for the agent's task, not just an average.
The same study built a model called DARNet that automatically predicts these attribute scores, and it correlated well with human ratings, outperforming nine existing methods [1]. This suggests that a well-designed automated model can approximate human subjective judgments, but only if it's trained on human-annotated data that captures the multi-attribute nature of quality.
The winning formula: human ratings + AI judges + task performance
For agentic video understanding, you can't rely on human ratings alone—they're expensive and slow—but you also can't trust a single automated metric. A 2026 framework for generating rare-event synthetic images used a multi-pronged evaluation: objective no-reference image quality metrics, a panel of large multimodal model (LMM) evaluators, human ratings, and downstream zero-shot object detection [2]. The study found that different models excelled on different criteria—Gemini 3 Pro Image led on perceptual quality, while OpenAI-generated images achieved stronger recall and mAP (mean average precision) in object detection [2]. This is a crucial lesson: a video might look good to a human but fail at the agent's actual task, or vice versa. So you need both perceptual and task-aware metrics.
This multi-metric approach is echoed in other domains. For high dynamic range (HDR) videos, researchers collected over 20,000 human quality judgments under two illumination conditions and then built a model (HDRMAX) that improved on the widely-used VMAF (Video Multimethod Assessment Fusion) metric [3]. Similarly, a study on free viewpoint videos (FVVs) created a large database with human QoE (quality of experience) ratings and trained a no-reference prediction model [4]. And a study on space-time subsampled videos collected about 15,000 subjective judgments to understand how resolution and frame rate reductions affect perceived quality [5]. Across these studies, the pattern is consistent: human subjective data is the gold standard, but you need automated models that can predict it, and you need to validate those models on task-specific outcomes.
How to actually do it: a practical checklist
Start by defining what 'quality' means for your agent's task. If the agent is analyzing medical videos, use attributes like visibility and utility, as in the colonoscopy study [1]. If it's generating synthetic training data, include detectability as a quality criterion, as in the rare-event framework [2]. Then collect human ratings on a representative sample of videos—even a few hundred can be enough, as the studies here used databases of 1,000 colonoscopy videos [1], 310 HDR videos [3], and 437 subsampled videos [5].
Next, train or select an automated model that predicts those human ratings. The studies show that models like DARNet [1] and HDRMAX [3] can achieve high correlation with human judgments. But don't stop there—evaluate the agent's actual performance on the task, using metrics like detection recall or mAP [2]. Finally, use a panel of LMM evaluators as a scalable proxy for human judgment, but validate them against human ratings first [2]. This multi-layered approach gives you a robust, defensible evaluation of subjective quality for agentic video understanding.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2026, 2 from 2024 or later, 5 in Q1 journals, collectively cited 95 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 46 papers retrieved from a database of over 500 million.
Sources used in this answer
Subjective and Objective Quality Assessment of Colonoscopy Videos
Collected 1,000 colonoscopy videos with human ratings on five distortion attributes plus overall quality, and built DARNet, which predicted these ratings better than nine existing methods.
Agentic ontology-guided image generation and evaluation for rare-event data augmentation in safety-critical perception
Proposed an agentic framework for rare-event synthetic image generation that used LMM-based evaluators, human ratings, and downstream object detection; found that perceptual quality and task-aware detectability can diverge across models.
A Study of Subjective and Objective Quality Assessment of HDR Videos
Created the LIVE HDR database with 310 HDR videos and over 20,000 human quality judgments under two illumination conditions, and developed HDRMAX, which improved VMAF's performance on both HDR and SDR videos.
Subjective and Objective Quality of Experience of Free Viewpoint Videos
Built the largest free viewpoint video QoE database (Youku-FVV) from real and virtual viewpoints, and trained a no-reference QoE prediction model that balances effectiveness and efficiency.
A Subjective and Objective Study of Space-Time Subsampled Video Quality
Constructed the ETRI-LIVE STSVQ database with 437 videos and about 15,000 subjective judgments to study the perceptual impact of space-time subsampling and compression, and evaluated leading video quality models on it.