How can subjective quality be evaluated for real-time open-ended video editing?

Why human judgment is still the gold standard—and how it's being made faster

Subjective quality assessment—asking people what they think—remains the most reliable way to measure video quality, but it's slow and expensive. A 2022 study introduced a method called HA-SVQA that uses active learning to pick only the most informative videos for humans to rate, cutting the human workload by about one-third while still achieving the same accuracy as a full study [4]. This matters for real-time editing because you can't pause a live broadcast to run a lengthy survey; you need quick, reliable feedback loops.

The same principle applies to open-ended editing, where the edited video may differ from the source in unpredictable ways. A 2024 benchmark for text-driven video editing (VE-Bench) collected Mean Opinion Scores from 24 human annotators across 8 different editing models, showing that human perception is the reference point even for creative edits [5]. This confirms that no automated metric can replace human judgment entirely—it must be trained on human data to be trustworthy.

Automated metrics that learn from humans—and what they miss

To make real-time evaluation practical, researchers build automated metrics that are trained to predict human scores. The VE-Bench study proposed a new assessment network that goes beyond traditional quality indicators like distortion and aesthetics to also measure text-video alignment and how well the edited video matches the source [5]. This is crucial for open-ended editing, where the 'correct' output isn't predefined—the metric must understand the intent behind the edit.

However, even the best automated metrics have limits. A 2023 study on HDR video quality found that existing models like VMAF (Video Multimethod Assessment Fusion) needed modifications to handle the wider brightness and color ranges of HDR content, and even then, they only achieved 'significantly elevated performance'—not perfect alignment with human perception [2]. This suggests that for real-time editing, you should use automated metrics as a fast filter, but still rely on periodic human checks for final quality assurance.

What's different about real-time open-ended editing?

Real-time editing adds a time constraint that traditional quality assessment doesn't account for. A 2024 study on real-time camera editing (Real Time GAZED) showed that it's possible to produce edits in real-time that are comparable in quality to offline methods, and a user study confirmed the aesthetic quality of the results [1]. This is encouraging, but it also means that any quality evaluation must be fast enough to keep up with the live stream—a challenge that the study addressed by optimizing the editing algorithm itself, not just the evaluation.

Open-ended editing, where the user can specify arbitrary changes (e.g., 'make it look like a painting'), adds another layer of complexity. The VE-Bench study is the first to create a quality assessment dataset specifically for text-driven video editing, highlighting that traditional metrics fail to align with human perception for such tasks [5]. This means that for open-ended editing, you can't rely on generic quality metrics—you need ones that understand the semantic intent of the edit, which is an active area of research.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2022 to 2025, 3 from 2024 or later, 2 in Q1 journals — selected as the most relevant from 5 studies that passed quality screening, drawn from 36 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions