[IEEE TMM] Beyond the MOS: Building a Trusted Framework for Crowdsourced Multimedia QoE

TL;DR

Evaluating the "Quality of Experience" (QoE) of multimedia has traditionally been a slow, expensive laboratory process using the Mean Opinion Score (MOS). This paper presents a paradigm shift: a crowdsourceable framework that uses Paired Comparisons and a Transitivity Satisfaction Rate (TSR) to detect "cheaters." The result? A system that is up to 66 times cheaper than laboratory tests while delivering identical accuracy (0.99 correlation).

The Problem: Why Lab-based MOS is Failing Modern Research

For decades, the gold standard for multimedia quality has been the Mean Opinion Score (MOS). Participants sit in a quiet room and rate a clip from 1 (Bad) to 5 (Excellent). However, this methodology faces three lethal flaws:

1. Scale Mapping Uncertainty: What is "Good" to one person might be "Fair" to another.
2. High Economic Cost: Hiring participants and setting up controlled labs limits the diversity and size of the test group.
3. The "Cheating" Dilemma: In an unsupervised crowdsourcing environment (like Amazon Mechanical Turk), participants often click randomly just to get paid. MOS has no internal mechanism to catch these "perfunctory" answers.

The Methodology: The Power of Transitivity

Instead of asking "How good is this video on a scale of 1 to 5?", the authors ask the crowd a simpler question: "Which of these two clips is better?"

1. Paired Comparison & Probabilistic Modeling

By using binary choices, the cognitive load on the user is slashed. To turn these relative "wins/losses" into a final score, the authors use the Bradley-Terry-Luce (BTL) model. This mathematical model predicts the probability that quality level $i$ is better than $j$ based on an underlying interval-scale score.

2. The Cheat Detector: TSR

The "secret sauce" of this framework is the Transitivity Satisfaction Rate (TSR). In a consistent logical mind, if:

• Video A > Video B
• Video B > Video C
• Then Video A MUST > Video C.

If a participant claims Video C is better than Video A, they have violated the transitivity rule. The algorithm calculates a TSR score for every participant. If the TSR falls below 0.75, their entire contribution is discarded.

Figure 1: The dual-state user interface allowing users to switch between stimuli seamlessly.

Experimental Battleground: Lab vs. Crowd

The authors conducted four extensive case studies involving MP3 bitrates, VoIP packet loss, video codecs (H.264 vs. WMV3), and loss concealment schemes.

Key Findings:

• Outcome Consistency: The correlation between the new Paired Comparison (PC) scores and traditional MOS was a staggering 0.99.
• Qualified Rate: In the lab, 70% of people are "honest." In the crowd, that drops to roughly 50%. However, because the TSR filter automatically removes the 50% of "trash" data, the final results remain incredibly precise.
• Convergence Speed: Paired comparisons were found to reach stable, reliable scores faster than MOS with the same amount of human effort (man-minutes).

Figure 2: The convergence rate shows that Paired Comparison (PC) provides more reliable estimates with less human effort compared to MOS.

Deep Insight: Is the Crowd Actually Better?

The paper argues that while lab environments are controlled, they are "artificial." Crowdsourcing captures the diversity of real-world environments—different headsets, lighting, and ambient noise. By using the TSR to prune malicious actors, we get the best of both worlds: the scale of the Internet and the reliability of a lab.

Limitations & Future Work

The primary drawback of Paired Comparison is the $O(n^2)$ complexity (N items require many pairs). While the authors suggest that random partial pairs can mitigate this, the computational overhead of calculating the BTL model for massive datasets remains an area for optimization.

Takeaway

This framework proves that mathematical constraints (transitivity) can replace physical supervision. For engineers and researchers, this opens the door to massive-scale subjective testing that was previously economically impossible.