Quadrant of Euphoria: Revolutionizing QoE Assessment via Paired Comparison and Crowdsourcing

TL;DR

"Quadrant of Euphoria" is a specialized platform designed to move Quality of Experience (QoE) testing out of the expensive laboratory and into the global "crowd." By utilizing paired comparisons instead of traditional 1-5 ratings and implementing a Transitivity Satisfaction Rate (TSR) to prune dishonest data, it achieves results comparable to lab studies at a fraction of the cost.

Problem & Motivation: The Flaws of the Mean Opinion Score (MOS)

For decades, the Mean Opinion Score (MOS) has been the gold standard for subjective quality assessment. However, the authors argue it is fundamentally flawed due to two reasons:

1. Scale Heterogeneity: One person's "Good" (4) is another person's "Fair" (3). There is no objective baseline for these labels.
2. Scale Ordinality: The "distance" between 1 and 2 isn't necessarily the same as between 4 and 5, yet researchers treat them as interval data by calculating arithmetic means.

Furthermore, conducting these tests in a lab is expensive, slow, and lacks diversity. Labs often use "professional" subjects who don't represent the average global user.

Methodology: The Core Insight

The authors propose a dual-solution: Methodological Reform and Operational Reform.

1. Paired Comparison & BTL Model

Instead of asking "How good is this video?", the platform asks "Which of these two is better?". This dichotomous choice is less burdensome for the user. These binary results are then transformed into a ranking through the Bradley-Terry-Luce (BTL) model, which estimates the probability $\pi (T_i)$ of a stimulus being preferred.

2. Crowdsourcing with TSR Filtering

To use the "Internet crowd" reliably, the platform must handle "bogus" or "random" clickers. The authors introduce the Transitivity Satisfaction Rate (TSR). If a user says A > B and B > C, but then says C > A, they have violated the transitive property of preference.

Figure 1: The Quadrant of Euphoria role-based layout and workflow.

Experiments & Results

The platform was tested on four scenarios, including VoIP packet loss and IPTV concealment.

• Cost Efficiency: Lab tests cost roughly $1.54perrunonaverage,whereasthecommunity-basedcrowdcostonly$0.07.
• Data Fidelity: Despite the "noisy" nature of the internet, the results from the crowd (after filtering for TSR > 0.8) were statistically equivalent to the controlled laboratory environment.
• Consistency: The average TSR across all qualified results reached 0.97, proving that the filtering mechanism effectively salvages the "wisdom of the crowd."

Figure 2: Performance comparison showing statistically equivalent QoE scores between lab, MTurk, and community subjects.

Critical Analysis & Conclusion

Takeaway: The "Quadrant of Euphoria" framework successfully bridges the gap between theoretical accuracy and practical scalability. It proves that we can trust the crowd if we change how we ask them (Paired Comparison) and how we verify them (Transitivity).

Limitations:

• The methodology currently focuses on static or short-duration stimuli (image/audio/video).
• Interactive applications (like cloud gaming or real-time conferencing) present higher complexity for paired comparisons since the user's own interaction changes the stimulus.
• Demographic Uncertainty: While diverse, the virtual nature of the internet makes it hard to verify age or race-specific preferences with 100% confidence.

Future Outlook: As we move toward more complex multimedia, extending this framework to support interactive streaming and multidimensional consistency metrics will be key to scaling user-centric research.