QUINCE: Revolutionizing QoE Measurement Through Gamified Crowdsourcing

TL;DR

Understanding how users perceive video quality in the wild (Quality of Experience, or QoE) is notoriously difficult due to the "noise" of different home networks and devices. QUINCE (Quality of Internet Consumer Experience) solves this by turning network testing into a longitudinal game. By keeping users engaged over weeks rather than minutes, it provides a much clearer picture of how specific network conditions—like interdomain congestion—actually impact the human viewer.

Background & Motivation: The Problem with One-Off Studies

Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) have long been used to gather subjective data. However, the standard methodology is inefficient:

• High Instructional Overhead: Users spend most of their paid time reading instructions.
• High Variance: Because users only participate once, it is impossible to distinguish if a bad experience was caused by a temporary network glitch or a permanent hardware limitation.
• Costly Scaling: Acquiring thousands of data points requires thousands of unique users, driving up recruitment costs.

The authors of QUINCE recognized that to understand the internet's complexity, we need to monitor the same users over different times and conditions.

Methodology: The QUINCE Framework

QUINCE is built on a three-pillar architecture that bridges the gap between subjective feeling and objective reality.

1. Unified Measurement Tasks

• Video QoE: Uses a custom player to simulate buffering or embeds actual YouTube streams, asking users for a 5-point Likert scale rating.
• Network Performance: Integrates M-Lab NDT and fast.com to measure available bandwidth.
• Topology: Instructs users to perform traceroutes (guided by a tutorial) to map the path data takes through the internet.

2. Gamification & Retention

To transform a boring task into a longitudinal study, QUINCE uses:

• Points & Credits: A scoring system where points translate to monetary rewards.
• Cool-down Timers: Prevents "spamming" by forcing users to wait between 2 minutes to 1 hour before repeating a task, ensuring data is spread out over time.

Figure 1: The architecture of QUINCE, showing the integration of measurement datasets, the user interface, and the backend tasks.

Experimental Insights: High Retention and Surprising Results

The pilot study recruited 70 subjects via MTurk for two one-week sessions. The results validated the "Stickiness" of the platform:

• User Engagement: 25% of participants were "power users," submitting 50+ tasks and earning 10x the minimum required points.
• Cost Efficiency: The researchers estimated that obtaining the same volume of data through traditional one-off sessions would have cost over double ($1,800vs.$700).

The QoE/Throughput Paradox

One of the most interesting findings from the QUINCE pilot involved YouTube streaming. Even for users with high-speed links (>50 Mbps), QoE ratings occasionally dropped below 3 (Bad/Fair). This suggests that raw bandwidth is a poor predictor of experience; local network congestion or interdomain bottlenecks often play a larger role than "last mile" speed.

Figure 2: CDF of user scores, demonstrating that the gamified system successfully retained users for repeated measurements.

Conclusion and Future Outlook

QUINCE demonstrates that the "human in the loop" doesn't have to be a source of noise. By treating human subjects as persistent measurement agents, the platform allows researchers to correlate objective topology changes and congestion events with subjective dissatisfaction.

Future Work: The team plans to scale QUINCE to investigate how interdomain congestion (the traffic handoffs between major ISPs) affects popular services beyond YouTube, potentially providing a roadmap for ISPs to optimize traffic based on actual user experience rather than just packet loss or throughput.

Takeaway for the Industry: For developers and researchers building QoE-sensitive applications, QUINCE proves that longitudinal, gamified crowdsourcing is the most cost-effective way to gain high-fidelity insights into the "real-world" internet experience.