Quizz: Turning the Billion-User Ad Ecosystem into an Expert Knowledge Factory

TL;DR

Researchers have developed Quizz, a system that ditches the "penny-for-your-thoughts" model of paid crowdsourcing in favor of a gamified, ad-driven approach. By leveraging the targeting power of Google AdWords and a sophisticated MDP-based question engine, Quizz identifies niche experts across the web and extracts high-quality knowledge at a lower cost than Amazon Mechanical Turk.

Context: The Crowdsourcing Paradox

Crowdsourcing typically falls into two camps: the "Lightning in a Bottle" success of Wikipedia (highly motivated volunteers but hard to replicate) and the "Digital Sweatshop" of Mechanical Turk (predictable but often low-quality and non-expert).

The authors identify a critical gap: if you need a Swahili speaker or a heart surgeon, you won't find them on a standard labor marketplace. However, these experts do browse the web. The key is to find them where they live and give them a reason to click.

Methodology: Ads as a Sourcing Engine

The breakthrough of Quizz isn't just the "quiz" mechanic—it’s the closed-loop feedback with the advertising ecosystem.

1. Targeted Recruitment: Quizz runs ads for specific topics (e.g., "Test your Medical Knowledge").
2. Information-Theoretic Measurement: Instead of just counting right/wrong answers, the system treats each user as a "noisy channel" and calculates the Expected Bayesian Information Gain.
3. The Feedback Loop: Quizz reports the "value" of each user back to the ad network's conversion optimizer. If a user from a specific medical blog provides high-information answers, the ad network automatically starts showing the ad to more people on that site.

The Question Engine (MDP)

To maximize utility, the system must decide: "Should I ask a question I know the answer to (Calibration) to check if this user is a genius, or a question I don't know the answer to (Collection) to actually get work done?"

Figure 1: The Quizz Interface - Balancing assessment and data acquisition.

Experimental Evidence: Expertise via Self-Selection

One of the most profound findings is the Self-Selection Effect. In paid crowdsourcing, low-quality workers stay for the money. In Quizz, unpaid experts stay because they enjoy the challenge, while those who find it too hard drop out naturally.

Table 1: Cost and capacity analysis across various specialized domains.

Key results include:

• Cost Efficiency: Amortized cost per fact @ 99% accuracy was ~$0.16.
• Speed: Users recruited through ads answered 6x faster than paid workers, suggesting they already knew the material and didn't need to "Google" the answers.
• Incentives: "Showing the correct answer" was the strongest motivator for continued participation, suggesting a desire for learning over competition (leaderboards actually discouraged some users).

Deep Insight: Beyond Manual Keywords

While manual keyword targeting is a start, the authors found that the Ad Optimizer found experts in places they didn't expect. For medical tasks, it moved past general keywords to specific health forums (Mayo Clinic, HealthLine), identifying a population of "prosumers"—knowledgeable consumers who are often more willing to help than busy professionals.

Figure 2: Quality of submissions vs. participation. Notice how heavy participants (experts) maintain ultra-high accuracy.

Critical Analysis & Conclusion

Quizz represents a paradigm shift. It moves crowdsourcing from a "labor market" problem to a "marketing and optimization" problem.

Takeaway: If you want to build a high-quality knowledge base, don't pay people to care. Find people who already care and give them a platform to prove it.

Limitations: The system relies on the existence of ad-supported expert communities and a budget to seed the "conversion optimizer." It also requires at least some "ground truth" (calibration questions) to start the assessment process.

Future Work: This framework could theoretically be applied to Citizen Science or building massive instruction-tuning datasets for LLMs, where domain expertise is currently the primary bottleneck.