Approval Voting in Crowdsourcing: Eliciting the "Hidden" Knowledge of the Crowd

TL;DR

Standard crowdsourcing tasks force workers to pick a single "best" answer, even when they are torn between two options. This paper by Shah and Zhou proves that by switching to an Approval Voting interface—where workers can select multiple "likely" candidates—and applying a specific multiplicative scoring rule, we can capture significantly more information from the crowd while ensuring they are incentivized to be 100% honest.

Background: The "Single-Selection" Trap

In the big data era, we rely on platforms like Amazon Mechanical Turk to label millions of images. Usually, we ask: "Is this a Leopard, a Jaguar, or a Cheetah?" A worker might know for sure it isn't a Cheetah, but be 50/50 on Leopard vs. Jaguar. In a traditional system, they flip a coin. We lose the information that they knew it wasn't a Cheetah.

The authors argue that this occurs due to two failures:

1. Interface Failure: Forcing a single choice.
2. Incentive Failure: Payments don't reward "reporting confusion" accurately.

The Core Mechanism: Two Ways to Vote

The paper introduces two ways to define what it means for an option to be "likely enough" to be selected:

1. Absolute Thresholding (The Additive Approach)

A worker selects every option where their internal belief $p$ is higher than a fixed threshold $\sigma$ (e.g., 20%). The authors propose an additive scoring rule where selecting the correct answer gives a reward, and every selected option incurs a small "cost" based on $\sigma$.

2. Relative Thresholding (The Multiplicative Approach)

This is the more robust "Relative" version. A worker selects the most likely option, then the second, and so on, as long as the next option's probability is a significant fraction ($\rho$) of the total belief captured so far.

The Magic Formula (Scoring Rule 2): $f^{\ast }(x_1,\dots ,x_G)={\kappa }^{\ast }{\prod }_{i=1}^G(1-\rho {)}^{|x_i|}\cdot 1\{x_i\ge 1\}$

• Translation: Your bonus starts high and is multiplied by a fraction (e.g., 0.9) for every extra option you select. If you select the wrong set (correct answer not included), your bonus for that task drops to zero immediately.

Figure 1: Comparison between (a) standard single-selection and (b) the proposed approval-voting interface.

Why is this "Unique"?

The authors prove a powerful "No-Free-Lunch" property. Their multiplicative rule is the only one that ensures if a worker's responses are consistently wrong, they get the minimum payment, while still being "strictly proper" (meaning the worker maximizes their pay only by telling the truth).

Experiments: Real-World Validation

The researchers ran three experiments on Mechanical Turk: identifying animals, textures, and languages.

Key Findings:

• Worker Flexibility: In 40% of cases, workers chose to select more than one option, proving they had partial knowledge they wanted to share.
• Incentives Matter: Workers under the "Strictly Proper" rule behaved differently and more carefully than those on fixed salaries (verified by Hotelling’s T-squared test with $p<0.01$).
• No Pushback: Despite the complex math behind the scenes, workers found the interface intuitive and provided positive feedback.

Figure 2: Distribution of worker responses. Note the significant portion of "Approval" (+2, +3) responses.

Critical Insight & Future Work

The "coarse belief" assumption (Definition 4) is a brilliant nod to human psychology—acknowledging that humans cannot distinguish between 0.001 and 0.002 probability, but can distinguish between "zero" and "likely."

The Limitation: While this paper solves the elicitation (getting data in), it leaves open the aggregation (combining data from 10 different workers). Future researchers need to build new versions of the Dawid-Skene model to handle this richer, multi-label "approval" data.

Summary

If you are building a data labeling pipeline, stop forcing your workers to guess. By allowing "Approval Voting" and using a multiplicative penalty for each additional choice, you get cleaner data and happier, more honest workers.