[SocialSens] Less is More: Why Confidence Reporting Might Be Useless in Crowdsourcing

TL;DR

In a world where we strive for more data granularity, this paper provides a sobering, counterintuitive reality check: asking crowd workers how "confident" they are might be a waste of time. By deriving an optimal mathematical aggregation rule for $M$-ary classification, the authors prove that while a reject option (allowing workers to skip tasks) is vital, quantized confidence scores add more noise than value.

Context & Positioning

Crowdsourcing is the backbone of modern AI (think RLHF or image tagging), but worker quality is notoriously variable. This work sits at the intersection of Information Fusion and Human-Centered Computing, specifically refining how we aggregate "wisdom of the crowd" when workers are allowed to say "I don't know."

The Problem: The Noise of "Sureness"

Existing systems often use simple majority voting or force binary choices. When workers are unsure, they guess, injecting pure noise into the system. While "confidence reporting" seems like a logical fix, humans are bad at quantifying their own internal probability. They categorize their confidence into 3, 5, or 7 levels (quantization). The authors argue that this quantization introduces quantization noise that can actually mislead the fusion center.

Methodology: The "Optimal Weight" Discovery

The researchers sought a weight assignment ($W_w$) for each worker that maximizes the probability of correct classification.

The Intuition

Through a formal proof (see Equation 14 in the paper), they discovered that the optimal weight for a worker is: $W_w={\mu }^{-n}$ Where:

• $\mu$ is the mean accuracy of the crowd.
• $n$ is the number of definitive answers provided by the worker.

The Insight: The "quantity" of questions a worker chooses to answer (given a reject option) is a more reliable proxy for their "quality" than the confidence score they self-report. A high-quality worker naturally answers more questions.

The indicator-based aggregation rule used to sum weights across different classes.

Experimental Validation

The paper compares three ways to estimate the worker quality parameter $\mu$:

1. Training-based: Using "gold standard" questions with known answers.
2. Majority-voting-based: Iteratively estimating quality from crowd consensus.
3. Confidence-based: Extracting quality from reported confidence levels.

Key Findings

As shown in the figures below, the confidence-based method (Fig 2a & 2b) consistently underperformed or saturated quickly. Even when increasing the confidence levels ($L$) to 8—the practical limit of human cognition—it couldn't beat simple training-based estimation.

Figure 2: Note how the confidence-based method (green line) saturates and fails to match the accuracy of training-based methods.

Furthermore, the proposed weighted voting with a reject option (Fig 1) showed a massive leap over simple majority voting, proving that who answers and how many they answer is the secret sauce.

Figure 1: Accuracy improvement ($P_c$) as the number of workers increases, highlighting the superiority of the weighted reject-option approach.

Critical Analysis & Conclusion

Why does confidence fail?

1. Quantization Error: Converting a continuous internal feeling of "82% sure" into a "Level 4" response loses critical precision.
2. Redundancy: In an optimal aggregation framework, the fact that a worker chose to answer a binary question already captures the most significant part of the "confidence" signal.

Takeaway for System Designers

If you are building a crowdsourcing pipeline:

• DO: Give workers a "Skip" or "Reject" button. It filters out low-information guesses.
• DON'T: Waste UI space or worker cognitive load on "How confident are you (1-5)?" sliders. The math suggests you won't use that data effectively anyway.

Future Outlook

This study focuses on "honest" workers. A fascinating next step would be analyzing how malicious or "lazy" workers might use confidence reporting to intentionally game the weight assignment, further strengthening the case for omitting it.