Beyond Binary: Managing Crowdsourcing Ambiguity with Interval-Valued Labels

TL;DR

In the world of crowdsourcing, we usually ask workers for a "Yes" or a "No." But human intuition isn't binary—it's fuzzy. This paper proposes Interval-Valued Labels (IVLs), allowing workers to express uncertainty as a range (e.g., "I'm 70% to 80% sure this is a 'Yes'"). This shift from points to intervals prevents information loss and enables a sophisticated probabilistic inference that outperforms traditional Majority Voting.

The Problem: The "Forced Choice" Trap

Crowdsourced data is the lifeblood of modern AI, yet it is notoriously noisy. Current systems typically compress human judgment into binary logic:

• Physical constraints: Is this a cat? (1 or 0).
• The issue: A worker might be uncertain due to poor image quality or lack of expertise. If forced to choose "1" or "0," their internal uncertainty—a valuable data point—is destroyed.
• Prior Work: Methods like "soft labels" or behavior prediction have tried to fix this, but they often rely on post-hoc guesses about worker reliability rather than capturing uncertainty at the moment of input.

Methodology: The Power of Interval Arithmetic

The core insight of this paper is that an interval contains more "signal" than a single point. If a worker provides a label $l_{ij}=[l_{lower},l_{upper}]$, they are providing a centroid (the belief) and a radius (the uncertainty).

1. Defining the IVL

An IVL is a sub-interval of $[0,1]$.

• Midpoint ($mid$): If $mid>0.5$, it's a positive lean; if $mid<0.5$, it's negative.
• Radius ($rad$): Represents the maximum variation or doubt.
• PDF Generation: Unlike binary labels that follow Bernoulli trials, IVLs allow for the construction of a continuous Probability Density Function (PDF) across all worker responses for a single task.

2. The Inference Algorithm

The authors propose Algorithm 2, which aggregates these intervals into a unified PDF $f(t)$. The final decision is made by calculating: $p^{+}={\int }_{0.5}^{1}f(t)dt$ If $p^{+}>0.5$, the system infers a "1". This ensures that the matching probability is always optimized based on the full distribution of crowd belief.

The algorithm for calculating the aggregate PDF from a multiset of interval labels.

Experiments and Results

The researchers compared their IVL inference against the industry-standard Majority Voting (MV).

Breaking the Ties

Majority Voting often hits a "tie" when votes are split. IVLs can break these ties by looking at the width and position of the intervals. Even if the count of positive/negative votes is equal, the ranges might lean statistically toward one side.

Resilience to Bias

As shown in the experimental results, when biased labels are introduced, the IVL method effectively shifts the matching probability, providing a quantitative metric for reliability.

Distribution of matching probability with and without bias. Biased labels shift the probability density, allowing for better management of skewed crowd input.

The Uncertainty Index ($\zeta$)

The paper introduces $\zeta =1-\max \{p^{+},1-p^{+}\}$. This index acts as a "health check" for data collection. If $\zeta$ is near 0.5, the crowd is completely confused, and the system knows it needs more labels to reach a confident conclusion.

Critical Insight & Conclusion

The true value of this work lies in its Inductive Bias: it assumes that human error is not just random noise to be filtered, but a structured uncertainty to be modeled.

Limitations: The paper primarily uses uniform distributions for intervals. In real-world scenarios, human uncertainty might follow a "Beta" or "Normal" distribution within the interval. Furthermore, the "collusion attack" (where workers coordinate to provide the same biased intervals) remains a challenge that requires separate adversarial detection.

Future Outlook: As we move toward more complex AI alignment tasks (like RLHF), moving away from "A vs B" testing toward "Interval-scale" feedback could significantly reduce the cost of reaching a high-confidence ground truth.