Turning Noise into Signal: Mitigating Sloppiness in Crowd Scoring Tasks

TL;DR

In the world of crowdsourcing, we often discard inaccurate workers as "spammers." This paper argues that many inaccurate workers are actually "sloppy"—their errors follow a predictable bias. By introducing ITSC-TD (Iterative Self-Correcting Truth Discovery), the authors demonstrate that detecting and correcting these systematic biases can boost truth-discovery accuracy by up to 16% in complex ordinal scoring tasks.

The "Sloppiness" Problem: Beyond Binary Zeroes and Ones

Most research in crowdsourcing focuses on binary classification (Yes/No). However, real-world tasks—like grading an essay or rating a product—are scoring tasks with ordinal scales (e.g., 1 to 5).

The authors identify a specific class of "unreliable" worker: the Sloppy Worker. Unlike a spammer who provides random noise, a sloppy worker's judgments fluctuate around the truth. Specifically, Biased Sloppy Workers consistently shift their scores (e.g., always giving a "4" when the truth is "3"). Standard models like Majority Voting (MV) or Expectation Maximization (EM) treat these workers as low-quality, but this paper realizes they are actually high-information sources—if you can identify the shift.

Methodology: The ITSC-TD Framework

The core of the paper is a two-step iterative process that doesn't requires "Gold Truths" (expert-verified labels) to work.

1. Bias Detection and Correction

Using a vanilla Bayesian estimation, the model treats worker accuracy ($het{a}_j$) and bias (${\alpha }_j$, ${\beta }_j$) as distributions. It calculates a Bias Score (BS) derived from Information Gain.

• Insight: If a worker has a high Bias Score and a high error rate, the model identifies them as "Biased Sloppy."
• Correction: The model "de-biases" these workers by shifting their observed labels (e.g., subtracting 1 from all their scores) before the next phase.

2. Optimization-Based Truth Discovery

Once labels are corrected, the model uses an optimization framework to minimize the weighted deviation between the (now corrected) observations and the hidden true labels.

Figure 1: The general graphical model for crowdsourcing aggregation, showing the relationship between true labels ($v^{\ast }$), observed labels ($v$), and worker reliability ($\omega$).

Experimental Proof: When Sloppiness Dominates

The authors tested ITSC-TD against heavyweights like GLAD and EM-DS (Dawid & Skene).

The "Knee Point" of Performance

The most striking discovery was that when the proportion of biased workers exceeds 50%, standard models' performance collapses. In contrast, ITSC-TD maintains high accuracy because it effectively "recruits" the biased workers into the reliable pool by correcting their scores.

Figure 2: Influence of sloppy workers on consensus labels. As the proportion of perfectly erred workers increases, the gap between traditional methods and corrected models widens.

Real-World Validation

On real datasets like TREC (relevance judgments) and AC2 (adult content filtering), the model successfully identified dozens of biased workers. In the TREC dataset, it achieved a 2-8% improvement in accuracy over traditional EM-based methods, proving that "sloppiness" is a tangible factor in large-scale human labeling.

Critical Insight & Future Work

The fundamental takeaway is that accuracy is not the only metric for worker value. A worker who is 100% wrong but always "off by one" is just as valuable as a worker who is 100% right.

However, the paper acknowledges a limitation: it assumes the bias is constant across all items. Future extensions could investigate task-dependent bias, where a worker might be "generous" when grading difficult items but "strict" on easy ones. This work paves the way for more resilient human-in-the-loop systems that can extract truth from even the most "sloppy" crowds.

Summary of Gains (TREC Dataset):

• Accuracy: +2.0% - 8.0%
• F1 Measure: +4.0% - 9.0%