TRR: Eliminating Waste in Crowdsourcing via Adaptive Consensus Detection

TL;DR

Crowdsourcing efficiency is often hampered by "Task Redundancy"—the habit of asking too many people the same easy question. TRR (Task Redundancy Reducer) solves this by treating task assignment as an iterative process. By measuring the diversity of worker opinions, TRR stops assigning tasks the moment a consensus is reached, slashing costs by up to 62% without sacrificing the accuracy of the final answer.

The "One-Size-Fits-All" Flaw in Modern Crowdsourcing

In the typical crowdsourcing workflow (e.g., Amazon Mechanical Turk), a requester sets a fixed redundancy—say, 10 workers per task. However, tasks vary wildly in difficulty:

• Easy: "Is this a picture of a cat?" (Might only need 2 workers).
• Hard: "Is this entity 'Apple Inc.' or 'Apple Corps'?" (Might need 10+ workers).

Existing SOTA methods often rely on complex Bayesian priors or worker reputation scores that aren't always available. Furthermore, most models only handle simple binary (Yes/No) questions, leaving more complex Rating Tasks (e.g., "Rate this headline's anger level from 0-100") in the dark.

Methodology: Logic Over Entropy

TRR introduces a framework that functions across three main task types: Boolean, Classification, and Rating. Its core innovation lies in the use of the Gini-Simpson Diversity Index rather than Shannon Entropy.

1. The Diversity Core

Unlike Entropy, which can be hard to interpret in a vacuum, the Gini-Simpson Index provides a clear percentage:

• 0% Diversity: Absolute consensus (all workers agree).
• 100% Diversity: Complete uncertainty (every worker gives a different answer).

2. The Iterative Workflow

Instead of dumping all assignments at once, TRR operates in loops:

1. Initial Push: Assign $NC$ (minimum workers).
2. Consensus Check: Calculate Diversity Level ($DL$).
3. Stop or Repeat: If $DL\le$ Target, mark as 'Completed'. If not, estimate how many more workers are needed to reach consensus and start the next iteration.

Figure 1: The TRR workflow illustrating the iterative decision-making process.

3. Handling Anonymous vs. Non-Anonymous Workers

• Anonymous: TRR uses simple vote distribution.
• Non-Anonymous: TRR uses a weighted probability distribution where high-quality (reputable) workers have a higher impact on the diversity calculation through the following formula: $V_{q_i,j}={\prod }_{n=1}^{r}W{A}_{n,j}$

Experimental Performance

The authors validated TRR using three real-world datasets: Web Relevance (Boolean), Sentiment Analysis (Classification), and Emotion Analysis (Rating).

Cost and Latency Reductions

For Boolean tasks, TRR reduced the total cent-cost of workloads by a staggering margin ($32\%-62\%$). In terms of time, the "human working hours" required for a 1000-task batch dropped significantly compared to the 18.12-hour baseline.

Figure 2: Cost comparison between TRR and Fixed Redundancy across different configurations.

Success in Rating Tasks

Rating tasks are notoriously difficult because consensus isn't a single "label" but a cluster of numerical values. TRR adapted Normalized Hamming Distance to measure fuzzy set diversity. The experiments showed that the non-anonymous model (considering worker quality) achieved results much closer to the "Gold Standard" answers than traditional aggregation.

Critical Insight & Conclusion

The true value of TRR lies in its interpretability. By allowing task requesters to set a "Diversity Level" (e.g., 35%), the system becomes a knob that balances budget and precision.

Limitations & Future Work

While TRR is highly effective for "Micro-tasks," it currently lacks the logic for "Macro-tasks" (e.g., writing an article or coding), where "consensus" is harder to define mathematically. Future iterations aim to integrate more advanced truth-inference algorithms (like Expectation-Maximization) to further refine the "optimistic" estimation of needed workers.

Final Takeaway: TRR proves that in crowdsourcing, "more" isn't always "better." Intelligence lies in knowing when you've heard enough.