Beyond the Simple Average: Engineering Trust in Crowdsourcing via Aggregation Functions

TL;DR

In industrial crowdsourcing, "quality" isn't a static attribute—it's a moving target. This paper moves beyond simple arithmetic means to rank workers using Aggregation Functions (like OWA and Twofold Integrals). By mathematically modeling how much weight we give to a worker's past successes versus their recent failures, the authors provide a robust framework for automatic promotion, fair rewarding, and high-fidelity quality control in professional translation tasks.

Background Positioning

While platforms like Amazon Mechanical Turk revolutionized micro-tasking, they often treat workers as interchangeable commodities. This work, situated in the intersection of Fuzzy Set Theory and Human Computation, argues for an industrial-grade "Worker Ranking" system. It moves the needle from "simple crowdsourcing" to "managed professional crowds."

The Problem: The Flaw of Averages

Why is ranking hard? Imagine two workers:

• Worker A: Started poorly (training phase) but is now delivering near-perfect results.
• Worker B: Consistently mediocre, never improving, never failing.

A simple average might rank them the same. In an industrial setting (like CA Technologies' software localization), this is a failure of logic. We need a system that:

1. Values Improvement: Recent performance should outweigh the "learning curve."
2. Penalizes Instability: High variance in quality is a risk.
3. Encourages Growth: Provides a clear mathematical path from "Newcomer" to "Senior."

Methodology: The Mathematics of Fusion

The core innovation lies in the transition from Arithmetic Means (AM) to Non-Additive Measures.

1. The AV-Unit (Action-Verification)

The workflow is built on the AV-Unit, a modular design where one worker "Acts" (translates) and a set of peers "Verify." This creates a closed-loop data source for quality metrics (TQI - Translation Quality Index).

2. Sophisticated Aggregators

The paper explores three levels of "Information Fusion":

• Weighted Mean (WM): Assigns weight $\mu$ based on time. Recent tasks get higher $\mu$.
• OWA (Ordered Weighted Averaging): Assigns weight based on the magnitude of the score. This allows the system to ignore "one-off" brilliant tasks or "one-off" failures by using fuzzy quantifiers.
• Twofold Integral: The "Gold Standard" of fusion. It combines two different fuzzy measures—one representing the "probabilistic flavor" (likelihood of performance) and the "fuzzy flavor" (vague satisfaction of quality).

Experiments: Real-World Translation

The authors tested these functions on real data from CA Technologies. They categorized translators into Newcomers, Associates, and Seniors based on their TQI scores.

Key Breakthroughs:

• Worker A (The Improver): Under the Weighted Mean, they were successfully promoted to Senior (80.1 ranking) because their recent 84 and 87 scores were prioritized.
• The Outlier Filter: Using the Twofold Integral ($T{I}_{Q_t}$), the system prevented "gaming." Even if a worker had a few lucky high scores, the $Q_t$ thresholding (alpha=0.1 or 0.2) smoothed the ranking, ensuring promotion was based on sustained excellence rather than statistical noise.

Critical Insight & Conclusion

The genius of this approach isn't just the math; it's the flexibility. By adjusting the "alpha" value in $Q(x)=x^{\alpha }$, a platform manager can decide the platform's "personality":

• Strict Platform: High alpha (penalizes even a single low-quality task).
• Forgiving Platform: Low alpha (focuses on the best work a person is capable of).

Limitations: The paper relies on human-labeled TQI as the "ground truth." In the future, the cost of these human verifiers could be a bottleneck, though the authors argue that the AV-Unit structure mitigates this through recursive peer-review.

Final Takeaway: For anyone building a Gig Economy platform or an RLHF (Reinforcement Learning from Human Feedback) pipeline, this paper provides the mathematical blueprint for moving from "volume" to "value."