Decoupling the Crowdsourcing Paradox: How Task Awards Shape Software Development Quality
2063_Award vs. Worker Behaviors in Competitive Crowdsourcing Tasks.
This paper investigates the dynamics of crowdsourced software development by analyzing the relationship between task awards and worker behavior. It proposes a conceptual Award-Worker behavior model and validates it using 494 development tasks from TopCoder, identifying how monetary incentives influence registration and submission quality.
TL;DR
Higher pay doesn't always equal better code. This paper analyzes nearly 500 TopCoder tasks to map the "Conceptual Award-Worker Behavior Model," revealing that while awards drive registration, they can sometimes negatively correlate with the final submission score if task complexity isn't managed.
Problem & Motivation: The Incentive Dilemma
In crowdsourced software engineering, the primary lever for attracting talent is the Award. Platforms like TopCoder rely on a competitive "winner-takes-all" or "top-few-take-parts" model. However, the industry faces a recurring challenge: The Crowdsourcing Paradox.
Increasing an award might attract more registrants, but it also increases perceived competition risk. For high-difficulty tasks, many talented workers might "self-select" out, fearing they will invest hours only to lose to a superior competitor, leaving a pool of less-qualified participants or suboptimal submissions.
Methodology: The Award-Worker Model
The authors propose a systematic approach to categorize tasks into four regions based on Award Value (Threshold: $750) and Registrant Count (Threshold: 18):
- Region I: Low Award, Low Competition (Economy).
- Region II: Low Award, High Competition (Popular).
- Region III: High Award, High Competition (Elite).
- Region IV: High Award, Low Competition (Niche/Risky).
The methodology focuses on how these external stimuli (Award, Task Size) impact internal metrics like the Submission Ratio (SR) and the final Winning Score.
Key Insights from the Data
The descriptive statistics show an average award of ~$753 and an average winning score of 92.5. However, the correlation table reveals a striking trend:
| Correlation Metric | Award vs. #Submissions | Award vs. Score |
|---|---|---|
| Value | -0.40 | -0.71 |
This strong negative correlation between high awards and scores (in specific binned groups) suggests that as awards grow very large, the quality of the winning submission actually tends to drop. This is likely because extremely high-award tasks are often significantly more complex (higher LoC), posing a barrier that even high incentives can't overcome.
Methodology Refinement: Task Size Matters
The research found that the "Predictability" of worker behavior changes based on code size. For tasks between 2k and 3k lines of code (loc), the model fitness for predicting the number of submissions was remarkably high (R² = 0.854). This suggests there is a "Sweet Spot" for task granularity where financial incentives work most predictably.
Critical Analysis & Conclusion
Takeaway
For project managers, the message is clear: Don't just throw money at a problem.
- Decomposition is key: Break large tasks into the 2k-3k loc range to make worker behavior more predictable.
- Avoid Region IV: High awards with low registration often result in low-quality outliers.
Limitations
The study is focused on TopCoder, a platform with a specific competitive culture. Results might differ on collaborative platforms or for internal enterprise crowdsourcing. Furthermore, "Lines of Code" is a proxy for complexity but doesn't always account for architectural difficulty or creative requirements.
Future Outlook
Future research should integrate Worker Reputation into this model. Understanding how a worker's past performance interacts with the award size could solve the "Paradox" by identifying who is actually being lured by the higher price tags—experts or gamblers?