CrowdSearcher: Leveraging Social Communities for High-Precision Crowdsourcing

TL;DR

This research moves beyond anonymous "Micro-task" platforms (like Amazon Mechanical Turk) by proposing a framework that targets specific social communities. By using an active rule-based system, the CrowdSearcher platform can dynamically shift tasks between "experts" (e.g., a specific research lab) and "generalists" (e.g., a whole department) to optimize for speed, cost, and accuracy.

Context: Why Identity Matters in the Crowd

Most crowdsourcing systems treat workers as interchangeable units of labor. However, a "crowd" of random internet users will likely fail at tasks requiring domain-specific context—such as identifying a specific professor from a grainy conference photo. The authors argue that leveraging communities—groups defined by shared interests, geography, or professional ties—is the key to unlocking higher-quality human computation.

Methodology: The Reactive Control Engine

The core innovation lies in the Control Mart and the use of Active Rules (ECA: Event-Condition-Action). Instead of a static workflow, the system monitors the campaign in real-time.

Key Operations

The system defines elementary tasks such as Choice, Like, Score, Tag, and Classify. These are then managed by rules like:

• Static Adaptation: If a task isn't finished within 48 hours, automatically enable a broader community.
• Dynamic Adaptation: If the expert community is "idle" (no responses for 1 hour), re-invite a wider social network group.
• Object-level Routing: If there is high disagreement (low consensus) on a specific image, send that specific object to a "Gold Standard" expert group for tie-breaking.

Figure 1: The interface allows designers to define agreement thresholds and platform routing.

Experimental Analysis: Inside-Out vs. Outside-In

The authors tested their hypothesis using an image dataset of university professors. They compared two strategies:

1. Inside-Out: Start with the immediate research group (the "experts"), then expand to the department, then to social networks.
2. Outside-In: Start with the broad department and narrow down to experts only for difficult cases.

Results & Precision

The data confirmed that the Inside-Out strategy achieved superior precision early on. Expert groups (Research Groups) had a precision of 0.68, whereas the general Department community dropped to 0.58.

Figure 2: Comparison of precision and object closing rates across different community strategies.

Critical Insights

The most striking finding wasn't just the data accuracy, but the qualitative shift in worker attitude. Unlike anonymous workers who seek only monetary reward, community members felt a sense of "mission." They provided unsolicited feedback on the UI and the dataset, showing that social identity increases Intrinsic Motivation.

Limitations & Future Work

• Latency: Because community sizes are smaller than global platforms, tasks can sit "idle" longer, making temporal management—like the idle-time rules proposed—essential.
• Scalability: Manually defining these communities requires prior knowledge of the social graph. Future iterations could benefit from automated "community discovery" via social media APIs.

Conclusion

CrowdSearcher demonstrates that the future of human computation isn't just "bigger crowds," but "smarter crowds." By bridging the gap between social networking and task execution, we can create reactive systems that are both cost-effective and highly accurate.