FROG: Revolutionizing Crowdsourcing with Active Scheduling and Social-Aware Notifications

TL;DR

Crowdsourcing often suffers from the "straggler" problem—where a few slow or difficult tasks delay a whole project for days. FROG (Fast and Reliable crOwdsourcinG) shifts the paradigm from passive task listing to active scheduling. By calculating "delay probabilities" and leveraging social network data to predict worker availability, FROG reduces maximum task latency by 50% without sacrificing accuracy.

The Problem: The High Cost of Unreliable "Pull" Markets

Most current platforms like Amazon Mechanical Turk (AMT) operate on a "Pull" mechanism: requesters post tasks, and workers choose what they like. This creates two major issues:

1. Task Neglect: Difficult or low-paying tasks stay in the queue indefinitely.
2. Latency Bottlenecks: In workflows like sentiment analysis of 1,000 tweets, the requester can only proceed once the last tweet is analyzed. A single "lazy" worker can stall the entire pipeline.

Existing solutions often try to solve this by throwing money at the problem (dynamic pricing) or filtering out slow workers, which wastes human potential.

Methodology: The FROG Framework

FROG introduces a two-pronged attack on latency and reliability: the Task Scheduler and the Notification Module.

1. Active Task Scheduling (FROG-TS)

The authors prove that finding the optimal worker-to-task assignment to minimize maximum latency is NP-hard (reducible from the Multiprocessor Scheduling Problem). To solve this, they define Delay Probability:

• It considers the current time lapse, the estimated difficulty of the task (based on entropy of previous answers), and the worker's historical response speed.
• Batch-Based Scheduling (BBS): Instead of one-off assignments, the system greets available workers with a curated batch of tasks, matching high-accuracy workers with high-difficulty "urgent" tasks.

The FROG Framework Overview: Integrating Worker Profiles, Schedulers, and Quality Controllers.

2. Intelligent Notifications (SKDE)

What if the worker pool is empty? Rather than spamming all registered users, FROG uses a Smooth Kernel Density Estimation (SKDE).

• The Insight: Worker availability follows a temporal pattern (e.g., commuting hours).
• The Social Fix: For new workers with no history ("Cold Start"), the model uses the online patterns of their social media friends (from networks like WeChat) as a proxy for their own availability.

Experimental Performance

The researchers tested FROG against baselines like iCrowd and fGreedy across five real-world scenarios, including Disaster Event Detection and App Search Matching.

Key Findings:

• Latency Reduction: The BBS approach consistently outperformed others. As the number of tasks increased, BBS kept the maximum latency significantly lower than the iCrowd framework.
• Accuracy Gains: By using the MinWorkerSetSelection algorithm, FROG ensures that even while moving fast, tasks meet a target quality threshold (typically 80-95%) by dynamically adjusting how many workers see a task based on their specific expertise and category accuracy.

Performance Comparison: BBS shows superior scalability in maintaining low latency as task volume (m) increases.

Critical Insight: Why it Works

FROG succeeds because it moves away from the "one-size-fits-all" approach to workers. By acknowledging that a worker might be an expert in "Politics" but slow in "Image Labeling," and by predicting when they will be active using social graphs, the system turns a chaotic market into a finely tuned engine.

Conclusion & Future Work

FROG represents a significant step toward Real-time Crowdsourcing. While the current framework relies on majority voting, future iterations could integrate more complex cognitive bias models to further refine accuracy. For industry practitioners, the takeaway is clear: don't just wait for workers; invite the right ones at the right time.

Note: This research was supported by grants from the Hong Kong RGC and Microsoft Research Asia.