Crowdsourcing Video Annotation: Scaling Human Activity Data Without Sacrificing Precision

TL;DR

Supervised activity recognition is "starving" for labeled data, yet professional video annotation is prohibitively expensive and slow. This paper investigates whether Amazon Mechanical Turk (AMT) workers—non-experts—can accurately tag the start, end, and labels of activities. By implementing a robust filtering pipeline involving individual and collaborative checks, the researchers achieved up to 92% accuracy, proving that the "crowd" can reach expert-level precision for a fraction of the cost.

Background: The Annotation Bottleneck

In the world of Activity Recognition (AR), researchers are often caught in a paradox: modern machine learning models (using video or on-body sensors) require massive amounts of precisely labeled data, but manual labeling is a "manual labor" hell. Annotating 30 minutes of fine-grained activity can take an expert up to 10 hours.

While crowdsourcing has been used for image classification, temporal video tagging is a different beast. It requires precise synchronization and segmenting of "start" and "end" times. This paper asks: Can we trust non-experts with this degree of precision?

Methodology: Filtering Out the Noise

The core innovation lies not just in the data collection, but in the Post-Processing Pipeline. Crowdsourced data is notoriously noisy, with roughly 40-60% of workers in the study being flagged as "non-serious" or "spammers."

1. The Multi-View Interface

The researchers built a custom HTML5 interface that syncs multiple camera angles. This is crucial because a single viewpoint often obscures activities (e.g., a person's back blocking a drawer).

2. The Data Processing Chain

To transform raw, messy worker inputs into a "Golden Standard," the paper proposes a chain of filters:

• Qualification Check: Verifying answers to "trap" questions (e.g., "At what time did the user pick up the egg?").
• Overlapping Boundary Removal: Logical checks to ensure a single actor isn't doing two mutually exclusive things at once (a common sign of a spammer).
• Spammer Removal (Collaborative Filtering): Using a disagreement score ($d$) to prune workers who consistently deviate from the group consensus.
• Majority Voting: A temporal segment-by-segment voting mechanism to finalize the labels.

Experiments and Insights

The method was tested on two classic datasets: CMU Kitchen and Opportunity.

• Speed and Cost: While the total "wall clock" time to finish a task was 2-3 days, the actual annotation hours were comparable to experts. However, the cost was significantly lower—approximately $50 for 30 minutes of video across multiple workers.
• Accuracy Boost: The filtering strategies weren't just "nice to have"—they were essential. Accuracy improved by 40% when moving from "No Check" (Type 1) to "All Checks" (Type 6).

Critical Analysis: Why This Matters

The most striking takeaway is the failure of reputation scores. Even though the authors only hired workers with a >90% approval rate, half of them still provided sub-par or "spammy" data for this specific task. This suggests that high-complexity tasks like video annotation require task-specific verification rather than relying on platform-level metrics.

Limitations:

• Physical Ambiguity: Activities like "toggling a switch" remain difficult even for experts to agree on the exact millisecond of the start/end.
• Semantic Ambiguity: The study used a predefined list of activities. Future work must handle "open-vocabulary" tagging where workers might use different synonyms for the same action.

Conclusion

This work provides a blueprint for any team needing large-scale video datasets. By treating the crowd not as a single source of truth, but as a noisy sensor array that requires filtering and calibration, we can unlock the potential of human computation for the most tedious tasks in AI development.