Modelling Crowdsourcing Originated Keywords within the Athletics Domain: Bridging the Semantic Gap

TL;DR

This study evaluates how to effectively turn crowdsourced image tags into reliable visual models for the athletics domain. By testing various combinations of machine learning classifiers (SMO, LibSVM, Random Forest, LMT) and visual features (HOG, SIFT, MPEG-7), the researchers found that SMO paired with MPEG-7 descriptors yields the highest classification accuracy (74.38%), effectively handling the ambiguity inherent in crowd-generated data.

Problem & Motivation: The Semantic Gap and the Crowd

Automatic image annotation faces a fundamental hurdle: the Semantic Gap. This is the disconnect between low-level pixel data (color, edges) and high-level human meaning ("Javelin", "Hurdles"). While crowdsourcing platforms like Amazon Mechanical Turk have made it easier to gather massive amounts of human-labeled data, these labels are often noisy or subjective.

The researchers sought to determine which technological stack—which specific features and algorithms—can best transform these "cheap" crowd labels into accurate, automated retrieval models specifically for sports imagery.

Methodology: The Core Architecture

The workflow follows a rigorous three-step process:

1. Crowdsourced Dataset: 500 images from the BOEMIE project, annotated by 15 users.
2. Feature Extraction: The team compared three distinct methodologies:
   • HOG (Histogram of Gradients): Capturing local edge distributions.
   • SIFT (Scale-Invariant Feature Transform): Identifying local points of interest.
   • MPEG-7: Extracting global characteristics like Color Layout and Edge Histograms.
3. Keyword Modelling: Using a one-against-rest approach, each keyword (e.g., "High Jump") was treated as a binary classification problem.

Figure 1: The proposed method for modeling crowdsourced keywords.

The study utilized Weka to compare four classifiers: SMO (a fast SVM implementation), LibSVM, Random Forest, and Logistic Model Trees (LMT).

Experiments & Results: Winners and Losers

The results provided clear insights into which features and classifiers dominate the athletics domain:

1. The Superiority of MPEG-7

Contrary to current trends favoring local descriptors in many CV tasks, MPEG-7 global features were the clear winners here. They achieved accuracy ranges of 71.25% to 81.25%. This suggests that in sports, the overall scene layout and color distribution are more indicative of the sport type than specific local keypoints.

2. Classifier Efficiency

SMO stood out as the most reliable and efficient classifier. It was not only the most accurate (74.38% average) but also significantly faster to train than LMT.

Figure 2: Performance of LibSVM across different keywords and features.

3. Class-Specific Performance

Keywords with distinct visual structures, like "Discus" and "Hurdles", were modeled with high accuracy (up to 95%). Conversely, "Running" and "Triple Jump" proved more difficult due to visual similarities in the background and athlete postures.

Critical Analysis & Conclusion

The study successfully demonstrates that visual models can indeed be built from noisy crowdsourced keywords. The primary takeaway is that SMO is a highly robust choice for this type of noisy, small-to-medium scale dataset, and that global features (MPEG-7) are unexpectedly powerful in the athletics context.

Limitations: The dataset size (500 images) is relatively small by modern standards. Furthermore, the exclusion of certain MPEG-7 descriptors due to "software bugs" suggests that even better results might be possible with a more comprehensive feature set.

Future Outlook: The next logical step for this research involves moving beyond "hand-crafted" features (like HOG/SIFT) toward Deep Convolutional Neural Networks (CNNs), which might capture the similarities between "Running" and "Triple Jump" that traditional methods missed.