Human-in-the-Loop 2.0: Hybrid Crowdsourcing for Linked Data Quality

TL;DR

The vastness of the Web of Data (Linked Data) makes manual error correction a daunting task, while automated scripts often miss nuanced contextual flaws. This paper introduces a sophisticated hybrid crowdsourcing workflow that pairs a "Contest" for experts with "Microtasks" for general workers. By applying the Find-Fix-Verify pattern to DBpedia, the researchers demonstrate that we can achieve near-perfect (94% precision) data quality assessment at a fraction of the traditional cost.

Problem & Motivation: The Quality Gap in the Web of Data

Linked Data (LD) is the backbone of the semantic web, yet its quality is notoriously uneven. Because datasets like DBpedia are extracted semi-automatically from Wikipedia, they inherit or create several "triple-level" flaws:

• Object Errors: Incomplete data (e.g., an ISBN showing only the first three digits).
• DataType Mismatches: Formatting errors (e.g., a "Year" field containing a full ISO timestamp).
• Broken Interlinking: "SameAs" links that point to irrelevant external resources.

Algorithms struggle here because they lack the "Common Sense" or real-world context needed to verify if a Wikipedia infobox was parsed correctly. However, hiring experts to check millions of triples is financially impossible. The author's insight? Differentiate the crowd. Not every task needs a PhD; some only need a pair of eyes and a clear interface.

Methodology: The Find-Fix-Verify Pattern

The authors implement a strategic pipeline that separates the "Discovery" of errors from the "Verification" of those errors.

1. The Expert Contest (The "Find" Stage)

Using a tool called TripleCheckMate, the researchers mobilized the Linked Data community. Participants were encouraged by a competitive prize (a tablet) to browse DBpedia resources and flag triples as "Wrong."

• Goal: Detect and classify complex errors using technical knowledge.
• Tooling: TripleCheckMate allowed experts to map errors to a specific taxonomy (Accuracy, Relevancy, etc.).

2. Microtask Crowdsourcing (The "Verify" Stage)

The flagged errors were then sent to Amazon Mechanical Turk (MTurk). Crucially, the authors didn't show raw RDF code to these workers. Instead, they transformed the data into human-readable questions (e.g., "Does this Wikipedia snippet match this birth date?").

Experiments & Results

The evaluation focused on whether the "lay crowd" could match or improve upon the expert findings.

• Interlinking Success: MTurk workers achieved a staggering 0.94 precision using majority voting, far outperforming the experts (0.15) who struggled with the tedious nature of checking external links.
• Cost Efficiency: By only sending "suspected" errors to MTurk rather than the entire dataset, the cost dropped by over 95% (from ~$1,000projectedto$43 actual).
• Expert Superiority in Technicals: For "Incorrect Data Types," experts remained the gold standard (0.83 precision). MTurk workers struggled here because they often confused general numerical types with specific semantic types like xsd:gYear.

Critical Analysis & Conclusion

The Synergy of Crowds

The paper definitively proves that crowdsourcing is not a monolith. Expert crowds are "High-Precision Filters" for technical taxonomy, while Paid microtasks are "High-Throughput Verifiers" for routine comparisons.

Limitations & Future Work

1. UI Dependency: The precision of MTurk workers was highly dependent on the quality of the "Human Intelligence Task" (HIT) interface. If the UI was ambiguous, the data quality plummeted.
2. The "Fix" Stage: This paper focused on Assessment. The actual Fixing (automatically updating the RDF triple) remains a challenge for future integration into live SPARQL endpoints.

Final Takeaway: To scale the Web of Data, we must stop viewing human curation as a single-tier process. By intelligently routing tasks based on complexity, we can maintain high-quality knowledge graphs at a sustainable cost.