[VLDB Demo] CrowdCleaner: Tackling the Volatility of Multi-Version Web Data via Hybrid Crowdsourcing

TL;DR

CrowdCleaner is a pioneering prototype that addresses the "freshness" problem in Web data. By blending passive error detection with active, credibility-aware repair tasks, it corrects errors in frequently updated information—like conference deadlines and retail prices—where traditional algorithms and static rules fall short.

Academic Positioning: This work bridges the gap between Web Data Integration and Human-in-the-loop (HITL) Computing, introducing a hybrid crowdsourcing model that optimizes for both cost and accuracy in dynamic environments.

Problem & Motivation: The "Multi-Version" Headache

We live in a world of "Multi-version data"—information that changes rapidly (e.g., fuel prices, real estate listings, or CFP deadlines). Automated systems rely on web crawling and machine learning to integrate these updates, but they are plagued by two issues:

1. Update Latency: The gap between a real-world change and a system update.
2. Integration Errors: Heuristics used to extract data often fail as website schemas shift.

Previous data cleaning approaches (like [4] Fan et al.) use "Master Data" or "Editing Rules," but these are too rigid for the fluid nature of the Web. Pure active crowdsourcing (querying the crowd constantly) is too expensive. CrowdCleaner asks: Can we let users tell us when something is wrong, and then use a "smart jury" to fix it?

Methodology: The Hybrid Architecture

CrowdCleaner operates through a four-module loop that balances cost, speed, and reliability.

1. The Passive Start: Error-Monitor

Instead of constantly checking every data point, the system waits for users to report errors. To prevent spam, it uses a credit-based scheme. Users spend credits to report; if they are right, they earn bonuses.

2. The Smart Fix: Entropy-based Decision Strategy

When the crowd submits multiple different repairs (e.g., different dates for a deadline), the system treats the suggestions as a discrete random variable $X$. It calculates the Shannon Entropy $H(X)$: $H(X)=-{\sum }_{i=1}^{n}Pr(x_i)\log Pr(x_i)$

• Low Entropy: The crowd agrees; the system accepts the majority vote.
• High Entropy: The crowd is confused; the system triggers a "whom-to-ask" search or consults an expert.

3. Identifying the "Elite": Whom-to-Ask-Finder

To avoid the bottleneck of busy experts, CrowdCleaner calculates Group Confidence (GC) to find a subgroup of highly credible workers $(C{W}_n)$. The goal is to ensure the probability of the majority of the group being correct exceeds a specific threshold.

Experiments & Results

The authors demonstrated the system using ICDE conference deadlines. In a scenario where 50 random workers provided conflicting dates (Entropy = 1.14), the system identified a "credible crowd."

Fig: (left) Random crowd distribution vs (right) Credible crowd distribution.

Key Findings:

• By routing tasks to the Whom-to-ask-finder, the system achieved a near-zero entropy repair, effectively matching expert performance at a fraction of the cost.
• The credit-based motivation system successfully deterred spam while incentivizing high-quality historical records for workers.

Critical Insight & Conclusion

Takeaway

CrowdCleaner's core value lies in its asymmetric logic: utilize the massive, unwashed masses for detection (passive), but pivot to a curated, high-credibility "jury" for resolution (active). This solves the "crowd noise" problem that plagues simple voting mechanisms.

Limitations & Future Work

While effective, the system assumes that "truth" can be verified via a URL provided by a worker. In scenarios with no single "source of truth" (like subjective sentiment cleaning), the entropy model might struggle. Future iterations could explore reinforcement learning to dynamically adjust the credit-weighting of workers based on longitudinal performance.