What has Worked and What Won’t Work in Crowdsourcing: A SIGCOMM Retrospective

TL;DR

This report summarizes the high-level expert panel from SIGCOMM '15 involving visionaries from MIT, INRIA, and Politecnico di Torino. It tackles the fundamental tension in network research: the need for massive, distributed data collection versus the technical and ethical limitations of crowdsourced infrastructure. The panel identifies the shift toward programmable, federated platforms as the successor to traditional, ad-hoc crowdsourcing.

Problem & Motivation

Network measurement has historically relied on fixed infrastructures (like PlanetLab). However, these do not reflect the diverse, "edge-heavy" reality of the modern internet.

• The Scalability Gap: Static nodes cannot capture the performance of mobile networks or localized ISP behaviors.
• The Control Dilemma: Crowdsourcing provides scale but sacrifices control. How can an experimenter ensure scientific validity when the "vantage point" belongs to a volunteer who might turn off their device at any time?
• Ethical Complexity: Unlike lab environments, crowdsourcing involves real users, raising questions about privacy, consent, and data sovereignty.

Methodology: Redefining the Vantage Point

The panel moves beyond the "What" of crowdsourcing to the "How" and "Why." They propose a multi-layered approach to solving the enlistment problem:

1. Vantage Point Acquisition: Identifying the "right locations" rather than just "any location." This involves incentives that go beyond altruism.
2. Platform Abstraction: Proposing a programming interface that allows researchers to deploy experiments across different crowdsourced fleets (e.g., mobile apps, browser extensions, home routers) without rewriting code for each.
3. Federation: The idea of a "Meta-Platform" that connects smaller crowdsourced projects into a singular, massive-scale measurement engine.

Key Questions & Discussions

The core of the panel revolved around the interactive Call for Papers (CFP) issues:

• Human vs. Automated: How do we move from "human-entered" data (like Speedtest results) to background agents that provide continuous, objective telemetry?
• Experimental Viability: Can we achieve statistical significance on platforms where we have "limited control"? The consensus points toward developing new stochastic experimental models that account for node churn.

Deep Insight & Conclusion

The true value of this work lies in its systemic perspective. Instead of viewing crowdsourcing as a "hack" to get more data, the panel treats it as a first-class networking primitive that requires its own architecture, API, and ethical "social contract."

Takeaways:

• Infrastructure over Apps: The future of crowdsourcing isn't in individual apps but in federated systems that share vantage points.
• Ethics as a Feature: Ethical viability isn't a hurdle but a requirement for the longevity of any crowdsourced platform.

Limitations: As a 2015 panel, it predates the massive shift toward Zero-Knowledge Proofs and Differential Privacy, which are now critical for the "ethical viability" the authors discussed.