Distributed Web 2.0 Crawling: Bridging the Gap Between Raw Data and Ontology Evolution

TL;DR

This research addresses the "invisibility" of Web 2.0 content to traditional crawlers. By integrating full browser rendering (DOM execution) with a distributed, site-aware partitioning strategy, the authors provide a framework that feeds high-quality, noise-reduced data into an ontology evolution system, ensuring that semantic models keep pace with the rapidly changing web.

Problem & Motivation: The Static Crawler’s Blind Spot

The transition from Web 1.0 to Web 2.0 turned websites into "infoware"—applications rather than just documents. Traditional crawlers, which simply download source code, are blind to content generated via AJAX (Asynchronous Javascript + XML) or hidden behind client-side scripts.

Furthermore, for Ontology Evolution—the process of updating formal knowledge structures—raw data isn't enough. The data must be:

1. High Fidelity: Exactly what a human sees.
2. Noise-Free: Excluding ads and menus that confuse text mining algorithms.
3. Time-Consistent: Captured in snapshots to track how concepts change over time.

Methodology: The Crawler as a Virtual User

The core innovation lies in the "Browser Rendering and Script Execution" block. Instead of a simple GET request, the system launches a controlled instance of Firefox.

1. Browser-Integrated Crawling

Using Selenium, the crawler executes Javascript and compiles a complete DOM tree. This ensures that dynamic content—the hallmark of Web 2.0—is fully captured. To stay efficient, the authors use Sitescripts: site-specific Javascripts injected into the page to strip away navigational chrome and advertisements before the data is persisted.

Figure 1: The enhanced crawling loop incorporating browser rendering.

2. Intelligent Distribution

To scale, the system uses a cluster. However, the authors found that standard MapReduce (like early Nutch/Hadoop) caused issues like "overlap" and "rapid-fire" requests to single servers. Their solution: Dynamic Partitioning. By combining Site-Hash and IP addresses, they ensure that all pages from one server are handled by the same worker, allowing for polite, stateful crawling while balancing the load across machines of varying power.

Figure 2: Selenium integration for programmatic browser control.

Ontology Evolution: From Data to Knowledge

The final stage of the pipeline is the Ontology Extension Loop. The crawler provides timestamped, clean text to a semantic network. Using a "Spreading Activation" algorithm, the system identifies new concepts and relationships, which are then presented to human experts for validation. This creates a semi-automatic feedback loop that keeps ontologies "alive."

Figure 3: The semi-automatic loop for updating ontologies based on crawled data.

Critical Analysis & Conclusion

The value of this work is its recognition that the "Web as a Platform" requires a "Crawler as a User." By moving the complexity of rendering to the crawling stage, the subsequent text mining and ontology tasks become significantly more accurate.

Takeaway: Effective knowledge management in the modern era requires a tight coupling between sophisticated data acquisition (browser-based) and semantic analysis (ontology evolution).

Limitations: Rendering every page in a full browser is computationally expensive compared to headless HTML parsing. While the authors found it "manageable" for their use case (thousands of sites), scaling this to a search-engine level (billions of sites) would require massive infrastructure.