Facebook Crawler via Interaction Simulation: Breaking the 400-Friend Barrier

TL;DR

Researchers from the National University of Defense Technology have developed a specialized Facebook crawler that bypasses strict pagination limits (60-400 friends) by simulating Ajax interaction behavior. Beyond the technical feat, the study reveals a massive shifts in digital privacy: 36.5% of users now opt out of public friend visibility, a near 130% increase compared to 2009 data.

Background & Motivation: The Walled Garden Problem

Online Social Networks (OSNs) are goldmines for sociologists and computer scientists, yet Facebook—the largest of them all—is also the most fortified. Historical methods for data extraction, such as Public Listings or the now-deprecated Facebook Query Language (FQL), have been rendered obsolete by frequent privacy updates.

The primary technical bottleneck identified by the authors was the pagination cap. Previous state-of-the-art crawlers were limited to 400 friends per user because Facebook's static page rendering stopped there. When Facebook further tightened this to 60 friends per dynamic load, traditional BFS (Breadth-First Search) crawlers became virtually useless for mapping complete social graphs.

Methodology: Simulating the Human Click

The core "Insight" of this paper is moving from static HTML parsing to Interaction Simulation. Instead of trying to "scrape" the page, the crawler mimics the asynchronous requests (Ajax) a browser makes as a user scrolls.

1. Traffic Analysis

By using HttpFox to sniff traffic, the authors discovered that Facebook uses a specific URI structure for fetching friend subsets: .../ajax/browser/list/friends/all/?uid=[ID]&offset=[60*i]

2. The Algorithmic Loop

The crawler follows a predictable yet effective logic:

1. Log in: Use real credentials as an entry point.
2. Get Friend Count: Extract the total friendNum from the profile.
3. Calculate Iterations: If a user has 600 friends, the crawler calculates 10 separate URI requests.
4. Simulate Ajax: It systematically visits each URI offset to fetch the next 60 friends.

Figure 1: Visualizing the captured Ajax traffic used to bypass pagination.

Experiments & Societal Findings

The authors deployed their crawler for one month (Oct-Nov 2011), collecting a dataset of 262,526 unique users.

The Privacy Awakening

One of the most striking results is the quantitative increase in privacy awareness. By checking the visibility status of crawled profiles, they found:

• 2009 (Prior Work): ~16% - 26.6% users protected their data.
• 2011 (This Study): 36.5% of users had changed default settings to hidden.

This suggests that as platform policies became more complex, users became more proactive in safeguarding their digital footprints.

Technical Validation

To verify that the crawler actually broke the "400 limit," the authors analyzed the "Visible" group. They found thousands of users where the list exceeded 130 friends (the Facebook average) and successfully fetched the complete lists for those with high degrees, proving the algorithm's robustness.

Table 1: Network metrics from the crawled dataset (Note the low Avg. Degree (2.59) due to BFS bias).

Critical Analysis & Future Outlook

While technically successful, the methodology faces one major academic hurdle: BFS Bias. As noted in the results, the average degree of 2.59 is significantly lower than the actual Facebook average (~130). This is because Breadth-First Search naturally gravitates toward high-degree nodes ("super-hubs") and often fails to explore the "long tail" of the network within a limited timeframe.

Key Takeaways:

• Technical: Interaction simulation is the only viable way to crawl modern, dynamic OSNs.
• Sociological: Privacy awareness is not static; it grows in tandem with platform complexity.
• Future Work: To obtain a truly "scientific" graph, authors suggest moving to Distributed Crawlers and Metropolis-Hastings Random Walk (MHRW) to eliminate the sampling bias inherent in BFS.

Ultimately, this work serves as a foundational bridge between the era of public web-scraping and the current era of protected, dynamic social silos.