Surpassing the Limit: Intelligent Keyword Clustering to Maximize Data Coverage

TL;DR

Twitter's 1% data limit is a major roadblock for researchers needing high-fidelity datasets. This paper proves that Twitter's "limit track" (their official missing-data counter) is inaccurate and proposes a way to "hack" the limit. By using K-Means Round Robin (KMRR) clustering to distribute keywords across multiple API connections, researchers can capture significantly more data—sometimes exceeding 137% of what Twitter claims is the "total" population—while minimizing redundant overlap.

The "Limit Track" Illusion

For years, researchers relied on Twitter’s limit track parameter to estimate how much data they were missing. The logic was simple: $Total=Caught+Missed$.

However, the authors of this paper discovered a startling anomaly: when they split their search terms across multiple streams, they consistently collected more unique tweets than the "Total" estimated by Twitter. This suggests the sampling mechanism is not only deterministic (non-random) but that the platform's self-reporting metrics are fundamentally flawed.

Methodology: Engineering the Perfect Split

The core challenge of using multiple API streams is Overlap. If Stream A tracks "Apple" and Stream B tracks "iPhone", a tweet containing both words will be sent to both streams, wasting precious bandwidth. To solve this, the authors tested three strategies:

1. Round Robin

A simple distribution where keywords are assigned to streams sequentially. While it ensures perfectly balanced stream loads, it ignores the biological "co-occurrence" of words, leading to significant overlap.

2. Spectral Clustering

This method uses a word-to-word co-occurrence graph to group words that often appear together.

• The Insight: If "Coffee" and "Starbucks" are in the same stream, they only count as one "hit" even if they appear in the same tweet.
• The Downside: It produces uneven cluster sizes. Some streams get overwhelmed (hit the 1% limit immediately), while others stay empty.

3. K-Means Round Robin (KMRR) - The Winner

KMRR is the "Goldilocks" solution. It uses Multidimensional Scaling (MDS) to map words into a 2D space based on their co-occurrence and then applies a balanced clustering algorithm.

Figure: The MDS-transformed space showing how keywords are grouped into balanced clusters to prevent "intruding" words from causing stream overlap.

Experimental Battleground

The researchers tested these methods using 21 simultaneous streams. They found that while Spectral Clustering grows fast initially, it hits a plateau because of its unbalanced nature. KMRR, however, maintains a steady, near-linear increase in data coverage as more crawlers are added.

Figure: Comparison of coverage rates. All proposed methods significantly outperform the standard Unsplit stream (the bottom horizontal line).

Key Results:

Critical Insight: Language as a Natural Filter

An unexpected "side effect" of the clustering was that it naturally separated the stream by language. Because words like "the" (English) and "la" (Spanish) rarely co-occur in the same tweet, the algorithms naturally assigned different languages to different crawlers. This reduced overlap even further, acting as a "free" optimization for the search strategy.

Conclusion & Future Outlook

This paper serves as both a technical blueprint for data engineers and a warning to social scientists.

• For Engineers: If you need more data, don't just add more crawlers; cluster your keywords using KMRR to maximize the "diversity" of each stream.
• For Scientists: Stop trusting platform-provided "limit" metrics. They are likely undercounting the true scale of the conversation, which introduces hidden biases into your models.

The future of this work lies in "Semantic Realignment"—moving beyond simple word co-occurrence to use deep learning embeddings (like BERT or modern LLMs) to ensure that streams are partitioned by topic and intent, not just string matching.