MOQuery: Balancing Relevance and Representativeness in Social Media Sampling

TL;DR

Collecting data from social media via APIs often leads to biased datasets because manual keywords can't capture the full diversity of a topic. This paper introduces MOQuery, an adaptive algorithm that automatically generates search queries to find "hidden" relevant documents while ensuring the resulting dataset mirrors the word distribution of the entire population. It consistently beats standard sampling methods in both accuracy and diversity.

The Motivation: Why Your Social Media Data Might Be Biased

In computational social science, we usually don't have the luxury of "downloading the whole internet." Instead, we use APIs (like Twitter or Instagram) to search for keywords. However, this creates two major risks:

1. Coverage Error: If you only search for "depressed," you might miss people expressing mental distress through other phrases, leaving your findings incomplete.
2. Sampling Bias: If your keywords are predominantly used by one demographic, your results will reflect that group’s bias rather than the general population.

The authors argue that when class imbalance is high (e.g., finding rare events like online harassment), these errors make most "big data" studies scientifically fragile.

Methodology: The Multi-Objective Framework

The core innovation is an "adaptive" loop. Instead of using a fixed list of words, the system updates its strategy after every document it retrieves. It scores every potential query word $x$ using three factors:

1. Relevance ($f_c$): How likely is this word to return a relevant document? (Trained on a small initial labeled set).
2. Marginal Representativeness ($f_r(x)$): Does this word help our collected sample look more like the overall unlabeled population in terms of general vocabulary?
3. Class Conditional Representativeness ($f_r(x|y)$): Does our subset of relevant documents look like the true distribution of all relevant documents?

These are combined using a Geometric Mean to find terms that satisfy all three goals simultaneously.

Figure 1: Title and core concept of the adaptive query generation framework.

Experiments: Putting MOQuery to the Test

The researchers tested MOQuery against two baselines: B1 (Random Sampling) and B2 (Most Predictive Words) across three datasets:

• Instagram: Online harassment (13.2% positive).
• Twitter: Smoking cessation intent (18.3% positive).
• 20 Newsgroups: A standard benchmark (5.2% positive).

Key Findings

• Superior Balance: In the Instagram and Twitter sets, MOQuery achieved significantly higher Coverage (finding more relevant hits) and Representativeness (lower Hellinger distance to the true population).
• Complexity Matters: On the simple "20 Newsgroups" task, selecting the top 10% of predictive words (B2) worked well. However, for "messy" real-world tasks like identifying smoking cessation or harassment, MOQuery's ability to explore the keyword space was far superior.

Figure 2: Performance comparison showing MOQuery (red line) consistently maintaining high metrics as the number of queries increases.

Critical Analysis & Future Outlook

The beauty of MOQuery lies in its unsupervised refinement. It doesn't need a human to label documents in real-time (unlike traditional Active Learning); it uses the documents it finds to adjust its own "internal map" of the search space.

Limitations: The method assumes you have a small "seed" of labeled data and a larger "pool" of unlabeled context. In cases where the target class is extremely rare (e.g., <1%), even MOQuery might struggle to find the first few "islands" of relevance.

Takeaway: This work is a wake-up call for researchers who rely on simple keyword searches. If we want social media science to be as rigorous as traditional survey science, we must use adaptive, multi-objective tools like MOQuery to ensure our data represents the world, not just our search terms.