VisualCit: Bridging AI and the Crowd to Monitor Global Policy Adherence

TL;DR

Researchers have developed VisualCit, a pipeline that turns Twitter's massive stream of images into actionable data for policy makers. By combining Deep Learning filters (to remove memes and "junk") with human crowdsourcing (to verify behavior), the system can accurately track COVID-19 mask-wearing and social distancing trends across 23+ countries in near real-time, often outperforming traditional surveys in speed and reach.

Background: The Hidden Value in Social Images

During global crises like COVID-19, governments need to know: Are people actually wearing masks? Are they social distancing? Traditionally, this requires surveys (like those by YouGov), which are slow, expensive, and often miss many countries.

While Twitter is a goldmine for this data, mining images is notoriously hard. Unlike text, images are full of memes, NSFW content, and "non-photos." VisualCit solves this by positioning itself as a hybrid system: AI handles the heavy lifting of filtration, while the "Citizen Science" crowd handles the nuance of human behavior.

Methodology: The "Human-in-the-Loop" Pipeline

The VisualCit approach follows a rigorous four-step process:

1. Smart Filtering (The AI Sieve)

The system doesn't just look for "people." It uses a layered approach to ensure data quality:

• VGG19 (Fine-tuned): Detects and removes memes, graphs, and ads.
• OpenNSFW: Filters out inappropriate content to protect crowd workers.
• YOLO: Identifies images containing two or more people (filtering out selfies).
• Places365: Specifically selects public scenes (streets, parks) where policy adherence is mandatory.

2. Geolocation (CIME)

Metadata is often stripped from social media photos. The CIME geolocation algorithm parses the tweet text and user profile to find a location via OpenStreetMap, assigning each image to a country.

3. Crowdsourcing (PyBossa)

The "Relevant" images are sent to human volunteers. Humans answer questions that AI still struggles with: Is the mask being worn correctly (covering the nose)? Is the setting truly a public square?

Figure 1: The VisualCit social sensing pipeline workflow, showing the transition from raw crawl to validated indicators.

Experiments & Real-World Validation

The authors tested VisualCit on a crawl of over 470,000 tweets. The AI filters were extremely effective, achieving F1-scores between 94% and 99%.

The ultimate test was comparing VisualCit's results to the CovidDataHub (official surveys).

• Correlation: For "Yes" responses on mask usage, the correlation was a staggering 0.91 in May 2020.
• Reach: In August 2020, while official surveys only covered 4 countries, VisualCit provided data for 15 countries.

Table 1: Evaluation of image filters showing high precision and recall, ensuring the crowd only sees relevant data.

Figure 2: Global indicators derived from social media images, visualizing mask-wearing and distancing percentages.

Critical Insight & Future Outlook

Why does this work? The brilliance of VisualCit isn't just the AI; it's the information reduction. By using AI to discard 90% of the "noise," the researchers made it possible for a small group of 38 volunteers to produce world-class statistical indicators.

Limitations:

1. Selection Bias: People who post on Twitter aren't perfectly representative of the entire population.
2. Scalability: While the AI scales, the human crowd is a bottleneck. Future iterations could use the crowd's labels to "teach" the AI to recognize mask-wearing automatically, eventually removing the human bottleneck.

Conclusion

VisualCit demonstrates that social sensing is moving beyond "keyword counting." By combining deep learning's scale with human intuition, we can now "see" social behavior globally, providing policy makers with a powerful tool for evidence-based decision-making in the face of future crises.