PolicyGapper: Catching the Silence—How LLMs Audit Hidden Data Practices in Google Play

TL;DR

Researchers have developed PolicyGapper, the first LLM-driven pipeline designed to automatically audit the alignment between an app's "Data Safety Section" (the summary you see in the store) and its full "Privacy Policy" (the legal text nobody reads). Analyzing 330 top apps, it uncovered nearly 2,700 missing declarations, proving that what apps tell you in the store is frequently a sanitized version of their actual legal claims.

The "Transparency Gap": Why App Stores Are Failing

In 2022, Google mandated the Data Safety Section (DSS) to act as a "nutrition label" for privacy. However, a label is only as good as its accuracy. Currently, Google relies largely on developer self-reporting. This creates a massive loophole: developers often list fewer data practices in the DSS than they admit to in their lengthy legal Privacy Policies (PP).

Existing tools either analyze code (static analysis), which is easily hidden by obfuscation, or use keyword matching, which fails to understand the complex legal nuances of intent and context. PolicyGapper bridges this by treating compliance as a semantic reasoning task.

Methodology: The Multi-Stage LLM Auditor

The methodology moves beyond simple chat queries. It utilizes a sophisticated workflow that transforms unstructured web-scraped policies into structured compliance reports.

1. The Multi-Prompt Strategy

The authors discovered that asking a model everything at once leads to "Lost in the Middle" syndrome—where the model ignores details in long texts. Instead, they broke the audit into three specialized prompt scopes:

• Device Data: Technical identifiers and hardware info.
• User Data: Personal identifiers (Name, Email).
• User-Generated Data: Photos, messages, and browsing history.

2. Architecture Overview

The system follows a five-module cycle:

The LLM Analyzer acts as the specialized auditor, while a Post-Processing module filters out "hallucinations" by checking results against Google's official exclusion constraints (e.g., data processed only on-device doesn't need to be declared).

Key Results: What Are Apps Hiding?

The results from the Q3 2025 dataset are startling. PolicyGapper identified an average of 8 omissions per app.

The Most Frequent "Omitted" Declarations:

1. Approximate Location (216 Top Apps): Developers often assume that if they collect an IP address, they don't need to check the "Location" box, despite IP being a method of location tracking.
2. Web Browsing History (153 Top Apps): Frequently omitted when apps use embedded "WebViews" for internal browsing.
3. Email Address (144 Top Apps): Often treated as "account management" and ignored in the summary, despite being a primary PII.

The study found that Gemini 2.5 Pro (PDF mode) significantly outperformed smaller or text-only models, achieving an F1-score of 0.77 by leveraging native document understanding.

Critical Insight: Semantic Reasoning vs. Pattern Matching

The paper's strongest contribution is proving that LLMs can handle Exemption Logic. For example, Google Play allows developers to skip certain disclosures if the data is end-to-end encrypted or processed locally. PolicyGapper’s post-processing prompt allows the model to distinguish between "we collect your data" (an omission) and "we process your data on-device only" (a valid exclusion), a feat previously requiring human lawyers.

Conclusion and Future Outlook

PolicyGapper represents a shift in regulatory technology (RegTech). By achieving an F1-score of 0.76 on a manually validated subset, it demonstrates that automated auditing is no longer a futuristic concept.

Limitations: The study notes that LLMs are still non-deterministic; three identical runs might produce slightly different counts of omissions (Standard Deviation ~10%). Furthermore, while it catches "legal-to-label" lies, it still doesn't verify if the app is lying in both the policy and the label by checking the actual binary code.

Future Work: The authors plan to expand this to Apple's Privacy Labels and integrate dynamic analysis to ensure the app's behavior matches its claims.