How can privacy risks in real-world LLM agent workflows be evaluated and audited?

Why final-response leakage misses most of the risk

Most existing agent privacy benchmarks judge an agent by what ends up in its final answer, or by whether a predefined failure case was triggered. The paper argues this framing is structurally blind to a common failure: an agent can read sensitive files, messages, or records that have nothing to do with the task and still produce a clean final response. Under an outcome-only metric, that unnecessary access is invisible.

The second gap is environmental. Simulated trajectories and synthetic toolkits reduce privacy violations to cases the benchmark author already imagined, and they cannot reproduce the long-horizon, cumulative context accumulation of a real agent that keeps calling tools across many turns. The paper's stated motivation is that privacy violations originate in the execution trajectory, not only at the response boundary, so the trajectory itself has to become the unit of measurement.

A sandbox that runs real MCP tools

AgentPrivArena is built in three layers: a local service environment that holds task state, an MCP tool layer that exposes those services to the agent, and an agent execution layer that runs the tested agent in a sandboxed workspace. The agent interacts with services only through MCP tools, performing searches, reads, writes, and final actions.

The services are six self-hosted open-source applications, each standing in for a privacy-sensitive application function: BookStack 26.03.3 for a knowledge base, Mattermost 11.6.0 for direct messaging, Rocket.Chat 6.13 for team chat, Mailpit 1.29.7 for email, GoToSocial 0.21.2 for social media, and Radicale 3.6.1 for calendar scheduling. Each live service is paired with an MCP server that wraps its native API into agent-facing tools, organized into three operations: discovery, access, and write. The tools deliberately mirror the interfaces of the PrivacyLens simulated toolkit, with the same operations and comparable parameter signatures, so trajectories can be compared across the two settings.

Execution runs inside a local Docker network containing the service instances, the MCP servers, and an OpenHands DockerWorkspace. For each task the runner starts a fresh agent container, resets and seeds the required services with task-specific records, records the trajectory, then removes the agent container and clears service state before the next run. This is what keeps runs reproducible and prevents real-world side effects.

Execution fidelity comes before any privacy number

Before reporting privacy results, the paper validates that the infrastructure itself is not distorting them. Across five models on a 389-task subset, the framework reaches an average task completion rate of 98.96%, with negligible runner-level failures. Individual completion rates range from 97.9% for Mistral-Large-3 to 100.0% for GPT-5.4.

The live environment also produces behaviors a static setting cannot show: agents ask the user for clarification before acting (Clarify, ranging from 1.3% to 4.6% of tasks) and recover from failed tool calls (Err. rec., the share of tasks that hit a tool error and still reached a committed action, 93.3% to 100.0%). The authors read these as evidence that the containerized MCP architecture is a reliable, bottleneck-free foundation for unconstrained agent evaluation, rather than as privacy findings in themselves.

Trajectory metrics count unnecessary access, not just leakage

The framework's measurement contribution is a set of trajectory-level privacy metrics that quantify unnecessary information access beyond final response leakage. The distinction matters because the two metrics answer different questions on different units: response leakage asks what left the system in the answer, while trajectory metrics ask what the agent touched during execution. An agent that reads an unrelated sensitive record and never mentions it scores clean on the first and dirty on the second.

The paper reports that extensive experiments on state-of-the-art LLM agents reveal substantial privacy risks overlooked by existing evaluation paradigms. That claim is specifically about the gap between the two measurement levels, and it is the reason the authors argue trajectory-level auditing matters for trustworthy agent deployment.

Auditing reads and writes at two boundaries

AgentPrivAudit is an active, plug-and-play runtime auditor that intervenes before information accumulated in the agent's context is propagated through an outbound write. It attaches at two boundaries. After each batch of reads — the reads issued in one agent turn — an extractor converts that batch's access-tool observations into a running inventory of explicit information flows, and may return a short advisory note that enters the agent's context before its next step. Tool observations are still passed to the agent unchanged; the extractor never withholds or alters a record the agent asked for.

Each flow candidate is represented by five attributes: sender (origin of the data), recipient (intended consumer), subject (the individual whose data is at stake), type (category of information, such as personal identifier or health record), and transmission principle (permitted conditions for sharing). Before any outbound write executes, a policy-driven arbiter evaluates the proposed transmission against this inventory and returns one of three decisions: PASS permits it, ABSTRACT replaces specific details with a generalized rendering, and BLOCK prevents the flow.

The criterion is pluggable, the mechanism is not

The decision rule is deliberately not hard-coded. A privacy criterion is a pluggable component, while the flow schema, extraction rules, output format, parser, and enforcement path stay fixed, so every policy emits the same structure through the same design. The paper instantiates three criteria, each defined by the question it asks of a flow: CI (the default) asks whether it is appropriate for this principal to share this fact about this subject, with this recipient, via this channel, for this purpose; PII asks whether the fact discloses personally identifiable information about someone other than the principal; and DATA MIN. asks whether the fact is necessary to complete the task.

The paper's discussion draws a sharp line between the rule and the machinery that enforces it. The same rule is close to inert as an instruction and decisive once a mechanism enforces it. The authors explicitly decline to attribute the effect to any single component, because every audited arm bundles a structured flow representation, an enforcement path, and a second model. They offer as a hypothesis, not a result, that most of the benefit comes from making the trajectory explicit rather than from the veto, and note that an arm in which the executor audits itself would test it. The hypothesis fits the criterion: contextual integrity asks about a relation among sender, recipient, subject, channel, and purpose that cannot be evaluated before those fields exist, whereas the PII question concerns a fact in isolation and gains little from the same representation.

Extraction recall bounds what the arbiter can gate

The arbiter only ever decides over flows the read boundary represented, which is 70.5% of what the agent reads. The practical consequence the authors draw is that extraction, not arbitration, bounds the mechanism, and that improving extraction recall is a more direct route to the residual than strengthening the arbiter.

That residual is quantified. The best configuration still leaks on 16–19% of tasks. Most of what the arbiter fails to gate is material it never saw: 29.5% of the facts the agent reads are not represented as flows, and counting items never read, 47.7% of protected facts sit outside the inventory altogether. The characteristic failure at the other end is repeated abstraction, and the paper states there is no mechanism that distinguishes an action that withheld a sensitive detail from one that withheld the answer. The audit is also fail-open by design, so the authors say it should not be relied on as a security control against an adversary who can induce the failure.

What the headline numbers rest on

Every headline number rests on a 2-of-3 majority of LLM judges. Agreement is high where the judgment is easy (κ = 0.82–0.87) and lowest on the condition the paper advocates (κ = 0.61–0.71), and three judges applying a rubric inherited from prior work can share its bias. A blinded two-annotator package is prepared but not yet labelled, so no claim in the paper has been checked against a human verdict. The authors describe the metric as a stable instrument, not a verified one.

Scope is also bounded. The setup covers one principal, one agent, and text, with multimodality treated as a scoped probe. Of 493 source scenarios, 389 convert, for service-wrapper and seed-mapping reasons rather than privacy difficulty. The conversion is not uniform: the direct-messaging wrapper cannot seed group conversations, so scenarios whose violation depends on a broad internal audience are absent, which the authors expect makes both the leak rates and the reduction estimates conservative. All records the agent reads are synthetic, derived from authored PrivacyLens vignettes, and no task touches a commercial account.

Where the audit pays off and where it does not

The paper's own priority ordering is the useful takeaway: the binding constraint on runtime privacy auditing is the read-boundary extractor, not the arbiter's policy choice. The 70.5% figure for facts represented as flows, the 29.5% of read facts never represented, and the 47.7% of protected facts outside the inventory altogether all point at the same place, and the authors say improving extraction recall is the more direct route to the residual than strengthening the arbiter.

That ordering also tells you how to read the three criteria. Because the criterion is pluggable while the flow schema, extraction rules, output format, parser, and enforcement path are held fixed, comparing CI, PII, and DATA MIN. is a comparison of criteria rather than of implementations — but the paper's own discussion warns that the criterion alone does not carry the effect, and that the benefit cannot yet be attributed to the veto rather than to making the trajectory explicit. For a reader deciding what to trust here, the split is between the framework's demonstrated execution fidelity and trajectory-level measurement on one side, and the audit's unverified judge-based metric and fail-open enforcement on the other.

Sources used in this answer