Prebuilt memory cannot anticipate later queries
Most agent memory systems build memory before they know what will be asked. The paper argues this query-agnostic construction is both wasteful and lossy: it spends preprocessing effort on details no query needs, while discarding details that later turn out to be essential.
Recent work moves adaptation to runtime, but the paper identifies three limits. Runtime control stays structurally rigid, relying on hand-designed operations, fixed pipelines, budget tiers, or heuristics. Trade-off optimization is incomplete, often targeting only token usage or monetary cost while user-facing latency is rarely an explicit objective. And most methods assume a fixed processing backend, ignoring how curation quality, cost, and latency vary across heterogeneous models and modalities.
MemPilot's design principle follows directly: keep a query-agnostic memory bank for cheap access, but also preserve the raw interaction history so a query-specific decision can still be made at runtime.
A policy chooses between retrieving and curating
At inference time, a multi-step LLM policy builds an orchestration trajectory by iteratively choosing between two kinds of action. A Retrieve call pulls evidence from the query-agnostic memory bank and incurs no external-model cost, though the orchestrator generations around it still count. A Curate call delegates query-specific processing of raw multimodal history to an external model.
Each Curate decision jointly controls four things: how much evidence to request, what curation instruction to give, which model to route to, and whether images are included. That combination is what lets the policy allocate runtime computation at a finer grain than a fixed pipeline, and it is why the paper frames the contribution as orchestration rather than a single new memory operation.
The external pool is heterogeneous by design, spanning text-only LLMs and vision-language models with different quality, price, and latency profiles. The policy model itself is Qwen3-4B-Instruct-2507, with Qwen3-Embedding-0.6B for embeddings; retrieval returns at most five items per call, and at most two tool calls run in parallel per turn.
Three objectives are optimized separately, then combined
MemPilot treats answer quality, monetary cost, and latency as competing objectives rather than folding them into one scalar. Cost is estimated in USD per million tokens from orchestrator and external-model prices, with exact API-reported cost used when available. Latency uses a deterministic affine proxy rather than observed wall-clock time, because real request latency varies with network conditions and transient serving load; the proxy combines a fixed per-call overhead with per-input-token, per-output-token, and per-image terms calibrated from repeated non-streaming requests.
Two training details matter for reading the results. Objective-wise advantage decoupling estimates each objective's advantage separately before aggregation, so one objective does not dominate the others during policy updates. Prefix-based marginal utility estimation provides finer credit assignment across multi-step rollouts, since a single trajectory contains several decisions whose individual contributions are otherwise hard to isolate.
A format gate also shapes the reward: a trajectory must contain exactly one non-empty final answer block, balanced and parseable action tags, and valid model or memory identifiers. Format-invalid trajectories receive a fixed reward of −1 and are excluded from cost and latency comparison, so a malformed trajectory cannot look cheap. Because the evaluated tasks require memory evidence, answering without calling Retrieve or Curate is also invalid.
Removing delegation costs the most quality
The ablation study averages results across five benchmarks using Qwen3-VL-4B-Instruct as the answer model. The full MemPilot configuration reaches an average Judge score of 37.40 at a cost of 1.1e-2 USD per sample.
Removing LLM/VLM delegation drops the average Judge score from 37.40 to 14.58, the largest single degradation. Removing marginal utility estimation gives 14.82, and removing retrieval–instruction decoupling gives 15.35. Removing objective-wise advantage decoupling is less damaging but still clear, falling to 21.94. Every ablated variant consumes less inference cost, yet the accompanying quality losses are large enough that the paper reads the full design as a better quality–cost trade-off rather than simply more computation.
The per-benchmark table shows the same ordering. The full configuration has the highest Judge score on all five benchmarks, and the same components produce the largest drops on both in-domain and out-of-distribution benchmarks.
Preference sweeps trace broader frontiers than baselines
Because the policy is trained under different optimization preferences, sweeping those preferences traces a frontier rather than a single operating point. The paper reports that these sweeps yield broader performance–cost–latency frontiers than existing trade-off-aware baselines across five multimodal agent-memory benchmarks.
The runtime allocation figures show how the policy spends computation as preferences shift. On Mem-Gallery, moving toward cheaper settings reduces policy steps from 2.16 to 1.00, cuts Curate actions from 51.4% to 0%, lowers evidence count from 3.93 to 3.00, and drops image access from 19.5% to 0%. The same pattern appears on WorldMemArena-Lifelong, H2HMem, MemEye, and MemLens: cheaper and faster settings mean fewer steps, fewer Curate calls, less evidence, and no images.
Model selection also shifts with the preference. The reported distributions split routed calls across LLMs under 70B parameters, LLMs at or above 70B, VLMs under 70B, VLMs at or above 70B, and proprietary VLMs, so the frontier reflects both how much work is done and which backend does it.
Memory bank choice sets the operating regime
MemPilot is not tied to one query-agnostic memory construction. Replacing the default bank with SimpleMem or A-MEM preserves clear performance–cost frontiers without retraining, and the paper reports that runtime orchestration behavior persists across all three.
The banks occupy distinct cost regimes. The default bank generally provides the more cost-efficient operating points. A-MEM incurs higher overall cost but often reaches a higher performance ceiling, particularly on WorldMemArena, MemEye, and MemLens. SimpleMem typically sits between the two and stays competitive on Mem-Gallery and H2HMem.
This matters for interpreting the frontier: the upstream memory representation sets where the curve sits, while runtime orchestration determines how far along it the system moves as more computation is allocated.
Where these results transfer and where they do not
The evidence covers five multimodal agent-memory benchmarks, with MemLens and MemEye marked as out-of-distribution evaluations. Quality is measured by benchmark-specific scoring: Mem-Gallery and H2HMem use a five-level rubric from 0 to 1, WorldMemArena assigns 1 to Correct and 0 to Hallucination or Omission, MemEye quantizes to {0, 0.5, 1}, and MemLens uses type-specific binary rubrics. Judge-model token usage is retained for audit but excluded from the system's cost, latency, and invocation metrics.
Resource numbers follow their own denominators. Cost per question is divided by the number of unique dataset–conversation pairs, latency is averaged over questions, and call counts include only attempted routed external-LLM invocations, excluding local policy generation, retrieval-only operations, judge requests, optional decoupled answer generation, and offline memory construction. A cost figure and a latency figure therefore answer different questions and should not be substituted for one another.
The latency numbers are proxy estimates, not measured wall-clock latency, and the cost numbers are token-based estimates unless the API returned an exact figure. Both are calibrated for the operating range used in these experiments, so the frontier shape is the transferable result rather than any single cost or latency value.
The preference setting is the real control
The most decision-relevant result is not a single score but the relationship between preference and allocation. Across all five benchmarks, tightening the cost or latency preference predictably reduces policy steps, Curate actions, evidence count, and image access, and the paper reports that the resulting frontiers are broader than those of trade-off-aware baselines.
That makes the preference setting the practical control surface. A reader comparing MemPilot against a fixed pipeline should compare frontier positions at the same cost or latency budget, not headline quality at an unspecified budget. And because the memory-bank analysis shows the default bank, SimpleMem, and A-MEM occupying different cost regimes without retraining, the achievable frontier depends on which upstream memory representation is already in place.
The ablation ordering gives the priority for anyone reproducing or extending the design: delegation to heterogeneous models matters most, followed by marginal utility estimation and retrieval–instruction decoupling, with objective-wise advantage decoupling contributing a smaller but consistent gain.
Sources used in this answer
MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents
A multi-step LLM policy trained with reinforcement learning chooses between retrieving from query-agnostic memory and delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs, producing broader performance–cost–latency frontiers than trade-off-aware baselines across five multimodal agent-memory benchmarks.