Agent Memory Migration: Fixed Schemas Survive Writer Swaps, Model Written Notes Do Not

Core Snapshot

TL;DR

This paper asks whether an agent memory store remains usable after the model around it changes, separating memory creation from later reading. The authors compare raw long-context history, retrieval chunks, model-written notes, and a fixed-schema knowledge graph under controlled model and embedding migrations. A fixed schema transfers almost perfectly, with absolute accuracy changing by only 0.0004, while notes transfer asymmetrically: one direction improves accuracy by 9.91 percentage points, while the reverse direction loses 13.28 percentage points. A mixed embedding index captures only 4.96 points of the 11.90 point full re-embedding gain, and store-only repair of notes never reaches the 90 percent target in any of the 48 histories.

Positioning in the Field

The work is best read as an empirical reliability study rather than a new agent architecture or a leaderboard submission. It does not claim that one memory format is universally best. Instead, it introduces a controlled testbed for a question that production systems usually treat as a deployment detail: how much of an agent's remembered past survives when the writer, reader, embedder, or repairer changes. Compared with prior memory papers that focus on long conversation evaluation or memory design, this paper is closer to a migration contract: it tests formats under identical history, budget, scoring, and reader settings, then decomposes loss into writing, retrieval, and reading stages.

Problem and Motivation

A memory store outlives the model that wrote it. A coding agent may accumulate repository decisions for months, while the base model is replaced because of latency, cost, context length, or provider availability. In such systems, the deployment rarely fails loudly. The database still loads, the retrieval service still returns results, and the new model still answers questions. Yet performance can fall because the new reader interprets old notes differently, because old and new embeddings occupy different vector spaces, or because the only surviving copy of a fact was already discarded during consolidation.

The failure mechanism is therefore not simply "accuracy is lower." It is a lifecycle mismatch: the writer commits a lossy interpretation of experience, the retriever routes evidence, and the later reader consumes that interpretation without access to the original evidence. The authors identify three concrete risks. First, natural-language notes are a model-specific compression, so one model's phrasing, omission, and merge choices may be difficult for another model to decode. Second, embedding versions can remain dimensionally compatible while becoming semantically incompatible, which allows a mixed index to run online while retrieval quality silently declines. Third, compact stores destroy the source of truth; if raw history is discarded, a repair model cannot recover omitted facts from the store alone.

This gap matters because existing memory systems already provide import, export, re-indexing, and migration operations. What they do not reliably provide is a measured, format-specific statement of how much capability remains after a migration, whether loss happens before or after retrieval, and what it costs to recover. The paper's central insight is that memory portability should be evaluated as a directional and component-specific property, not as a single scalar memory score.

Core Chapter: A Controlled Framework for Memory Portability

The Unit of Analysis Is a Migration, Not a Store

The paper divides an agent memory system into four roles: the actor producing experience, the writer creating memory, the embedder supporting vector search, and the reader later answering queries. This decomposition is the essential methodological move. Instead of asking whether a memory format is good in isolation, the study asks whether a store created by one configuration remains useful under a later configuration. The fixed history is treated as the control variable, while the writer, reader, embedding model, or repair source changes one at a time.

Figure 1 illustrates the four evaluated formats. LC-RAW preserves the entire transcript and can be read directly, but every read pays the full context cost. RAG preserves original transcript chunks and retrieves a subset, reducing reading cost but depending on search quality. NOTES asks a model to compress the history into natural-language summaries under a byte budget, making it compact and human-readable but highly lossy. KG-fixed converts experience into subject-predicate-object claims under a shared schema, preserving only facts that the schema can represent. The important contrast is not storage size alone: NOTES and KG-fixed are constructed by models, but the latter constrains the writer through schema, relation names, source fields, and reading rules.

The first formal measure is Retained Performance After Swap, or RPAS. For migration from writer A to reader B under memory format $s$:

where $s$ is the memory format, $W$ is the writer, $R$ is the reader, $c$ is the chance baseline, and $N_s(B)$ is the score achieved by reader B when reading a store that B itself wrote. The numerator measures how well the new reader performs using inherited memory, while the denominator normalizes that number by the reader's own capability on self-created memory. In the argument chain, RPAS separates migration loss from a reader's general competence: a low RPAS means the inherited store fails to preserve the new model's ability, not merely that the model is weak. An RPAS of 1 means no inherited-store loss, below 1 means degradation, and above 1 means inherited memory works better than the reader's own store. The authors guard against unstable ratios by using RPAS only when the own-store score is at least 0.20 above chance; the paper states that this threshold never excluded the primary reported configurations, which strengthens the use of the metric but also shows that the experimental floor was well above chance.

Recovery is measured separately by Cost to Recover, or CTR. For history $i$, target $X$, repair method $\rho$, budget rung $b$, and inherited store $S_A$:

Here $X$ is a threshold such as 0.90, 0.95, or 0.99; $c(\rho ,b)$ is the repair cost, reported as tokens, GPU-hours, or dollar cost; and $N_{s,i}(B)$ is the new reader's own-store performance on history $i$. The constraint says that the repaired store must recover at least a fraction $X$ of the fresh-store performance. If no method and budget pair reaches the target, ${\mathrm{CTR}}_i(X)$ is infinite. This formula is the bridge from diagnosis to engineering decision: it tells teams not just whether memory is damaged, but which repair path is cheapest to return the agent to an acceptable operating point. It also makes the raw-retained versus store-only repair comparison meaningful, because some losses are not bugs in the reader but irreversible consequences of the store construction process.

Figure 2 summarizes the full pipeline. Synthetic histories are converted into four memory formats. The evaluation then changes one component at a time: the model pair, the embedding index, the repair source, or the injected evidence. This design lets the paper attribute failure to a stage rather than merely observing an end-to-end drop.

Four Migration Tests Across Two Open-Weight Models

The empirical setup is deliberately synthetic to make loss traceable. The study uses 48 scripted histories, each with 160 questions covering direct facts, changes over time, contradictions, multi-step relations, and aliases. Randomized entity and value codes prevent pretraining leakage, and exact scoring avoids LLM judges. The models are two similarly scaled open-weight systems, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M. For embeddings, the upgrade path is BAAI/bge-large-en v1.0 to v1.5. Both produce 1024-dimensional vectors, so a mixed index can coexist without a dimension error, yet the paper verifies that the spaces are actually different through a cross-space cosine of 0.904, below its identity threshold of 0.999.

The main experiments are organized as four tests. The first compares writer-swap behavior for NOTES and KG-fixed, using LC-RAW and RAG as writer-free controls. The second tests embedding migration across old, new, full rebuild, mixed 50/50, and an ideal-router upper bound. The third compares repair strategies under matched budgets: raw-retained reconstruction of notes, store-only rewrite of notes, RAG re-embedding, and KG-fixed schema rebuild. The fourth supplies controlled evidence to separate construction loss, retrieval loss, and reader loss. The latter is diagnostic rather than one of the original four planned hypothesis tests.

The planned contrasts are summarized below. Each test uses a one-sided comparison against a five-point threshold, with Holm adjustment across the four hypotheses.

Table 2 reports the four planned contrasts across 48 histories.

The result pattern is instructive. The strongest planned findings are not that notes are generally worse than schemas, but that mixed embedding indexes are materially inferior to full rebuilds and that raw-history repair is more effective than store-only repair. The apparent failure of H1 and H4a in the one-sided planned tests does not mean notes are safe. It means that averaged symmetric measures can cancel directional effects. The paper later reports unplanned checks that place H1 and H4a within five points, but those were not part of the original signed test family. This distinction is crucial for interpreting the study: the signed plan and Holm correction are credible, while the post hoc symmetry checks are supplementary.

The resource controls also matter. Writers receive the same token allowance, NOTES stores are capped at 128 KiB, readers use the same context budget, and repair budgets are matched. Still, the formats are not identical in how they are read. A NOTES reader receives about 111 KiB of consolidated text under the cap, while a KG-fixed reader queries a larger structured store. Thus the NOTES versus KG-fixed comparison confounds format with access method. The authors are explicit about this, and it should constrain any practical recommendation.

Diagnostics Locate Loss Before the Reader Sees Evidence

End-to-end answer accuracy is too coarse for memory migration. If a migrated agent gives a wrong answer, developers may replace the reader and still miss the failure. The paper's diagnostic intervention compares three reading conditions for a matched writer-reader cell: normal reading, reading with the exact stored evidence supplied, and reading with the original raw event supplied. This gives a clean additive decomposition of total loss:

where $s$ is the memory format, $i$ indexes histories, $u$ indexes matched writer-reader cells, and $T_{s,i,u}=1-a_{s,i,u}^{\mathrm{normal}}$ is loss relative to perfect exact-match accuracy. The retrieval component is $L_{s,i,u}^{\mathrm{retrieval}}=a_{s,i,u}^{\mathrm{stored}}-a_{s,i,u}^{\mathrm{normal}}$, the construction component is $L_{s,i,u}^{\mathrm{construction}}=a_{s,i,u}^{\mathrm{raw}}-a_{s,i,u}^{\mathrm{stored}}$, and the reader residual is $L_{s,i,u}^{\mathrm{reader}}=1-a_{s,i,u}^{\mathrm{raw}}$. In the paper's argument chain, these terms distinguish whether the system failed because the right evidence was missing from the store, failed to retrieve it, or received it but still did not use it. The ceiling of 1 is useful but conservative: it makes channels additive, while the reader residual still absorbs model limitations, probe difficulty, and any artifacts of controlled evidence injection. The decomposition is therefore diagnostic, not a full causal proof.

Table 6 reports the diagnostic decomposition pooled across 48 histories.

The table changes the engineering diagnosis. For NOTES, most loss happens before the reader ever sees evidence: the store writer omitted or distorted critical content. For RAG, the stored chunks are nearly intact, but retrieval fails to deliver them. This supports a practical order of operations: first check whether evidence exists and is delivered, then consider changing the reader. For notes, the remedy is better evidence preservation during writing. For retrieval, the remedy is better chunking, indexing, or ranking.

Experiments and Evidence

Structured Facts Transfer; Natural-Language Notes Do Not

Figure 3 is the paper's most compact summary of the central findings. It shows that notes transfer sharply by direction, that the mixed embedding index leaves a large portion of upgrade gain unrecovered, that NOTES loss concentrates in store construction while RAG loss concentrates in retrieval, and that store-only NOTES repair fails to reach the 90 percent target.

Table 3 reorganizes the main accuracy results around the distinction between own-store and inherited-store reading.

Table 3 reports accuracy across 48 histories.

The fixed-schema result is the clearest positive finding. For KG-fixed, the absolute accuracy change after a writer swap is only 0.0004, and both readers maintain high own-store accuracy: 0.8456 for Llama and 0.9878 for Qwen. This supports the authors' design rationale: when the schema fixes record structure, relation names, source fields, and reading rules, the writer's role becomes value extraction rather than free-form representation. The migration becomes a mechanical fill of known slots, which is easier for a different model to interpret.

The NOTES result is the strongest cautionary finding. The average swap effect appears small only because two large directional effects cancel. Qwen-written notes help Llama by 9.91 points over Llama's own notes, while Llama-written notes hurt Qwen by 13.28 points relative to Qwen's own notes. The paper's Appendix B evidence shows that this asymmetry is not merely a byte-limit artifact. In a 160 KiB calibration sweep, Qwen retained 85.6 percent of required evidence in about 143 KiB, whereas Llama retained 64.5 percent despite using about 159 KiB. The practical lesson is that a note-writing model is not a neutral encoder; it is part of the memory definition. A migration test that averages over writer-reader pairs can hide catastrophic failures.

The writer-free controls are also important for interpreting absolute numbers. LC-RAW is high because the reader sees the full transcript, while RAG is much lower because retrieval often fails even before model migration. This is not merely a long-horizon problem; it is an independent retrieval quality problem. The RAG baseline sits between 0.535 and 0.565, while LC-RAW ranges from 0.712 to 0.911. The diagnostic table below explains why that gap matters.

Embedding Migration Fails Silently, and Partial Repair Captures Only Part of the Gain

The embedding test uses bge-large-en v1.0 as the old index and v1.5 as the new index. Table 4 gives the core comparison across 48 histories and two readers.

Table 4 reports embedding migration results.

The most operationally relevant number is the gap between the mixed index and full re-embedding. A full rebuild adds 11.90 points over the old index. A 50/50 mixed index adds only 4.96 points. The ideal-routing upper bound adds 17.03 points, but it assumes the system already knows which index contains the answer, so it is an oracle rather than a deployable design. The mixed index therefore recovers less than half of the full rebuild benefit and, as the paper's Figure 3 summarizes, leaves about seven percentage points unrecovered. Because both embedding models produce 1024-dimensional vectors, this failure can occur without any runtime error. The service remains online, but retrieval quality declines.

The paper's design choice is also a warning. The RAG pipeline uses a single-stage dense retriever, event-based chunks of about 512 characters, cosine top-8 retrieval, no reranker, and no lexical search. The authors explicitly state that this setup is not intended as a universal limit on RAG. Still, the structural point survives the specific pipeline: chunking and retrieval should be evaluated independently of the reader model. A strong reader cannot synthesize evidence that never reaches its context.

Repair Is Cheap for Structures and Expensive for Notes When Source Evidence Is Missing

Repair experiments distinguish three modes of damage. A compact store may omit facts. A vector index may use an incompatible space. A raw event history may still exist and allow reconstruction. Table 7 reports repair success and median cost among successful histories for the tested grid.

Table 7 reports repair success across 48 histories.

The repair pattern is almost as important as the initial migration result. Structured repairs are cheap and reliable: RAG re-embedding reaches all targets for all 48 histories in both directions at a median cost of 0.013 USD, and KG-fixed schema rebuilds succeed nearly universally at near-zero additional cost. NOTES repair is a different story. Store-only rewriting never reaches the 90 percent target for any history, because a rewrite cannot recreate evidence that was omitted when the notes were written. Raw-history repair helps only when the repair model can actually finish the reconstruction. With Qwen as repairer, raw-history reconstruction reaches 90 percent for 34 of 48 histories at a conditional median cost of 0.76 USD. With Llama as repairer, no history reaches the target because every attempt reaches the output-token limit. The paper acknowledges that repair model and migration direction change together, so this is not a clean capability comparison.

This result gives a concrete engineering principle: retain protected source histories where policy allows, but test the exact repair model and budget before relying on them. Raw retention is not a guarantee. It is an insurance policy whose payout depends on the repairer's context, output limits, and ability to traverse the history.

Diagnostic Evidence Shows Where to Intervene

The diagnostic decomposition in Table 6 gives the troubleshooting order. For NOTES, store construction accounts for 0.467 of the pooled mean deficit of 0.584, or about 80 percent of loss. Retrieval inside notes accounts for only about 6 percent. This means that if a note-based agent fails after migration, the first suspect is not the new reader's search or phrasing; it is the old writer's omission or distortion. The paper's style-rewriting intervention supports this conclusion: Qwen rewriting Llama notes changes accuracy by only -0.012, and Llama rewriting Qwen notes changes accuracy by -0.042 despite reaching only 42 percent of intended notes. In the worst direction, Llama-written notes read by Qwen have a construction share of 88.8 percent.

For RAG, the failure is almost entirely upstream of the reader but different in mechanism. Retrieval accounts for 0.364 of the 0.450 pooled deficit, or about 81 percent, while construction loss is only 0.005. Once the correct chunks are supplied, performance rises from the 0.53 to 0.56 range to 0.88 to 0.95. This tells us that the stored text is generally sound; the router picks the wrong evidence. A developer who sees low RAG accuracy and replaces the reader may fix nothing.

The two cases share a common lesson with different remedies. NOTES needs evidence-preserving writing and raw retention. RAG needs recall measurement, better indexing, and embedding-space isolation. Neither failure should be blamed primarily on the reader unless the correct evidence is already present and delivered.

Evidence Quality and Scope

The study has several strong methodological features. Exact scoring with randomized codes removes LLM-judge variance and pretraining leakage. The use of only two similarly scaled open-weight models is a limitation, but the paper supports its measurement reliability: repeated live reads vary by only 0.0016, far below the effects reported. The sample size calculation is transparent: 12 histories estimate variance, a planned five-point effect and 80 percent power lead to N=48, and 48 histories are then used. The signed Git tag records four planned tests, a five-point threshold, correction procedure, and exclusion rules before main collection. Holm correction across the four hypotheses reduces the family-wise error rate. The later 10,000-resample bootstrap confirms the same decisions for H2 and H7a, with lower bounds of 5.8 and 7.9 percentage points and corrected p-values below 4.0 times ten to the minus fourth.

The limits are also substantial and are honestly reported. The workload is synthetic and scripted, and all histories fit within both context windows with at least 4,096 tokens unused. Real conversations, subjective preferences, conflicting instructions, and much longer histories may introduce additional portability problems. The study tests only one cross-family migration and one embedding version pair, so it should not be read as a survey of all model upgrades. KG-fixed portability depends heavily on the schema being well aligned with the task; the authors built the schema from event fields before seeing results and did not tune it using accuracy, but this does not imply that arbitrary knowledge graphs are universally portable.

There is a format-access confound in the main NOTES versus KG-fixed comparison. NOTES readers consume compressed text, while KG-fixed readers query structured records. The paper's fixed token and byte budgets do not equalize access method. This weakens any claim that schemas beat notes as a pure representation effect. It strengthens, however, a narrower claim: schema-bound facts are easier to reuse when the reader changes. The embedding result has a similar boundary. The exact 4.96 versus 11.90 numbers apply to bge-large-en v1.0 to v1.5 in a specific dense retrieval setup, but the underlying issue—compatible dimensions do not imply compatible retrieval spaces—is general.

Repair experiments are the least clean part of the evidence because repair model and migration direction are confounded. Raw history helps Qwen repair Llama-written stores but fails when Llama is the repairer, partly because every attempt hits an output-token limit. That result is meaningful operationally, but it should not be interpreted as a pure capability ranking. The diagnostic interventions are also not a complete causal proof: style rewriting is an incomplete intervention, and KG-fixed is excluded from the Table 6 decomposition because isolated item injection did not reproduce its normal traversal behavior.

Finally, the signed plan is best described as an internal analysis lock, not a full preregistration. The bootstrap was run after the t-tests, although the decisions matched. This matters less for the supported H2 and H7a findings than for H1 and H4a, where later unplanned checks are used to interpret direction cancellation. The most robust claim is therefore the direction-specific nature of notes, not a symmetric average penalty.

Deep Insight and Conclusion

The paper's lasting contribution is conceptual: agent memory should be evaluated as a lifecycle object, not as the current model's output. The strongest mechanism-level result is that fixed schemas reduce coupling by moving the model from "compose the representation" to "extract values into a predeclared structure." This does not make knowledge graphs magic, but it gives future models a stable contract. NOTES, by contrast, are model-written encodings whose meaning depends on the writer's compression policy. The asymmetric migration result shows why averaging directions is dangerous: a store can appear portable only because one model's weakness cancels another's strength.

The embedding result exposes another silent failure class. Vector dimensions can match while semantic spaces diverge, so an old-new mixed index can continue serving requests while retrieving the wrong evidence. The practical response is not clever blending but isolation: build and validate the new index separately, then cut over only after testing. The repair result closes the loop: compact stores are only as recoverable as the evidence they retain. If raw history is discarded, a repair model cannot reconstruct omitted facts; if raw history is retained, repair still requires a model that can finish within budget.

The most useful engineering takeaway is a migration checklist derived from the evidence. First, define the class of memories that must persist: facts, preferences, decisions, and unfinished tasks may fail differently. Second, record provenance for stores: writer model, prompt version, embedding version, chunking parameters, and schema. Third, test each writer-to-reader direction separately, especially for notes. Fourth, never treat mixed embedding indexes as safe unless queries are explicitly version-routed. Fifth, keep protected raw evidence where policy allows and test the exact repairer before relying on reconstruction.

The limitations point to concrete future work rather than vague extension. A next step is format-specific calibration of small migration probes: the paper's 20-probe check tracks full scores reasonably across histories and model pairs but is biased for unseen memory formats, so it cannot replace full evaluation for final approval. Another is schema portability under natural task distributions, where the relevant entities and relations are not pre-scripted. A third is multi-stage retrieval robustness under mixed embedding versions, including hybrid lexical and dense retrieval, rerankers, and query rewriting. A fourth is privacy-preserving evidence retention: if raw histories are needed for repair, systems need encrypted, access-controlled, and deletion-policy-aware source stores.

In conclusion, the study shows that an agent can keep a memory store and still lose its past. Fixed schemas survive writer swaps almost perfectly in this controlled setting, model-written notes survive only when direction-specific compatibility is tested, partial embedding migrations forfeit most of the available upgrade gain, and repair without source evidence is impossible for notes that have already dropped facts. The broader design lesson is that durable agent memory is not a database bolted to a model. It is an interface between generations of models, and it should be versioned, tested, and kept repairable.