[ArXiv 2025] Memex: Scaling Long-Horizon Agents via Indexed Experience Memory

TL;DR

Memex transforms the LLM agent from a "forgetful reader" into an "organized archivist." By treating memory operations as first-class actions (like using a tool), Memex keeps the active reasoning state small while archiving full-fidelity evidence in an external store. Through a specialized RL framework (MemexRL), the agent learns to build its own "index map," leading to a 3.5x jump in task success while significantly cutting context costs.

Problem: The "Long-Horizon" Bottleneck

Modern LLM agents are fundamentally throttled by their context windows. As an agent explores a repository or a house, its trajectory grows. Current solutions usually fall into two traps:

1. Lossy Compression: Summarizing 1,000 lines of logs into "the test failed" loses the specific Error ID needed to fix it.
2. Ambiguity in RAG: Semantic retrieval often surfaces near-duplicates, forcing the model to re-parse noisy history.

The authors note that humans don't keep every word of a 50-page doc in working memory; we keep a compact mental index and refer back to specific notes. Memex brings this "extended mind" philosophy to AI.

Methodology: The Memex Loop

Memex splits information into two layers:

• The Working Context (${\mathbf{M}}_{\mathrm{work}}$): A compact "Index Map" containing high-level progress and stable pointers (e.g., ctx_locations).
• The Experience Store ($\mathcal{D}$): A key-value database holding verbatim artifacts (tool logs, code snippets) indexed by those pointers.

Two-Phase Execution

1. CompressExperience: The agent decides to "fold" its history. It writes a summary for its own future self and archives the "raw" data into the database.
2. ReadExperience: When the agent needs a specific ID or log, it doesn't guess; it explicitly "dereferences" the index to bring that specific block back into context.

MemexRL: Teaching the Agent to Value its Future

Memory management is a "delayed gratification" problem. A good index today only pays off 50 steps later. The authors use GRPO (Group Relative Policy Optimization) with three specific penalties:

• Context Overflow: If the agent forgets to compress and exceeds the token budget.
• Redundancy: If the agent re-runs a tool (e.g., ls) instead of reading its previous notes.
• Malformed Invocations: Ensuring the agent maintains the internal database structure correctly.

The Performance Leap

The results on a hardened version of ALFWorld (where navigation is impossible without remembering specific hidden IDs) are striking.

Key findings include:

• Success Rate: Jumped from ~24% (base) to 85.6% (MemexRL).
• Efficiency: Peak context usage dropped by 43%.
• Learned Strategy: Remarkably, after RL, the agent actually called Compress less often but called Read 7x more. This suggests the agent transitioned from "panicked summarizing" to "strategic archiving."

Professional Insight: Why This Matters

The industry is currently obsessed with "Infinite Context" (e.g., Gemini's 2M context). However, Memex argues that raw length isn't enough—attention still dilutes over long sequences (the "lost in the middle" phenomenon).

By enforcing a Pointer-Based Architecture, Memex forces the model to attend only to the "Index Map" to plan, and only to "Dereferenced Evidence" to act. This modularity resembles modern software engineering—keeping the main function clean while offloading data to a database.

Critical Analysis & Limitations

• Cold Start: The model requires supervised "warm-starting" because RL struggle to discover the complex CompressExperience syntax from scratch.
• Heuristic Truncation: In experiments, summaries were hard-truncated to 300 tokens, which might be too aggressive for tasks requiring dense global state.
• Future Direction: The next step is clearly state-tracking across episodes, allowing an agent to build a "Memex" of an entire codebase or environment over weeks, not just minutes.

Conclusion

Memex proves that the bottleneck for long-horizon agents isn't just the model's size, but its ability to curate its own history. By turning memory management into a learnable skill, we can build agents that operate for hundreds of steps without getting "lost in the sauce" of their own logs.