PRO-LONG: Decoding the Power of Programmatic Memory in Long-Horizon Tasks

TL;DR

Large Language Model (LLM) agents often struggle with "long-horizon" tasks—problems that require memory and planning over hundreds or thousands of steps. PRO-LONG is a breakthrough framework that solves this by giving agents a programmatic memory. Instead of summarizing the past (and losing detail), it logs every single interaction and teaches the agent to use code (like Python or Grep) to search its own history. The results? It hits SOTA on the ARC-AGI-3 benchmark while slashing token costs by up to 5.8x.

The Problem: The High Cost of Forgetting

In long experiments or games, agents face a brutal tradeoff. If they try to remember everything, their context window overflows, leading to context rot and soaring costs. If they summarize their history (the "Write" operation), they inevitably prune details that might only become important much later.

The researchers found that existing "agent harnesses" are often too complex, relying on sub-agents or specific "world models" that are hard to generalize. The fundamental bottleneck isn't just how much the model remembers, but how it accesses that information without getting lost in the noise.

Methodology: Simplicity through Programmatic Search

The core philosophy of PRO-LONG is built on three pillars: Simplicity, Losslessness, and Coding-Native Compatibility.

• The Write Operation: Every action, observation (board state), and plan is appended to a simple logs.txt file. No compression, no filtering.
• The Read Operation: Instead of a "black box" retrieval system (like a Vector DB), the agent is given tools like grep and a Python interpreter.
• The Intuition: A coding agent can write a script to "grep" for every time a score increased, or write a Python function to compare the current board with one from 500 steps ago. This turns a massive text file into a searchable, structured database.

Figure 1: PRO-LONG integrates a structured log with a tool space that enables programmatic search (Grep, Python, Bash).

Experimental Results: SOTA Efficiency

The team tested PRO-LONG on ARC-AGI-3, a benchmark of various interactive logic games. The performance gains were staggering:

1. Massive Gains over Base Models: Simply adding the PRO-LONG log to a standard coding agent improved scores by an average of 18 percentage points.
2. Token Efficiency: Compared to specialized harnesses like WorldModeler, PRO-LONG achieved similar or better results while using 4.2x to 5.8x fewer tokens.
3. Peak Performance: Using the latest frontier models (like Fable 5), PRO-LONG reached a near-perfect 97.4% best@2 score.

Figure 2: Performance vs. Billed Tokens. Note how PRO-LONG (purple/orange lines) achieves high scores with significantly lower token consumption than prior harnesses.

Why it Works: A Shift in Strategy

Ablation studies revealed that when agents have access to the log, their behavior changes. Instead of spending tokens on "writing notes" to themselves, they spend those tokens on programmatic analysis (see Table 3). For example, in a complex maze game, a PRO-LONG agent might write a script to replay its actions and build a custom Breadth-First Search (BFS) algorithm to solve the level—something a "static" agent could never do.

Critical Analysis & Conclusion

The brilliance of PRO-LONG lies in its minimalism. It admits that LLMs are bad at managing their own long-term memory but excellent at writing code to manage data. By offloading the "thinking" about memory to Python scripts, the agent maintains a clear "mental" focus on the current task.

Limitations:

• The method relies heavily on the quality of the base model's coding abilities.
• While cost-effective compared to other harnesses, $1,750 for a full benchmark run is still significant, highlighting the inherent expense of long-horizon AI evaluation.

The Takeaway: As LLMs move from simple chatbots into long-running "agents," the best way to handle memory isn't to build better "brains" for memory, but better "libraries" and "search engines" that the brain can control via code.