From Static Models to Evolving Systems: The Rise of Self-Improving Agents

TL;DR

The "Era of Experience" marks a pivot from training AI on static datasets to deploying systems that learn from doing. This survey by Tsinghua University and Horizon Research defines the Harness— the control layer around the model—as the "experience infrastructure" that allows agents to evolve their skills, memory, and even their own neural weights in real-time. It moves beyond simple "tool-use" to a vision of autonomous Meta-Evolution.

The Problem: The "Memory Leak" of Knowledge

Traditional LLM agents are often "task-bounded." Once an episode ends, the valuable trial-and-error data evaporates. Even in RAG-based systems, the agent is a passive consumer of information. The authors argue that current AI suffers from a Timescale Asymmetry: we can't retrain a 70B model every hour, yet the agent needs to adapt to new user preferences or environment bugs instantly.

The solution? Don't just update the model; update the Harness.

Methodology: The Deployed System Tuple

The paper formalizes the agent as a dynamic object: ${\mathcal{A}}_t=⟨M_{het{a}_t},H_t,U_t,E_t⟩$

• $M$ (Model): The "brain" (slow to change).
• $H$ (Harness): The mutable infrastructure defining skills, memory, and permissions (fast to change).
• $U/E$ (User/Environment): The dual interfaces providing feedback.

The Two Paths of Improvement

1. External Path (The "Library" Approach): Interaction traces are distilled into SKILL.md files or indexed memory clusters. The model doesn't change, but what it sees and what it knows how to do improves.
2. Parameter Path (The "Internalization" Approach): Stable lessons from the harness are consolidated into the model's weights ($heta$) via Reinforcement Learning (RL) and On-Policy Distillation (OPD).

Figure 1: The architecture of self-evolution, moving from raw traces to durable capabilities.

The Core: Skills and Memory as Software

A standout insight of this survey is the standardization of Skills. The industry is converging on a SKILL.md schema: a package containing instructions, reference docs, and executable assets. Effectively, self-improvement becomes a software engineering problem: Loop Engineering.

Figure 2: The evolution from Task Loops (Gen 1) to Persistent Runtimes (Gen 2) to Self-Evolving Systems (Gen 3).

Experiments & Results: The Reality Check

While the potential is vast, the survey highlights a sobering reality:

• Skill Gains: While curated skills boost performance by ~16%, self-generated skills can lead to "Negative Transfer," where the agent actually gets worse at old tasks while trying to learn new ones.
• Safety Gaps: A self-improving agent is a "moving attack surface." A poisoned memory can "steer" future reasoning without needing a jailbreak prompt.

Table 1: Comparison of how modern agents evolve across different surfaces.

Critical Insight: The Meta-Evolution Frontier

The most advanced concept discussed is the Meta-Evolving Agent. In this regime, the agent doesn't just fix a bug in its code; it improves the process it uses to find bugs. The "Hyperagent" (like DGM-Hyperagents) can rewrite its own meta_agent.py, searching for better ways to learn.

Challenges & Future Work

The authors identify three massive roadblocks:

1. Credit Assignment: If an agent fails a 100-step coding task, which of the 10 recalled memories was wrong?
2. Longitudinal Evaluation: We need benchmarks like SIP-Bench that measure retention over months, not just accuracy on a one-off test.
3. Stability: How do we prevent an agent from "collapsing" into its own self-generated biases?

Conclusion

This paper effectively serves as a manifesto for the next decade of AI. We are moving away from "Foundation Models" as the end-goal and toward Foundation Agnostic Runtimes—systems where the intelligence lies as much in the interaction loop as it does in the transformer weights.