Can a recurrent harness enable online continual learning for language model agents?

Forward-only harness state goes stale

A language model agent that works through a long stream of tasks can improve without changing its weights by keeping a persistent runtime scaffold around the model. The paper calls this scaffold the harness, and it holds memory, reflections and prompts that the agent reads before acting and writes after finishing an instance. Existing harnesses mostly evolve this state in one direction: extract something useful from a finished trajectory, store it, and retrieve it later.

The paper identifies the cost of that one-way flow. Once a memory or reflection is written, it is rarely revised according to how well it actually worked when later retrieved. Some systems add or delete entries to keep the store coherent, and others optimize the harness with extra training or validation data. Under a strictly online streaming protocol, where each instance arrives once and no replay or held-out set is available, neither route fixes an outdated or overconfident entry. Under distribution shift, such entries can keep misguiding decisions.

Revision asks what should have been captured

ReHarness closes the loop by adding a backward step to the familiar forward read and write. After each instance, a revision agent revisits the reflections that were retrieved for that instance and asks what information from past experience could have been captured differently to better inform the current case. The completed trajectory serves as feedback on whether the retrieved guidance helped.

The key constraint is grounding. Any new content in a revised reflection must remain supported by the source trajectory that originally produced it; the later trajectory may only refine the matching conditions and expose omissions that the source already supports. This is what separates revision from simply copying the newest outcome into memory. The revised state is then carried forward to the next instance, so feedback propagates backward through the textual harness state without replaying instances or updating model parameters.

Grounding holds above 95 percent of claims

The paper first checks whether revision respects that constraint. Each revised reflection is decomposed into factual claims and classified as grounded if entailed by its source trajectory, leaked if entailed only by the later trajectory, or unsupported if entailed by neither. Across all three backbones, over 95% of claims remain source-grounded, leakage is 0.8–2.1% and unsupported claims are 0.5–2.6%. Revision changes these rates only marginally, with grounding dropping by at most 0.3% per revision.

That matters because a revision mechanism could easily become a channel for smuggling the current answer into past experience, which would inflate later scores without teaching anything reusable. The measured rates indicate the backbone models largely follow the grounding instruction and use the later trajectory as feedback rather than as a content source.

Counterfactual replay shows revision has content

A grounding check alone cannot show that a revision is useful. The paper therefore runs a counterfactual replay that bends the streaming protocol: after revising the harness state using the completed trajectory, it reapplies that revised state to the same instance instead of the next one. This asks what past information should have been captured differently to help that instance.

On CL-Bench with Claude, average normalized gain rises from 0.144 with no revision to 0.217 for ReHarness and 0.335 under counterfactual replay, and counterfactual replay improves on ReHarness across all tasks. The authors read this as evidence that the backward mechanism produces meaningful revisions rather than cosmetic edits. The replay itself violates the online protocol, so it functions as a diagnostic; ReHarness applies the revised state only to the next streamed instance and still beats the no-revision baseline on every task, with the largest gains in database exploration, sales prediction and blind spectrum monitoring.

Two cases show overgeneralization and staleness corrected

The qualitative examples make the mechanism concrete. In blind spectrum monitoring, the memory agent overgeneralized from five observed transmitter peaks and advised inventing plausible channels to fill silent gaps. When that reflection was retrieved, the decision agent added unobserved transmitters and its reward fell to 0.219. Using that outcome as feedback, the revision agent marked the hypothesis as speculative and shifted guidance to prioritizing confirmed peaks; the next retrieval produced a reward of 0.367.

In database exploration, an early reflection consolidated the belief that the graded price came from a column named prc. After a schema shift introduced prc_v2, the retrieved old rule led the agent to check only prc and score 0. When the failure was fed back, the revision replaced the rule with an instruction not to assume which column is graded and to enumerate all price-like columns. The next retrieval checked both columns and solved the task in two queries for a reward of 0.867. These are single illustrative trajectories, not aggregate estimates, but they show the two failure modes the design targets: overgeneralization early in a stream and outdated strategy after a shift.

Gains hold across backbones, with exceptions

The main comparison reports normalized gain per task for Claude Opus 4.8, GPT-5.5 and Gemini 3.7 Flash against memory methods (Mem0, RBank), reflection and prompt methods (DoT, ACE, GEPA-Online) and two raw-context baselines that keep full or top-k raw trajectories. ReHarness records the highest average in each backbone block: 0.305 with Claude, 0.358 with GPT-5.5 and 0.408 with Gemini 3.7 Flash.

The per-task picture is not uniform, and the paper says so. With GPT-5.5, ReHarness reaches 0.816 on cohort but 0.407 on codebase, below Full Raw at 0.471. With Gemini 3.7 Flash it reaches 0.930 on cohort, essentially level with Full Raw at 0.945, and 0.226 on codebase against 0.182 for Full Raw. The authors also report that ReHarness underperforms on exploitable poker, where downstream rewards provide noisy signals for backward revision. The stability–plasticity decomposition reported in the paper finds the highest average gains in both components, which the authors interpret as robust transfer across shifts plus effective adaptation afterward.

Token savings outweigh extra output

Revision is not free: ReHarness makes one reflection call for forward write and k revision calls for backward revision, on top of decision calls shared by all methods. The paper measures the net effect over one rollout across all CL-Bench tasks with Claude. ReHarness consumes about 37.19M input and 4.38M output tokens, roughly 41.57M total, against about 52.66M for Top-K Raw and 142.12M for Full Raw, or 1.3× and 3.4× as many total tokens as ReHarness.

The saving comes from input tokens: compact reflections shrink the repeated decision-time context that raw trajectories inflate on long, multi-round instances. With prompt caching and Claude API pricing, the reported costs are about $85 for Top-K Raw, $254 for ReHarness and $314 for Full Raw. ReHarness therefore costs more than Top-K Raw but achieves a higher normalized gain, 0.217 versus 0.165, and costs less than Full Raw while also gaining more, 0.217 versus 0.191. The paper notes a remaining overhead: raw trajectories are still stored for future revision.

Sources used in this answer