How does EvoBelief use falsifiable world models for self-improving LLM agents?

The failure is wrong beliefs, not weak skills

EvoBelief targets a gap that procedure-focused self-improvement leaves open. Agents that learn from past trajectories, reflection, memories, or distilled skills get better at how to act, but they still act on prior beliefs about how the environment responds. When those beliefs are incomplete, incorrect, or outdated, a procedurally capable agent can still fail.

The paper's motivating case is a synthesis task: the agent knows how to heat, purify, and analyze samples, but its prior says compound X is stable up to 120 °C while X actually decomposes at 100 °C. A task-specific patch such as "use a lower temperature" fixes one episode without capturing the underlying property. Testing how X's stability depends on temperature and keeping the resulting constraint would instead inform synthesis, shipping, storage, and analysis.

This is why the paper calls its contribution belief evolution: a persistent model of environment dynamics that can support many tasks, and whose errors can also propagate across them. The agent's policy answers how to act toward a goal; the world model answers how the environment responds to actions.

Hypotheses are written as testable claims

EvoBelief keeps two externalized states while the model parameters stay fixed. One is a per-episode state estimate recording observed entities, relations, quantities, and task progress, reset at each episode boundary. The other is a persistent world belief that survives episode resets and changes only as new evidence arrives. The frozen language model supplies a general prior; the persistent store holds environment-specific knowledge learned during deployment.

The basic unit of that store is a falsifiable hypothesis stated as: when condition C holds, taking action or procedure A should produce observable consequence Y. Because C can be established, A executed, and Y checked, the claim is directly testable. Each hypothesis also carries the scope in which it is supported, a recipe for testing it, and an evidence record. A may be a single action, a multi-step procedure, or a measurement intervention; Y may be a state change or a measured quantity.

Hypotheses can mix fixed entities with abstract object roles. When used, roles bind to concrete objects in the current context, and broader scope must be earned from evidence across varied bindings rather than assumed. A rule that heating above about 80 °C decomposes the sample starts bound to compound X and covers other compounds only after evidence from them.

Surprises and decision-relevant unknowns trigger hypotheses

Hypothesis formation starts from an empty belief store and the agent's own interaction. After acting, EvoBelief compares the realized outcome with the prediction made before acting. A grounded prediction mismatch suggests the current model is wrong; a decision-critical unknown, one whose resolution would change what the agent should do next, suggests the model is incomplete.

One surprising outcome can have several explanations, so EvoBelief may keep rival hypotheses until later evidence distinguishes them. In the compound X case, decomposition could depend on temperature, heating duration, or the vessel. Crucially, forming a hypothesis is not evidence for it: an unvalidated hypothesis cannot justify a task-directed action as established knowledge, though it may motivate an experiment.

Evidence arrives passively or through admitted experiments

Evidence reaches a hypothesis by two routes. Passive evidence comes from an ordinary task transition that happens to test an existing hypothesis, such as a later task heating compound X and recording whether it remains intact. Active evidence comes from an experiment run specifically to resolve an uncertainty or separate rival hypotheses. Both update the same evidence record.

Active experiments use live environment actions and therefore consume the task's interaction budget. EvoBelief admits one only when its outcome can change a decision the agent must make, no validated rule already resolves the question, and the required setup is executable. Support and falsification criteria are fixed before the experiment runs, and the realized outcome settles the test as supported, counterexampled, or inconclusive. An inconclusive test counts as neither support nor counterevidence, and unresolved hypotheses stay pending for later testing.

A rule earns authority only from independent evidence

Authority is the paper's term for permission for a learned claim to guide a later prediction or plan. A newly formed hypothesis has none. Commitment to a validated rule requires thresholds on three distinct counts in its supports: separate episodes, separate bindings, and independent tests. The thresholds are set before any benchmark run and never tuned on benchmark results.

The event that first formed a hypothesis can never by itself grant reusable authority, and each generalized role must vary across supporting bindings. One batch of compound X decomposing at 90 °C is not enough; the claim must hold again in a later episode on a different batch before it may guide heating decisions.

Counterevidence can remove authority. EvoBelief first narrows the rule to the scope still supported by surviving evidence and recomputes the commitment condition. If the surviving evidence no longer satisfies it, the rule is demoted: it loses reusable authority but keeps its evidence, so later support can restore it. A rule is retired only when repeated counterexamples leave it with no supporting evidence. Because the record is preserved, every status change traces back to the observations that caused it.

Validated rules change predictions, not the actor's prompt

At each decision, the agent forms candidate actions without access to the world belief. For each candidate, the frozen predictor supplies an anchor prediction of what would happen without learned rules. Only a validated rule whose condition holds and whose scope covers the current binding may change that anchor. Retrieval alone has no effect.

Rule-supported consequences affect behavior in two ways. They can revise a candidate's expected outcome, changing its ranking or ruling it out when a required precondition is violated; once the decomposition rule for compound X is validated, heating X to 100 °C falls below a lower-temperature route. Compatible rules can also be chained into a short plan when one rule's consequence establishes another's condition, and the agent may execute that plan's first step even if it was not among the candidates. A validated quantitative relation can set parameter values needed for a target consequence.

If no validated rule applies, EvoBelief falls back to the same choice it would have made without the world model. The implementation detail matters for interpreting the results: the rule lifecycle and its authority checks are enforced outside the actor, and the actor never receives the validated rule text. On ALFWorld, MiniGrid, and ScienceWorld, rule-supported consequences are computed directly from the structured representation; DiscoveryWorld is the only benchmark where the frozen predictor engine may also make a rule-conditioned model call.

Gains appear across static, drifting, and discovery settings

The evaluation spans four benchmarks covering static dynamics, hidden drift, and scientific discovery. On ALFWorld without actor-side transition memory, EvoBelief raises success over the matched Frozen World Model by 23.9 percentage points with Claude Haiku 4.5 and 31.3 points with GPT-5.4. The absolute rates are 66.42% versus 42.54% for Haiku and 76.12% versus 44.78% for GPT-5.4.

The memory setting changes the comparison. With actor-side transition memory, EvoBelief reaches 68.66% on ALFWorld with Haiku and 76.87% with GPT-5.4, against 57.46% and 55.22% for Frozen WM. The paper's own reading is that where episodic memory already supplies what the prior lacks, rules and memory act partly as substitutes, which is why the no-memory setting isolates the larger gain.

On MiniGrid, where the agent must identify the key that actually opens a locked door and reach a target box within 24 steps, EvoBelief revises stale rules and recovers after unannounced dynamics changes. In DiscoveryWorld's Reactor Lab, where each reactor activates only within ±2 Hz of a resonance governed by a hidden parametric law, EvoBelief completes 83.3% of Normal-tier and 58.3% of Challenge-tier instances, versus 0% for each evaluated baseline. Those discovery runs used twelve fresh task instances per tier on GPT-5.4, four methods on the same instances, one attempt each, and no transition memory.

The advantage depends on what memory already provides

The paper's central distinction is between improving how an agent acts and correcting what it believes about the environment. EvoBelief's gains are largest where the frozen prior is wrong and nothing else supplies the missing environment knowledge, as in the ALFWorld no-memory comparison. Where transition memory already carries that information, the margin narrows, and the two mechanisms partly substitute for each other.

A second distinction is between learning a rule and being allowed to use it. The commitment thresholds on episodes, bindings, and tests, plus demotion and retirement under counterevidence, are what keep accumulated knowledge from becoming a liability when dynamics shift. That is also why a validated rule can help tasks the agent has never seen: it describes the environment rather than a single task.

The practical priority follows from these two conditions. EvoBelief is most worth the added machinery when failures trace to environment dynamics that are hidden, drifting, or expressible as quantitative laws, and when the agent lacks another channel that already records those dynamics. When episodic memory already covers the gap, expect the rule layer to add less, and judge it by whether it can withdraw authority from rules that stop holding.

Sources used in this answer