Can runtime contracts for agent safety avoid repeating mistakes across long multi-step tasks?

Why training-time safety alone fails for long multi-step tasks

The dominant approach to AI safety has been to train models to be helpful and harmless, but this is structurally insufficient for agents that execute code, send messages, or modify databases. A 2026 paper [3] argues that safety must be a runtime contract enforced by the harness, not just a property instilled during training. This is backed by a survey of 52 documented AI-agent and LLM safety incidents, which found that many failures occur because there is no runtime enforcement to catch mistakes as they happen. Additionally, an audit of all 28,560 papers accepted at top AI conferences (NeurIPS, ICML, ICLR) from 2023-2025 found an 8-12x imbalance between training-time and deployment-time safety research, meaning the field has been neglecting the very mechanisms needed to prevent repeated errors in real-world tasks.

The problem is especially acute in long multi-step tasks because a single mistake early on can cascade into many subsequent errors. Training-time methods like RLHF (Reinforcement Learning from Human Feedback) or DPO (Direct Preference Optimization) cannot anticipate every possible action sequence. As [3] notes, the right unit of safety is the 'trajectory-with-checkable-evidence,' not the model itself. This means we need to monitor and verify each step of the agent's execution, not just rely on the model's learned behavior.

How runtime contracts actually prevent repeated mistakes

Runtime contracts work by adding two complementary layers: preventive and evidential. The preventive face blocks dangerous actions before they happen, using sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence like test runs, log captures, file diffs, and citation grounding. This dual approach is directly supported by [3]'s formalization of an Agent Trajectory Schema and Evidence Chain, which provides a compositional way to verify each step.

Other research reinforces this idea from different angles. For instance, [2] and [4] present a runtime verification framework for multi-agent systems that controls events during execution without requiring each agent to be specifically programmed to recognize them. They demonstrate this in a hospital bed allocation scenario, where the monitor ensures the dialogue flow stays on track, preventing topic drift that could impair reasoning. This is a concrete example of how runtime monitoring can catch and correct deviations in real time, which is exactly what's needed to avoid repeating mistakes in long tasks.

Similarly, [5] introduces a runtime environment for contract automata that guarantees the implementation adheres to its contract, showing that formal contracts can be enforced at runtime in service coordination. And [6] proposes an adaptive runtime verification method for multi-agent self-adaptive systems, converting environmental factors into probabilistic forms to enable self-regulation. Together, these studies converge on the same conclusion: runtime contracts provide a robust mechanism to catch and correct errors as they occur, which is essential for long multi-step tasks.

The catch: contracts must be maintained, not just asserted

While runtime contracts are powerful, they are not a one-time fix. A 2026 paper [1] highlights the issue of 'constraint drift' in LLM-based multi-agent systems, where the original safety constraints can become outdated or ignored as the task evolves. The paper argues that safe multi-agent behavior must be maintained, not merely asserted, and proposes an illustrative runtime contract that adapts to the current task context. This means that for long tasks, the contract itself needs to be updated to reflect new information or changing requirements, otherwise it may fail to catch new types of mistakes.

This is a crucial caveat: runtime contracts are necessary but not sufficient. They must be designed to be dynamic and responsive. The evidence from [1] suggests that a static contract might work for short tasks, but for long multi-step tasks, the contract needs to be re-evaluated and adjusted. This is an active area of research, and the papers here provide a foundation but also highlight the need for ongoing work to make contracts truly adaptive.

About These Sources

This answer is built on 6 studies (4 peer-reviewed, 2 preprints) — published from 2021 to 2026, 2 from 2024 or later, 1 in Q1–Q2 journals — selected as the most relevant from 6 studies that passed quality screening, drawn from 35 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions