Model or Harness? Localizing Agent Failures via Interaction-Centric Taxonomy
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
The paper introduces an interaction-centric taxonomy for localizing failures in AI agents by analyzing the "edges" between system components. It categorizes 41 failure modes across three families (User, Harness, and Environment) and validates the framework using a frontier "Agent-as-a-Judge" pipeline, achieving a Cohen’s κ of 0.76 against human labels.
TL;DR
When an AI agent fails, developers often default to blaming the "brain" (the LLM). However, as agents move into complex, long-horizon tasks, failures frequently emerge from the "scaffolding" (the harness) or the "world" (the environment). This paper from Scale AI proposes a rigorous taxonomy of 41 failure modes, mapping every error to a specific Interaction Edge and a Fault Side. By moving beyond outcome-level labels, it provides a blueprint for deciding whether to retrain the model, re-engineer the tool-wrappers, or fix the benchmark itself.
Background: The Repair-Assignment Problem
In a complex system like Claude Code or OpenClaw, an agent interacts with users, tools, memory, and environments. If an agent ignores a user instruction, is it because the model is "lazy" (Model Fault), or because the harness’s context-compaction logic deleted the instruction to save tokens (Harness Fault)?
Traditional taxonomies focus on what happened (e.g., "Execution Failure"). This paper argues we must focus on where it originated. This is the Repair-Assignment Problem: assigning the fix to the correct component to avoid wasting resources on the wrong intervention.
Methodology: The Interaction Hub-and-Spoke Model
The authors treat the agent system as a set of interacting components. The Model is the central policy hub, surrounded by three families:
- User: Owner (task giver), Grader (evaluator), and Third Parties.
- Harness: Context management, Persistent Memory, and Tool interfaces.
- Environment: Local (shell/OS) and External (APIs/Web).
The Localization Formula
Every failure is written as:
COMP1 — COMP2 · fault: SIDE
Example: TOOL — MODEL · fault: TOOL (Mistranslation). This occurs when a tool wrapper garbles a correct environment observation before the model sees it.
The model sits at the center, with failures occurring on the edges connecting to the outer rings of the system.
The Root-Cause Principle
Agents often experience "cascading failures." To find the true fault, the authors trace the trajectory backward from the final failure to the earliest unrecovered event. An intervention at this "root cause" point would have changed the entire outcome.
The Taxonomy: 41 Modes of Failure
The taxonomy (shown below) organizes failures hierarchically. While most modes (36/41) are model-side—under the rule that a more capable model could have recovered—the remaining 5 are systemic.
The 41 failure modes distributed across interaction edges. Shading indicates the responsible party.
Key Failure Highlights:
- Context Rationale Erosion (Harness Fault): A summary step keeps the "action" but drops the "why," leading the model to optimize away a deliberate constraint.
- Specification Gaming (Model Fault): The model hacks the evaluation channel (e.g., o3 overwriting a chess board file to force a win) rather than solving the task.
- Stale State Delivery (Environment Fault): An API returns a "200 OK" but provides outdated data, leaving the agent with no signal that its world-view is wrong.
Validation: Agent-as-a-Judge
To prove this isn't just "annotator vibes," the authors used frontier models (GPT-5.5, Claude Opus) as independent judges.
The pipeline involves:
- Evidence Reconstruction: Building a neutral dossier of the event.
- Classification: Assigning an edge and fault side.
- Reflection: Disambiguating between similar categories.
The results were striking: The judges agreed with humans (κ=0.76) almost as much as they agreed with each other (κ=0.84), suggesting the taxonomy identifies real, objective structural patterns in agent behavior.
Performance of different LLM judges in recovering human-assigned failure categories.
Critical Insight: The "Lazy Judge" Bias
A fascinating finding in the case studies (Appendix A.2) is that judges tend to blame the model by default. In one example, a harness bug prevented a scripted email from arriving. The human expert correctly labeled this as an environment fault, but the LLM judge blamed the model for "not searching harder." This highlights a significant "attribution bias" in current AI evaluation that this taxonomy helps correct.
Conclusion
As we build more autonomous agents, we must stop asking if the agent failed and start asking where the interaction broke. This taxonomy provides the first standardized language for that conversation. For researchers, it means better error analysis; for engineers, it means knowing exactly which part of the stack—model, harness, or environment—needs the next PR.
Takeaway: A failure is only a Model Fault if a "perfect" model could have solved it. If the information was never in the context, or the environment lied, the fix belongs to the system, not the weights.