[Research Deep Dive] The AGENTS.md Paradox: Why More Context Is Making Coding Agents Worse

TL;DR

A new study from researchers at ETH Zurich and Trace AI reveals a startling counter-intuition: the context files (like AGENTS.md) designed to help AI coding agents are actually tanking their performance. In a rigorous evaluation using the new AGENT-BENCH, LLM-generated context files increased inference costs by 20-23% while frequently reducing success rates.

The Motivation: A Growing Standard Without a Proof

The software industry has rapidly converged on a new standard: the "Agent-specific README." Whether it's AGENTS.md for general agents or CLAUDE.md for Anthropic's ecosystem, developers are told that providing a repository overview, style guides, and tooling instructions is the key to agentic productivity.

Over 60,000 GitHub repositories have already adopted this practice. Yet, the authors of this paper noticed a gap: Does this actually work? Or are we just feeding agents redundant noise that confuses their reasoning?

Methodology: Introducing AGENT-BENCH

Evaluating this required moving beyond the classic SWE-bench. The authors created AGENT-BENCH, a collection of 138 unique Python tasks from 12 "niche" repositories that already featured developer-committed context files.

The experimental setup was robust, testing:

1. Claude Code (with Sonnet-4.5)
2. Codex (with GPT-5.2 and GPT-5.1 Mini)
3. Qwen Code (with Qwen3-30B-Coder)

The authors compared settings with no context, LLM-generated context, and human-written context files.

Key Findings: The Cost of "Helpfulness"

1. Performance vs. Cost

Across the board, LLM-generated context files were a net negative. On SWE-BENCH LITE, success rates dropped by 0.5%; on AGENT-BENCH, they dropped by 2%. More critically, the number of steps and inference cost spiked. For example, using GPT-5.2 on AGENT-BENCH with an LLM context file increased the cost from $0.38to$0.57 per task—a 50% increase for worse results.

2. The Exploration Trap

Why does more info lead to worse results? The authors performed a "trace analysis" to see what the agents were doing. Context files act as a set of "distractions." When an agent reads an overview of the whole repo, it feels compelled to "explore" more—searching files (grep), reading docs, and running more tests—even if it doesn't need to.

Resolution rates showing that LLM-generated context (bright green) often underperforms the "None" setting (gray).

3. Redundancy is the Killer

The study found that LLM-generated context files are often just a "rehash" of existing documentation. When the authors manually removed all other .md files and docs, the context files finally became helpful. This suggests that in a well-documented repo, an AGENTS.md is essentially redundant noise.

Deep Insight: "Following Instructions" isn't the same as "Solving the Task"

One of the most profound observations in the paper is that agents do follow the instructions in context files. If the file mentions a specific tool like uv or pytest, the agent uses it significantly more.

However, this adherence comes at a "cognitive cost." Analysis of Reasoning Tokens (for O1-style models) showed a 10-22% increase in thinking time when context files were present. The agent is working harder to satisfy the "requirements" of the context file, which actually makes the core task of fixing the bug harder to reach.

Agents spend significantly more "thinking budget" when a context file is present.

Critical Analysis & Takeaways

The paper delivers a sobering message to the "Agentic Workflow" community: Stop over-prompting.

• For Developers: If you write an AGENTS.md, keep it focused. Don't describe the file structure; the agent can see that. Only include "hidden" knowledge, like specific environment variables or unique CLI tools.
• For Agent Architects: The default /init commands in modern agents are currently generating "bloatware" that increases customer costs while decreasing reliability. We need more principled, sparse context generation.

Limitations: The study focuses on Python. In more obscure languages (like Mojo or OCaml), where the LLM has less internal knowledge, context files might still be "the hero." But for the majority of mainstream dev work, we might be better off letting the agent figure it out on its own.

Conclusion

The industry trend of "Agent Context Engineering" is at a crossroads. This research suggests that our current approach of providing comprehensive overviews is an "Efficiency Tax." Future work must focus on dynamic context—providing only the information the agent needs, exactly when it needs it, rather than a giant, distracting manual at the start of every session.