[ArXiv 2026] Theory of Code Space: Do Code Agents Actually "Get" Software Architecture?

TL;DR

While AI code agents are great at writing snippets, they often fail at "big picture" software engineering. The Theory of Code Space (TOCS) benchmark marks a shift from measuring what an agent can write to how it thinks about a codebase. By forcing models like GPT-5.3 and Claude to periodically "dump" their architectural beliefs into JSON, the researchers discovered that most models struggle to maintain a stable mental map, with some even suffering from "belief collapse" as they explore.

The Missing Link: Why Good Coders are Bad Architects

Most LLM benchmarks are static: give a prompt, get a code block. However, real-world engineering is a process of active exploration. A developer doesn't see the whole repository at once; they open files, follow imports, and build a mental model of how Module A talks to Module B.

The authors argue that current agents fail because their "mental maps" are fragile. They suffer from two major issues:

1. The Active-Passive Gap: Models often perform worse when they have to find the information themselves rather than having it served in a single context window.
2. Belief Instability: Models "forget" the architecture they just discovered as they move to new files.

Methodology: Probing the Latent State

TOCS uses procedurally generated Python codebases where the architecture is intentionally obscured (e.g., using dynamic registries instead of simple imports).

The Exploration Loop:

• Budgeted Actions: The agent has 20 actions (OPEN, SEARCH, LIST).
• Periodic Probes: Every 3 actions, the agent must externalize its "Belief State" (JSON).
• Hidden Invariants: The agent is scored on its ability to find "planted" constraints, like "Module X must only be accessed through Interface Z."

(Above: The F1 performance over time shows how different agents accumulate—or lose—knowledge during exploration.)

Key Insights: The Surprise in the Data

1. Active Exploration is a Skill

The paper found a fascinating Active-Passive Gap. For GPT-5.3-Codex, active exploration was actually better than seeing the whole codebase at once (APG = -0.22). The sequential nature of exploration likely acted as a focus mechanism. Conversely, Gemini 2.5 Flash was overwhelmed by active decision-making (APG = +0.23), performing much better when the full code was provided upfront.

2. Catastrophic Belief Collapse

Perhaps the most shocking finding was how larger models are not always better at "remembering." While the smaller Gemini 2.5 Flash kept a perfectly stable map, its larger sibling Gemini 2.5 Pro showed "belief collapse"—it would build a great map and then suddenly "forget" 12 correct edges in a single step (see Figure 2 in the paper).

(Above: Growth vs. Collapse. Note how some models lose information as they continue to explore.)

3. The Power of the "Architectural Scratchpad"

When agents were allowed to see their previous JSON probes (functioning as a "scratchpad"), GPT-5.3-Codex saw a massive 14-point F1 boost. This suggests that explicitly maintaining an architectural map in the context window is a key "self-scaffolding" capability that separates top-tier agents from the rest.

Detailed Results & Benchmarking

The frontier models GPT-5.3-Codex and Claude Sonnet 4.6 dominated the leaderboard, significantly outperforming rule-based "Config-Aware" baselines. They were particularly adept at discovering complex "Data Flow" and "Registry Wiring" edges that simple static analysis tools miss.

Critical Analysis & Future Outlook

The TOCS benchmark highlights a critical bottleneck: Belief Externalization. A model might "know" the architecture but fail to format it correctly, or it might be "gullible," changing its correct belief based on a single distractor file.

Takeaways for AI Researchers:

• Don't just increase context: More context doesn't fix poor architectural reasoning (the APG proves this).
• State Management is Key: Future agents need a "long-term memory" or a persistent graph structure to store architectural insights.
• Synthesizing Invariants: Models still struggle to discover why a design choice was made, focusing instead on how files connect.

TOCS opens a new front in AI evaluation—moving past "code completion" and toward true "software engineering intelligence."