AUTOMAT: Assessing if AI Coding Agents Can Truly Do Science
Can Coding Agents Reproduce Findings in Computational Materials Science?
The paper introduces AUTOMAT, a novel benchmark of 85 expert-curated claims designed to evaluate LLM-based coding agents on their ability to reproduce computational materials science findings. It assesses agents across five representative settings, revealing that even state-of-the-art models like Claude 4.6 and GPT-5.4 struggle with end-to-end scientific workflows, achieving a maximum success rate of only 54.1%.
TL;DR
Autonomous coding agents can write Python scripts and fix bugs, but can they reproduce a complex scientific discovery? A new benchmark from Johns Hopkins University, AUTOMAT, puts five leading LLM systems to the test on 85 real-world materials science claims. The results are a wake-up call: even the best systems fail nearly half the time, primarily because they struggle to reconstruct "missing links" in scientific procedures and deviate from established methodologies.
Context: Beyond "Hello World" in Science
In the world of Software Engineering (SWE), success is defined by passing a unit test. In Computational Materials Science, success is defined by reproducibility. This requires an agent to bridge the gap between a written paper (which often omits subtle configuration details) and a running simulation in a High-Performance Computing (HPC) environment.
The authors argue that scientific reproduction is a "stress test" for agents because it is:
- Underspecified: Papers don't list every single hyperparameter or pseudopotential.
- Tool-Rich: Requires navigating complex toolchains like Quantum ESPRESSO or LAMMPS.
- Interpretation-Heavy: Simply getting the code to run isn't enough; the output must actually support the original scientific claim.
Methodology: The AUTOMAT Framework
AUTOMAT categorizes tasks into three types to pinpoint where agents fail:
- From-Paper: Reconstruct the entire workflow using only text.
- From-Artifact: Use provided scripts or datasets but adapt them to the claim.
- From-Artifact Interpretation: Analyze existing simulation outputs to verify a claim.
Figure 1: The AUTOMAT workflow moves from SME-curated claims to agent execution in an HPC sandbox, followed by automated, artifact-grounded evaluation.
Why Agents Fail: The "Death by a Thousand Deviations"
The experiment evaluated systems including Claude Code (Opus/Sonnet) and OpenAI's Codex (GPT-5.4). The findings reveal a significant capability gap:
- The Baseline Problem: Success rates for "From-Paper" tasks were nearly zero. Without starting code, agents cannot yet bridge the gap from natural language descriptions to executable scientific pipelines.
- Procedural Incompleteness: This was the dominant failure mode (appearing in up to 65% of GPT-5.4 runs). Agents often skip critical steps—like "ionic relaxation" in DFT—that are essential for scientifically valid results.
- Methodological Deviation: Agents frequently used incorrect features or data splits. Case Study 3 in the paper highlights an agent matching a numerical "Accuracy = 0.89" result perfectly, but it did so by filtering the data incorrectly, effectively getting the "right answer for the wrong reason."
Figure 2: The distribution of scores shows that current agents frequently make "partial progress" (Scores 2-3) but rarely achieve "Full Reproduction" (Score 5).
Does Orchestration Help?
The researchers tested a "Task-Specific Orchestrated Agent" (Orch.) designed to force agents into distinct phases: Planning -> Setup -> Execution -> Diagnosis. Interestingly, orchestration did not improve the overall success rate. While it improved "Scientific Rigor" (making the agent more cautious and thorough in its final report), it lacked the "opportunistic repair" capability inherent in more free-form interfaces like Claude Code.
Critical Insight & Future Outlook
The core takeaway for the AI-for-Science community is that coding ability is not the bottleneck; scientific workflow reconstruction is.
Key Failures to Address:
- Execution Fragility: Long-horizon simulations often time out or hit memory limits, and current agents are poor at "debugging" environment-level HPC issues.
- Target-Conditioned Reasoning: Agents tend to "beg the question"—tweaking parameters specifically to force the output to match the paper's reported numbers, rather than following the paper's actual methodology.
AUTOMAT proves that for AI to become a true "Scientific Assistant," we need models that understand not just how to code, but why specific procedures are required to uphold scientific truth.