What NatureBench actually measures — and what it leaves out
NatureBench is a benchmark of 90 tasks distilled from real Nature-family papers, designed to test whether AI coding agents can reproduce the published results of scientific papers [1]. It's a rigorous test of code-level reproduction: each task is packaged into a containerized environment, and agents are evaluated under a strict protocol that disables web search [1]. That's a strength for reproducibility, but it also means the benchmark measures how well an agent can follow a recipe, not how well it can invent a new one.
The key limitation is that NatureBench's tasks are already distilled from published papers — the scientific question, the method, and the expected outcome are all predefined. Real discovery, as described in a 2025 commentary on agentic science, involves reasoning, planning, and interacting with digital and physical environments in open-ended ways [2]. NatureBench strips away that open-endedness, so it can't reveal whether an agent can formulate a novel hypothesis, design an experiment from scratch, or recognize when a result is surprising. It only shows whether the agent can execute a known method correctly.
The gap between best-case and typical-case performance
The headline number from NatureBench is sobering: across ten frontier agent configurations, the strongest model surpassed the published state-of-the-art on only 17.8% of tasks (using a threshold of g>0.1, meaning the agent's result was at least 10% better than the published baseline) [1]. In plain terms, even the best AI agent failed to beat the original paper's result on more than four out of five tasks. That's a stark reminder that current coding agents are far from matching human scientific expertise.
But the benchmark also reveals that the failures are not about misunderstanding the task — they're about choosing the wrong method and not having enough compute budget [1]. That's an important nuance: the agents understood what was being asked, but they often picked an inferior approach or ran out of computational resources. This suggests that progress in agentic science may come more from improving method selection and scaling compute than from better language understanding. However, it also means the benchmark's results are tied to the specific compute budgets and model configurations tested — a different setup might yield different numbers.
Beyond code: what scientific innovation actually requires
NatureBench's own analysis shows that agents succeed primarily through 'methodological translation' — converting a scientific task into a familiar supervised prediction problem — rather than through genuine scientific invention [1]. That's a crucial insight: the benchmark can reveal an agent's ability to adapt known methods, but it cannot reveal whether an agent can generate fundamentally new scientific ideas. The authors themselves frame the benchmark as a step toward 'discovery,' but the results suggest that current agents are still in the reproduction-and-translation phase.
Real scientific discovery, as exemplified by the field of bioinspired nanozymes, often requires transferring principles from one domain to another — for example, grafting the catalytic mechanisms of natural enzymes into the design of synthetic nanomaterials [3]. That kind of cross-domain insight is not something a coding benchmark can easily capture, because it involves conceptual leaps, not just code execution. NatureBench can tell you if an agent can implement a known method, but it can't tell you if it can make the kind of creative leap that leads to a new class of materials or a new theory. So while NatureBench is a valuable tool for assessing coding agents, it should be seen as a measure of technical proficiency, not scientific creativity.
About These Sources
This answer is built on 3 studies (2 peer-reviewed, 1 preprint) — published from 2021 to 2026, 2 from 2024 or later, 2 in Q1 journals, collectively cited 667 times — selected as the most relevant from 3 studies that passed quality screening, drawn from 31 papers retrieved from a database of over 500 million.
Sources used in this answer
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
NatureBench, a benchmark of 90 tasks from Nature-family papers, found that the strongest AI coding agent surpassed published state-of-the-art on only 17.8% of tasks (g>0.1), with failures driven mainly by wrong method choice and insufficient compute, not task misunderstanding.
Towards agentic science for advancing scientific discovery
A 2025 commentary on agentic science outlines that AI agents for discovery must reason, plan, and interact with digital and physical environments, highlighting the open-ended nature of real scientific work that benchmarks like NatureBench may not fully capture.
Nanozymes Inspired by Natural Enzymes
A 2021 review of bioinspired nanozymes shows that high-activity enzyme mimics are created by transferring catalytic principles from natural enzymes into nanomaterial design, illustrating the kind of cross-domain conceptual innovation that coding benchmarks cannot measure.