Shadow Evaluations: Why Today's AI Agents Can Engineer Research but Fail to Think It
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Summary
Problem
Method
Results
Takeaways
This paper introduces "shadow evaluations" as a novel method to assess AI agents' ability to conduct open-ended AI research, by tasking frontier models with the central research question of unpublished NeurIPS papers and having the original authors grade the output. Using Claude Opus 4.8 and GPT-5.6 Sol, both agents failed to produce publishable work, receiving unambiguous rejections (overall scores of 2/6 and 1/6). The study identifies five recurring failure modes—poor judgment, uncreative problem-solving, ineffective backtracking, poor resource awareness, and instruction drift—that go beyond mere engineering limitations.
## TL;DR
The paper introduces shadow evaluations—a new method to measure AI R&D automation by tasking agents with the central research question of unpublished top-conference papers and having the original authors grade the output. Using Claude Opus 4.8 and GPT-5.6 Sol, both agents were unambiguously rejected (scores 2/6 and 1/6). The agents completed all engineering autonomously but failed on five critical dimensions: judgment, creativity, backtracking, resource awareness, and instruction following.
**Background Positioning:** This work sits in the evolving landscape of AI R&D evaluation, bridging the gap between narrow verifiable benchmarks (e.g., MLE-Bench, RE-Bench) and blind peer review (e.g., AI Scientist-v2). It provides a third, complementary method that combines open-ended tasks with expert grading, offering early evidence that today's agents can do the *engineering* of AI research but not the *science*.
## Problem & Motivation
**Why is this hard?**
Most AI R&D evaluations fall into two camps:
- **Verifiable benchmarks:** Agents optimize a fixed metric (e.g., Kaggle score, training loss). These are objective and scalable but exclude open-ended research where the goal is ambiguous.
- **Blind peer review:** Agents submit papers to conferences. But peer review is highly stochastic (NeurIPS 2014/2021 experiments show ~50% of decisions are random), overstretched, and doesn't reveal rejection rates.
**The gap:** Neither approach tells us whether an agent can formulate hypotheses, decide what evidence matters, recognize when an approach is failing, and pivot creatively. Yet forecasts of explosive AI progress hinge on exactly this capability.
**The authors' insight:** Use the authors themselves as reviewers. By taking an unpublished paper's central question, giving it to an agent, and having the original authors grade the output, we get:
- Open-ended task (not a fixed metric)
- Uncontaminated question (not in training data)
- Expert reviewers who deeply understand the problem
## Methodology: The Core Mechanism
**Shadow Evaluations:**
The authors partnered with authors of two unpublished NeurIPS 2026 submissions:
- **Personas paper:** Can LLM personas be decomposed, measured, and controlled as positions in a structured "trait space" using weight-space interventions?
- **TabPFN paper:** Design a deployment-time distribution-shift detector for tabular foundation models (PFNs).
**Experimental Setup:**
The agents (Claude Opus 4.8 with extra-high reasoning on OpenClaw; later GPT-5.6 Sol with Codex as robustness check) received:
- 6 days wall-clock time
- $3,000 in API credits
- GPU credits for experiments
- Full Linux VM and open web access
- Subagent for self-review, plus external AI review tools (Stanford Agentic Reviewer, CMU Paper Reviewer, refine.ink)
The scaffold architecture is shown below:
The agent could delegate to subagents, run experiments, check its own budgets, and receive AI feedback. The authors only intervened for a scaffold bug, a deadline extension, and a readability request—all non-specific to the research questions.
## Experiments & Results
**Main Result: Unambiguous Rejection**
| Criterion | Personas | TabPFN | Summary of expert comments |
|---|---|---|---|
| Quality | 2/4 | 1/4 | Unprincipled data and experiment choices; conclusions did not follow from the evidence. |
| Clarity | 1/4 | 2/4 | Dense, unclear writing; hard to tell what matters. |
| Significance | 2/4 | 2/4 | Of limited interest; not well justified over prior and comparable work. |
| Originality | 3/4 | 2/4 | New datasets and some new methods, but built primarily on prior work. |
| Overall | 2/6 | 1/6 | Both unambiguous rejections. |
| Confidence | 4/5 | 5/5 | Both reviewers were confident or certain. |
*Table 1: Summary of the original paper authors' reviews of the agents' submitted work. Source: Table 1 from the paper.*
**Key Quantitative Observations:**
- The agents left **>50% of their API budget unused** despite being able to monitor it in real time.
- The self-review process **never returned a single acceptance** across dozens of rounds.
- Both agents **retired their most ambitious hypotheses within the first 10 hours** and never fundamentally shifted approach.
- The final papers **exceeded page limits** (10 pages vs. 9-page limit) and had **fewer references** than the original papers (16 vs. 52 for Personas; 36 vs. 69 for TabPFN).
**Planned vs. Realized Exploration:**
*Figure 2: The Personas agent planned 42 hours of exploration but coalesced around a method after only 5 hours. The TabPFN agent also committed to its headline finding 40 hours early.*
**Five Recurring Failure Modes:**
1. **Lack of judgment:** The agents didn't recognize what constitutes a compelling result. They used small, hand-curated datasets and presented underpowered negative results as substantive findings.
2. **Uncreative responses to feedback:** When AI reviews surfaced fundamental issues (e.g., "the results hinge on n=1 cells"), the agents responded by adding caveats rather than redesigning experiments.
3. **Ineffective backtracking:** Agents made local pivots but never fundamentally restarted. Both committed to a weak approach within the first 10 hours and never changed course.
4. **Poor resource awareness:** Agents left over half the API budget unused and finished hours early despite papers that didn't meet their own quality bar.
5. **Instruction drift:** Agents ignored explicit rules about exploration time, page limits, and use of external review tools.
**Robustness Check:**
Rerunning TabPFN on GPT-5.6 Sol with Codex reproduced nearly all failure modes. The agent exhausted its API budget in 2 days (leaving 100 hours of time unused), failed to run appropriately powered experiments, and returned a draft with misformatted figures.
**Positive Findings:**
- The agents completed all engineering steps autonomously: literature review, GPU debugging, hundreds of experiments, robustness checks, and full LaTeX compilation.
- The initial hypotheses were judged reasonable and interesting by the original authors.
- No reward hacking was found; the agents were honest about negative results and provided reproducible code.
## Critical Analysis & Conclusion
**Takeaway:**
This paper provides early evidence that today's frontier AI agents can do the *engineering* of AI research (debugging, running experiments, writing papers) but cannot do the *science* (exercising judgment, creatively solving problems, backtracking from dead ends). The gap between verifiable task performance and open-ended research ability is stark.
**Limitations:**
- **Small sample size:** Only two papers studied, with five runs total.
- **Non-blind reviewing:** Original authors knew the papers were AI-generated, which could introduce bias.
- **Question selection:** The two papers represent empirical NeurIPS research; results may not generalize to other AI research types (e.g., theoretical, incremental).
- **Scaffold and model limitations:** While a robustness check reproduced the failures, better scaffolds or models (e.g., Anthropic's Fable 5, which was deliberately limited) might yield different results.
- **Elicitation threats:** The authors may have under-elicited agent performance due to scaffold bugs or suboptimal prompting.
**Future Work:**
- Test on a larger set of papers with more capable models (GPT-5.6 Sol, Opus 5, Fable 5).
- Optimize scaffolds to better handle context management and resource awareness.
- Explore whether the "generator-verifier gap" can be exploited: since AI reviews reliably rejected weak papers, could they be used to train better generators via reinforcement learning?
- Investigate whether these failure modes are fundamental or can be overcome with architectural changes (e.g., better long-term memory, more robust planning).
**Broader Implications:**
This work raises important questions about the path to recursive self-improvement. If the critical bottleneck is not engineering skill but creative judgment, then simply scaling compute and data may not be sufficient. The paper suggests that achieving autonomous AI research may require breakthroughs in how agents handle ambiguity, reassess their own assumptions, and creatively pivot—capabilities that are not yet present in frontier models.