[ICLR 2025] DARE-bench: Why Your LLM Data Scientist Fails (and How to Fix It with Verifiable Training)

TL;DR

Data Science is not just about the final number; it's about the process. Most LLMs are "lucky" guessers but "messy" coders. DARE-bench introduces a massive 6,300-task benchmark that forces models to follow strict instructions and provides the first large-scale verifiable dataset to train DS agents via RL. The result? A massive 8x jump in performance for open-source models after training.

The Hidden Crisis in Data Science Agents

Current benchmarks (like DS-1000 or MLE-bench) mostly focus on The Result. If the model gets the accuracy right, it passes. However, in the real world, a Data Scientist must follow specific steps: "Use this specific scaler," "Handle outliers this way," or "Ensure reproducibility."

Prior work failed to measure Instruction Fidelity. Without a way to verify how a model got to the result, we cannot effectively train them using Reinforcement Learning. DARE-bench solves this by providing "Reference Traces"—if the model deviates from the expert process, it fails, even if the final prediction is "close enough."

Methodology: The Verifiable Pipeline

The authors didn't just scrape Kaggle; they built an automated "Task Factory."

1. Instruction Following (IF): The agent is given a strict recipe (e.g., "Use Logistic Regression with seed 42"). Performance is measured by whether the agent's output exactly matches a reference solution's output.
2. ML Modeling (MM): The agent is given freedom to optimize for the best F1-score or R2, simulating an outcome-driven Kaggle competition.

The Architecture of Verification

The core innovation lies in the Sandbox Integration. By enforcing a 10-minute wall-clock limit and fixed environments, the benchmark removes the "it works on my machine" randomness that plagues LLM code evaluation.

Figure 1: The DARE-bench workflow—from natural language questions to sandboxed code execution and ground-truth comparison.

Experimental Results: The Open-Source Gap

The evaluation reveals a sobering reality: Most open-source models are incompetent at multi-step Data Science.

• Claude-3.7 Sonnet and GPT-5 currenty dominate, showing high "Extended Thinking" capabilities.
• Claude-4 (beta) surprisingly struggled because it over-decomposes tasks into too many tool calls, hitting execution limits.
• Qwen3-32B (Baseline) scored only 23.25, failing nearly all complex time-series tasks.

Table 1: Benchmark results across different LLMs. Notice the near-zero performance of standard open-source models on Time-Series tasks (Time-CF).

The "Moonshot" of Training: SFT and RL

The most exciting part of this paper is the Training Resource aspect. By using the DARE-bench training set, the authors performed:

1. Supervised Fine-Tuning (SFT): Using "Best-Valid" (BV) and "All-Valid" (AV) traces.
2. Reinforcement Learning (GRPO): Directly optimizing the model to maximize the F1/R2 rewards in the sandbox.

The results are staggering:

• Qwen3-32B jumped from 23.25 to 42.91 (nearly 2x improvement).
• Qwen3-4B (a tiny model) moved from 4.39 to 37.40 (over 8x improvement) via RL.

This proves that the "reasoning gap" in Data Science agents isn't just about model size—it's about alignment with the DS workflow.

Critical Analysis & Insights

Why do they fail?

The authors performed a "Post-Mortem" on failed trajectories. The top killers of DS Agents are:

1. Tool Argument Mismatch: Models forget to pass the filename into the custom code executor.
2. Implicit Bias: Models often use column names from the description without checking the actual metadata.txt, leading to KeyErrors.
3. Time-Series Illiteracy: Most models default to a "mean" or "last value" heuristic because they lack the deep temporal reasoning required for exogenous feature integration.

The Takeaway

DARE-bench is a call to action. We have reached the limit of what "prompt engineering" can do for Data Science. If we want agents that can actually replace or assist senior data scientists, we must treat Code Execution Traces as first-class citizens in our training datasets.

For the developer community, the release of 6,300 verifiable tasks is a goldmine for anyone looking to build a specialized "DataScience-Llama" or "Qwen-Analyst."

Senior Editor's Note: This work successfully bridges the gap between Software Engineering (verifiable tests) and Data Science (outcome metrics). It is a rare benchmark that provides both a yardstick and a fuel for future models.