[ArXiv 2024] Code2Math: Breaking the Math Data Bottleneck via Agentic Exploration

Executive Summary

TL;DR: The "Data Wall" is real in AI mathematics. As models approach IMO levels, we lack new, hard problems to test them. Code2Math introduces a multi-agent system that treats math problem creation as an iterative, code-driven exploration process. By using Python tools and Theory of Mind, agents don’t just "rephrase" questions—they evolve them into structurally deeper challenges that baffle even the models that created them.

Academic Positioning: This work moves beyond simple data augmentation (like MATH-Perturb) toward Autonomous Curriculum Generation. It leverages the "capability asymmetry" of LLMs—the fact that it is often easier for a model to construct a valid trap than to escape one.

The Problem: The Scarcity of "Aha!" Moments

Traditional math data synthesis focuses on Artificial Complexity: making numbers messier or steps more tedious. However, true mathematical difficulty stems from the Burden of Discovery—the hidden insight or "Aha!" moment required to break a problem wide open.

Existing LLMs are becoming "template-matched" to standard competition problems (AIME, IMO). To push boundaries, we need problems that are:

1. Mathematically Sound: No logical fallacies.
2. Anti-Templating: They must break standard heuristic paths.
3. Scalable: Generated without expensive human experts.

Methodology: The Three-Agent Pipeline

The authors propose a specialized multi-agent architecture built on the Smolagents framework, where agents have access to a Python sandbox (NumPy, Z3, SymPy).

1. The Evolution Agent (The Architect)

This agent performs "Free Exploration." It analyzes the seed problem's solution, identifies the cognitive bottleneck, and then uses code to search for tighter bounds or more complex combinatorial structures. It adopts Theory of Mind to anticipate how a human solver would approach the problem and deliberately conceals the entry point.

2. The Solvability Verification Agent (The Proofreader)

Solvability is the hardest part of generation. This agent attempts to reconstruct a full, logically consistent solution. The logic is simple: if a flawless logical chain exists, the problem is solvable.

3. The Difficulty Verification Agent (The Judge)

This agent scores the adaptation (1-5) based on Cognitive Depth.

• Score 2: Tedious algebra (Fail).
• Score 4-5: Requires a profound "Aha!" moment or connects disparate fields (Pass).

Figure 1: The Code2Math Multi-Agent System. Note the integration of math-specific code tools.

Experiments: Do Evolved Problems Actually Stymie SOTA?

The authors tested 100 seed problems evolved by models like DeepSeek-Reasoner and Gemini-3-Pro.

Key Finding 1: Capability Asymmetry

One of the most striking results is that models can create problems they cannot solve. For instance, problems evolved by DeepSeek-Reasoner caused a 21% drop in Gemini-3-Flash-Thinking's solve rate and even challenged GPT-5.2-High. This suggests that the "synthesis" capability of LLMs is currently outpacing their "reasoning" capability.

Key Finding 2: The "Rightward Shift" in Reasoning

To solve evolved problems, models didn't just fail; they tried harder. The Average Token Consumption (ATC) significantly shifted to the right, indicating that the evolved problems successfully forced models into prolonged exploration and self-correction.

Table 1: Solvability and Difficulty metrics. "Evolution-SR" (Solve Rate) is consistently lower than the "Origin-SR".

Case Study: From Inequality to Moment Theory

In one instance, the agent took a localized inequality proof about random variables and evolved it into a structural problem requiring the characterization of the "fourth central moment" as a function of the mean.

• Original: A basic quadratic argument.
• Evolved: Requires recognizing that extrema under moment constraints are achieved by discrete distributions, forcing a jump from "bounding tricks" to "moment theory."

Critical Analysis & Conclusion

Takeaway: Code2Math proves that we don't need humans to build the next generation of Math benchmarks. By providing agents with execution environments, we allow them to move from "stochastic parroting" to "empirical discovery."

Limitations:

• Efficiency: The process is computationally expensive, often needing 10+ rollouts to get one qualified problem.
• Solvability: Logical consistency remains the primary bottleneck (averaging 2-5 failures per success).

Future Outlook: The ability for models to autonomously scale difficulty is the first step toward Recursive Self-Improvement. If a model can generate a curriculum of "just-hard-enough" problems, it can train on its own synthetic data to reach superhuman performance.