MATHNET: Why Your AI is Good at Solving Math but Terrible at Finding It

TL;DR

Researchers from MIT and other institutions have released MATHNET, the most comprehensive multimodal and multilingual Olympiad-level dataset to date. Spanning 40 years of competitions across 47 countries, it reveals a shocking gap: while models like Gemini-3.1-Pro can solve nearly 80% of these elite problems, modern embedding models are almost completely "blind" to mathematical structure when asked to retrieve similar problems, achieving a dismal 5% Recall@1 on equivalence tasks.

The "Equivalence" Crisis in AI Math

Current LLMs are often evaluated on their ability to output a final answer. However, true mathematical mastery involves Analogical Reasoning—recognizing that a problem about unit circles in 2D space is structurally identical to one about vector norms, even if the variables change from $x$ to $\alpha$.

Prior benchmarks like GSM8K or MATH focus on English-centric, high-school level word problems. They miss the "structural resonance" that human mathematicians use to solve new problems by recalling old ones. MATHNET addresses this by introducing a fine-grained taxonomy of similarity:

• Invariance: Strict equivalence (e.g., variable renaming).
• Resonance: Different problems solvable by the same core trick or lemma.
• Affinity: Thematic relation (e.g., both are about Prime Number Theory).

Methodology: Scalable Expert Curation

The authors didn't just scrape the web; they built a pipeline to ingest 25,000+ pages of official national contest booklets.

1. Extraction: Using dots-ocr to parse complex LaTeX and diagrams from 17 different languages.
2. Alignment: A three-stage LLM pipeline (Gemini/GPT-4) to match problems with sections of solutions often found hundreds of pages apart.
3. Verification: Human experts and rule-based checkers ensured that the solutions provided were complete and accurate.

The Experiment: Solving vs. Retrieving

The benchmark tests 27 state-of-the-art models across three tasks. The results show a clear stratification of intelligence.

1. Problem Solving (The Frontier)

Reasoning models like Gemini-3.1-Pro (78.4%) and GPT-5 (69.3%) are approaching human expert levels in Algebra. However, Geometry and Discrete Math remain significantly harder, likely due to the complex spatial and logical invariants required.

2. The Retrieval Failure

This is the paper’s most critical insight. When using standard embeddings (like OpenAI’s text-embedding-3-small or cohere-embed), the models fail to distinguish between an equivalent problem and a "hard negative" (a problem that looks identical but changes one operator, making the math completely different).

As seen in Figure 6, the similarity distributions for equivalent vs. non-equivalent problems overlap significantly, suggesting embeddings are distracted by "surface-level noise" like keywords.

Does RAG Help?

The authors found that Retrieval-Augmented Generation (RAG) is a double-edged sword.

• Expert-RAG: Providing a hand-picked, structurally similar problem boosted DeepSeek-V3.2-Speciale to a staggering 97.3%.
• Embed-RAG: Using an automated retriever often hurt performance because the retriever surfaced "near-miss" problems that confused the reasoning engine.

Critical Analysis & Future Outlook

MATHNET proves that we have solved the "language" of math (syntax) but not yet the "manifold" of math (structure).

The Takeaway: Future mathematical AI must move toward Symbolic-Aware Embeddings. We need models that don't just see the word "Triangle" but understand the underlying geometric constraints. MATHNET provides the data infrastructure to train the next generation of retrievers that can actually "think" in analogies.

Access the benchmark at: mathnet.mit.edu