EURUS: Bridging the Reasoning Gap with Preference Trees and Interaction

TL;DR

The divide between open-source LLMs and proprietary giants like GPT-4 has long been widest in the realm of complex reasoning. EURUS, a new suite of models from Tsinghua University and partners, effectively bridges this gap. By training on ULTRAINTERACT—a massive dataset of "preference trees" featuring multi-turn environment interactions—EURUS-70B manages to eclipse GPT-3.5 Turbo on hardcore coding and math benchmarks.

Motivation: Why Relative Preference Isn't Enough for Logic

Most alignment techniques today rely on DPO (Direct Preference Optimization), which was designed for general conversation where "better" is often subjective. However, reasoning is different: an answer is either objectively right or wrong.

The authors discovered a startling trend: while DPO works for chat, it often hurts reasoning performance. The culprit? DPO only cares about the gap between a good and bad answer. In reasoning, if the "good" answer's absolute reward drops too low, the model loses its "grounding" in logic. This insight led to the creation of a more robust way to teach models how to think, not just how to rank.

Methodology: The Power of the Preference Tree

The core innovation of this work is ULTRAINTERACT. Instead of simple prompt-response pairs, it organizes data into Preference Trees.

1. Unified Reasoning Patterns

The model doesn't just guess; it's taught to use diverse strategies like Chain-of-Thought (CoT) or Modularization Programming (treating logic like building tools).

2. Multi-Turn Interaction

Reasoning is an iterative process. EURUS is trained on trajectories where a model:

• Proposes a solution.
• Receives feedback from a Python Interpreter (Environment) or a Critique Model (GPT-4).
• Refines its answer based on the error logs or suggestions.

3. The Reward Modeling Breakthrough

To solve the DPO failure, the authors introduced an additional objective for their Reward Model (RM): ${\mathcal{L}}_{DR}=-\log (\sigma (r_{h}eta(x,y_c)))-\log (\sigma (-r_{h}eta(x,y_r)))$ This forces the model to not just prefer $y_c$ over $y_r$, but to explicitly push the "correct" reward up and the "incorrect" reward down.

Experiments: SOTA Performance

EURUS was tested across 12 benchmarks, with a focus on Out-of-Distribution (OOD) tasks to ensure the model wasn't just memorizing.

• LeetCode Contest: EURUS-70B hits 33.3% pass@1, leaving other open-source models in the dust.
• TheoremQA: At 32.6%, it is the only open-source model currently on par with GPT-3.5 Turbo for university-level STEM.

Interestingly, the 7B version of EURUS is so efficient it rivals models ten times its size in specific reasoning tasks, making it a highly distilled logic engine.

Deep Insight: Absolute Rewards vs. Relative Preference

The paper's analysis of reward patterns (Figure 5) provides a critical lesson for the AI community. In DPO, the rewards for both chosen and rejected responses often decrease over time. For general chat, this is fine. For math and code, this "pessimism" ruins the model's ability to identify a correct solution confidently. Algorithms like KTO and NCA, which maintain positive reward growth for correct solutions, are significantly more effective for reasoning generalists.

Conclusion & Future Outlook

EURUS proves that the secret to "reasoning" isn't just more parameters; it's high-fidelity interaction data. By treating reasoning as a tree-search and refinement process rather than a single-shot generation, open-source models can finally compete with the likes of GPT-3.5 and beyond.

Limitations: While EURUS excels at logic, its general knowledge (MMLU) is bound by its base models (Mistral/CodeLlama). The next frontier will likely be merging this intense reasoning capability with broader world knowledge.