[ICLR 2025] Durian: Taming RL Instability in Multimodal Reasoning via Difficulty-Aware Re-grouping

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR) is the engine behind reasoning models like DeepSeek-R1. However, applying it to images and text simultaneously often breaks due to "extreme samples" that crash the reward variance. Durian solves this by re-grouping samples based on Visual Entropy and Model Confidence, ensuring stable training and a massive 11.3% average performance boost across multimodal benchmarks.

The "Zero-Variance" Trap in Multimodal RL

In the world of Group Relative Policy Optimization (GRPO), we normalize rewards by the standard deviation (std) of a group. This works great until you hit a group where every response is either perfectly right or perfectly wrong.

In MLLMs, this happens constantly. A simple image leads to "Easy Success" (std ≈ 0), while a blurry or logically impossible one leads to "Hard Failure" (std ≈ 0). When the std is near zero, the advantage calculation explodes, causing the model to overfit to outliers and ignore the "Medium" difficulty samples where actual learning happens.

Methodology: The Two Dimensions of Difficulty

The authors argue that difficulty isn't just about the text; it's a mix of what the model sees (Perceptual) and what the model thinks (Reasoning).

1. Perceptual Difficulty (The Data-Centric View)

Durian analyzes the visual input using spectral analysis. It extracts patch-level features, computes a covariance matrix, and looks at the Eigenvalue Entropy.

• Low Entropy: Variance is concentrated in a few dimensions (simple, repetitive patterns).
• High Entropy: Variance is spread across many dimensions (complex, information-rich scenes).

2. Reasoning Difficulty (The Model-Centric View)

Even if an image is simple, the logic might be hard. Durian uses the model’s own sequence-level log probabilities as a proxy for confidence. If the model is "unsure" (low log-prob), the sample is treated as high-difficulty.

How Re-grouping Cleans Up the Gradients

Instead of calculating std for each individual question's rollouts, Durian pools samples with similar difficulty scores together. By sharing the std within these difficulty-based "buckets," the normalization becomes robust. Extreme samples no longer dominate the gradient because they are compared against their peers, not evaluated in isolation where their internal variance might be zero.

Experimental Results: SOTA Performance with Less Data

Durian was tested on a suite of benchmarks including MathVista, MathVision, and HallusionBench.

Notably, even when trained on a tiny dataset of 2.1k samples (Geometry3K), Durian matched or outperformed models trained on 200k+ samples. This suggests that how you optimize (strategy) is just as important as what you optimize (data).

Deep Insight: Why Why This Matters for the "R1" Era

The industry is moving toward "Verifiable Rewards" (using code executors or math kernels to grade models). Durian shows that for multimodal reasoning—where the "visual ground truth" is often noisy—we cannot treat the RL process as a black box. By injecting a physical/statistical understanding of the input (Visual Entropy), we make RL more "aware" of why it is failing or succeeding.

Limitations

While highly effective, Durian introduces hyperparameters like the weighting coefficients ($\alpha$) and the number of difficulty bins. While the paper shows these are stable, an "Auto-Durian" that dynamically adjusts these would be the next logical step in this research lineage.

Conclusion

Durian proves that stabilizing the variance in GRPO is the "hidden key" to unlocking MLLM reasoning. By simply re-organizing how we look at difficulty, we can turn a jittery training process into a SOTA-beating engine.