PRISM: Fixing the SFT Distributional Drift in Multimodal RL

TL;DR

The standard recipe for training Large Multimodal Models (LMMs)—Fine-tuning (SFT) followed by Reinforcement Learning (RL)—is fundamentally flawed. SFT often introduces "distributional drift" that displaces the model's native strengths. PRISM introduces a three-stage pipeline (SFT → Alignment → RLVR) that uses a specialized Mixture-of-Experts (MoE) discriminator to "repair" the model's perception and reasoning before the final RL stage.

Background: Why SFT is a Double-Edged Sword

In traditional post-training, SFT acts as a "cold start" to teach the model how to reason. However, recent evidence suggests that for very strong base models (like the Qwen3-VL series), token-level imitation of an external teacher during SFT can actually hurt performance.

The model learns to "mimic" the teacher's style without internalizing the logic, leading to a distribution that is neither the teacher's nor its own. In multimodal tasks, this is twice as dangerous: a tiny "perception drift" (misidentifying an object in an image) cascades into a "reasoning failure."

The Core Innovation: PRISM (Pre-Alignment)

PRISM addresses this by inserting a stage called Adversarial On-Policy Distillation. Instead of just matching tokens, the model generates its own "on-policy" responses, and a discriminator tells it how much those responses deviate from the "ideal" distribution.

1. The MoE Discriminator: Disentangling Perception and Logic

One of PRISM's most clever designs is its Mixture-of-Experts (MoE) Discriminator. A single score is often too "blunt" to fix a multimodal model. PRISM splits the job:

• Perception Expert ($D_v$): Measures how well the response is grounded in the visual input.
• Reasoning Expert ($D_r$): Evaluates the consistency and validity of the logical deduction.

By providing decoupled feedback, the model receives a "sharper" gradient for optimization.

2. Black-Box Adversarial Game

Unlike traditional knowledge distillation, PRISM is logit-free. It doesn't need to know the internal probability distributions of the teacher (like Gemini 3 Flash). It treats alignment as a minimax game: the policy tries to "fool" the discriminator into thinking its responses came from the high-quality supervision pool.

Experimental Results: Better Accuracy, Fewer Tokens

The authors tested PRISM on Qwen3-VL-4B and 8B across rigorous math and general multimodal benchmarks (MathVision, MMMU-Pro).

• Consistent Gains: PRISM+GRPO outperformed the standard SFT→GRPO by +4.4 (4B) and +6.0 (8B) points.
• Algorithm Agnostic: The benefits were visible regardless of whether the final RL stage used GRPO, DAPO, or GSPO.
• Efficiency: Interestingly, the aligned models became more concise. They achieved higher accuracy using fewer tokens, suggesting the alignment stage pruned "hallucinated" filler text.

Deep Insight: Alignment as Refinement

The ablation studies offer a fascinating look at why this works. Simply running more RL isn't enough; the model needs to be in the "right neighborhood" of the output distribution first.

The training dynamics (shown below) reveal that the Perception and Reasoning experts follow different convergence speeds. Perceptual alignment peaks early, while reasoning requires more gradual, oscillating correction. A monolithic (non-MoE) discriminator would conflate these signals, leading to noisier updates.

Conclusion and Takeaways

PRISM proves that we can't treat multimodal reasoning as a single objective. By decoupling perception from reasoning and introducing an explicit "distribution repair" stage, we can unlock much higher performance from existing LMMs.

Key takeaway for researchers: If your RL stage is plateaus, your problem might not be the RL algorithm—it might be the distributional drift you inherited from SFT.