CriPO: Rescuing Suppressed Intelligence in Rubric-Based RL
Enhancing Rubric-based RL via Self-Distillation
The paper introduces Criterion-Distilled Policy Optimization (CriPO), an on-policy training framework designed to enhance rubric-based Reinforcement Learning (RL) for open-ended LLM tasks. CriPO addresses the limitations of standard algorithms like GRPO by using localized on-policy self-distillation to fix "Unexplored" and "Suppressed" criteria, achieving SOTA performance on medical and scientific benchmarks with 2x better sample efficiency.
TL;DR
Standard Reinforcement Learning from Human Feedback (RLHF) often collapses multi-dimensional quality rubrics (e.g., "be accurate," "be empathetic," "be concise") into a single scalar reward. This aggregation leads to "Unexplored" and "Suppressed" criteria where the model either never discovers a behavior or accidentally learns to penalize a good behavior because the rest of the response was poor. CriPO (Criterion-Distilled Policy Optimization) fixes this by using the model's own "inner teacher" to inject missing behaviors and protect useful tokens from being penalized, all while staying strictly on-policy.
The "Losing the Signal in the Noise" Problem
In verifiable domains like Math or Coding, rewards are binary and clear. In open-ended domains like medical consultation, we use rubrics. However, current algorithms like GRPO (Group Relative Policy Optimization) face a fundamental "blind spot":
- Unexplored Criteria: If none of the 8 sampled responses satisfy "Criterion A," the model learns nothing about Criterion A.
- Suppressed Criteria: If a response satisfies "Criterion B" but is otherwise terrible, it gets a negative advantage. The model then learns to avoid the good behavior associated with Criterion B.
Statistical analysis by the authors shows that in medical tasks, over 57% of samples suffer from suppressed criteria, where useful intelligence is systematically discarded during training.
Methodology: The CriPO Framework
CriPO doesn't replace GRPO; it augments it. It acts as a surgical intervention at the token level using On-Policy Self-Distillation (OPSD).
1. Behavior Injection (For the Unexplored)
Instead of just hoping the model stumbles upon a missing criterion, CriPO takes the best current rollout and asks the model itself: "How would you revise this to meet the missing criteria?"
- Forward-KL Distillation: The student model is trained to match this "revised" distribution.
- Localized Filtering: To avoid noise, CriPO only distills tokens where the KL divergence is high—meaning only the tokens that actually changed to satisfy the rubric.
2. Advantage Flipping (For the Suppressed)
This is the most novel part of the paper. When a rollout has a negative reward but satisfies some rubrics, CriPO identifies the tokens responsible for that "good" part.
- Counterfactual Teacher: The model is asked to remove the satisfied parts of the response. By comparing the original and the "weakened" response, CriPO pinpoints exactly which tokens represent the high-quality content.
- The Flip: These specific tokens are given a positive advantage () even if the overall response is negative. This prevents the model from "throwing the baby out with the bathwater."
Experimental Performance
The authors tested CriPO on Qwen3 models across Medicine (HealthBench) and Science (ResearchQA) benchmarks.
Key Breakthroughs:
- Efficiency: CriPO is a "speedrunner." It reaches the peak performance of standard GRPO in half the time (2x speedup).
- SOTA Results: On the Medicine benchmark, it achieved a +3.2 average gain over the baseline.
- Reasoning Depth: Case studies show that CriPO-trained models produce longer, more structured diagnostic rationales compared to GRPO, which often jumps to incorrect conclusions.
Critical Insight: Why it Works
The "magic" of CriPO lies in its On-Policy nature. Unlike previous methods (like HeRL) that generate entirely new trajectories under "privileged" guidance (leading to exposure bias/mismatch), CriPO provides token-level supervision on the model's own existing outputs. It doesn't ask the model to become someone else; it asks the model to be the best version of itself.
Conclusion & Limitations
CriPO represents a significant step forward in making RL more interpretability-aware at the token level. However, it does add per-step computational overhead due to the multiple teacher inferences required. Additionally, the reliance on an "LLM-as-a-Judge" (Qwen3-32B) means the policy's quality is ultimately capped by the judge's ability to distinguish subtle rubric nuances.
Future Work: Scaling this to massive 70B+ models and investigating if "Advantage Flipping" can be automated without manual counterfactual prompting.