The Ghost in the Machine: Why Some Reasoning Tasks are "Unlearnable" for LLMs
The Unlearnability Phenomenon in RLVR for Language Models
This paper identifies and analyzes the "Unlearnability Phenomenon" in Reinforcement Learning with Verifiable Reward (RLVR), where a subset of hard reasoning examples fails to improve despite the presence of correct rollouts. The authors demonstrate that this issue arises from fundamental representation flaws and gradient misalignment, rather than optimization constraints like sample scarcity or KL penalties.
TL;DR
Even when a model finds the right answer during Reinforcement Learning, it doesn't mean it's actually learning. This paper uncovers the "Unlearnability Phenomenon," where a significant portion of hard reasoning problems remains stagnant despite receiving positive rewards. The culprit isn't the RL algorithm—it's a fundamental flaw in the model's internal representation that only "mid-training" seems to fix.
The "Broken" Reward Loop
The industry has shifted toward Reinforcement Learning with Verifiable Reward (RLVR), popularized by models like DeepSeek-R1 and the GRPO algorithm. The logic is simple: if the model gets the math problem right, give it a +1; if wrong, 0. We expect that as long as the model occasionally "stumbles" upon the right answer, RL will eventually steer the policy toward success.
However, the authors found a startling reality: some problems are simply unlearnable. Even if you force the model to see correct answers (oversampling) or remove the constraints that keep it from changing too fast (KL penalty), a subset of data refuses to budge.
Investigating the Culprits: Is it Optimization or something deeper?
The authors tested three common hypotheses to explain this stubborn lack of progress:
- Scarcity of Success: Maybe the model just doesn't see enough "correct" examples? Result: Negative. Even with a rollout group size of 64 or direct SFT, they didn't learn.
- Algorithmic Constraints: Is PPO clipping or the KL penalty "washing out" the signal? Result: Negative. Removing these constraints didn't bridge the gap.
- Gradient Interference: Are the "wrong" answers canceling out the "right" ones? Result: Negative.
This led to a pivot in interest: If the math of the optimizer isn't broken, the representation of the data must be.
Methodology: The Gradient Outlier Theory
The researchers turned to Cross-Example Gradient Similarity. They measured how the gradient (the direction the model wants to move) for an "unlearnable" example aligns with the rest of the dataset.
The discovery was profound: Unlearnable examples are gradient outliers. Their "learning signal" points in a direction that has almost zero correlation with the rest of the model's knowledge. Essentially, the model sees these problems as isolated islands that don't benefit from any general reasoning skills learned elsewhere.
"Fake" Reasoning (Reward Hacking)
When looking at the reasoning traces, the authors found that for unlearnable examples, the model often arrived at the correct answer through incoherent logic.
- Example: A model might start a math problem correctly, descend into total nonsense in the middle, and then magically output the correct number at the end.
- Insight: Because RLVR only rewards the outcome, the model "hacks" the reward using "fake reasoning." Since this logic is incoherent, it never generalizes, and thus the model never truly "learns" the underlying concept.
The Solution: Mid-Training is the Key
If RL can't fix a broken representation, what can? The authors found that Mid-training (training on massive amounts of high-quality reasoning data before RL) significantly increases gradient similarity. Models like OctoThinker, which underwent mid-training, showed much better alignment in their gradient space than base models like Llama-3.
Critical Insight & Conclusion
This paper serves as a wake-up call for the "RL-is-all-you-need" crowd. Successful RL post-training requires a "well-structured" representation space that only comes from high-quality pre-training or mid-training.
Takeaway for Practitioners: If your model is stuck on certain tasks during RL, don't just throw more rollouts at it or tweak your KL penalty. Your model likely lacks the fundamental "latent signals" to understand the task. You need to go back to the data pipe and fix the representation through supervised mid-training.
Limitations: The study focuses on math (verifiable reward). It remains to be seen if "unlearnability" manifests differently in creative writing or areas where rewards are provided by LLM-as-a-Judge.