Is One Layer Enough? The Surprising Concentration of RL Gains in LLMs

TL;DR

Contrary to the common practice of updating every parameter during Reinforcement Learning (RL) post-training, a new systematic study reveals that training just a single transformer layer can recover—and sometimes even beat—full-parameter RL training. Across multiple model families (Qwen) and tasks (Math, Code, Agents), RL gains are not uniform; they are heavily concentrated in a "sweet spot" located in the middle of the transformer stack.

The Illusion of Uniform Adaptation

When we perform RL post-training (like GRPO or PPO) to make a model better at math or coding, we typically update the entire network. The implicit assumption is that the "knowledge" gained from the reward signal is distributed across all layers.

The authors of this paper challenged this assumption. By measuring Layer Contribution—the fraction of full RL improvement recovered by training a layer in isolation—they found a striking disparity. In many cases, specific middle layers achieved a contribution score > 1.0, meaning they were more effective at learning from RL when trained alone than the entire model was when trained together.

The "Middle-Layer" Insight

Across seven different models and three RL algorithms, a consistent structural pattern emerged:

1. The Dead Ends: Layers near the input (embeddings) and output (head) contribute significantly less, with Layer 0 in some models even showing negative contribution.
2. The Engine Room: High-contribution layers are strictly concentrated between 40% and 60% of the network depth.
3. Cross-Task Stability: A layer that is good for math is almost always good for coding or agentic decision-making. This suggests that "layer importance" is a property of the pretrained weights, not the fine-tuning task.

Figure 1: Showing how contribution peaks in the middle layers across different model scales and tasks.

Why Does This Happen? (Methodology & Intuition)

The authors used Group Relative Policy Optimization (GRPO) as their primary tool. By freezing everything except one layer $k$, they forced the model to absorb all the RL "reasoning" into that specific parameter subspace.

The discovery that single layers can outperform the whole model suggests a dilution effect: when all layers are updated, weaker layers may introduce noise or "dilute" the specialized adaptation occurring in the middle layers.

Interestingly, the Weight Change Magnitude ($\Vert \Delta heta\Vert$) does not explain this. Even if a low-contribution layer is forced to change its weights significantly, it doesn't achieve the same performance gains as a middle layer. This implies that the middle layers possess a unique functional capacity for reasoning that other layers lack.

Exploiting the Structure: Guided RL Training

The researchers didn't just observe; they optimized. They proposed two main strategies:

• Layer-Selective Training: Freezing low-contribution layers and only training the top $k$ (e.g., top 10).
• Heuristic Middle-Layer Training: A "zero-cost" version where you simply train the middle layers by position without profiling.

Figure 2: Performance of guided strategies vs. full-parameter baseline. Selective training (green) consistently yields the highest gains.

Key Result: On Qwen3-8B, training only the 10 best layers resulted in a +2.68 point lead over full-parameter training.

Diverse Experts in Middle Layers

The paper also uncovers a fascinating diversity. Even if two middle layers (e.g., Layer 10 and Layer 13) have similar accuracy, they solve different problems. By using a simple majority vote among models trained on different individual layers, the authors achieved results far superior to the full-parameter baseline and even better than standard self-consistency sampling.

Critical Analysis & Conclusion

This work provides a paradigm shift for LLM post-training:

• Efficiency: You don't need to update every parameter to get SOTA reasoning gains.
• Performance: Selective training is a form of regularized optimization that prevents "noise" from weak layers.
• Limitation: While the middle-layer concentration is robust, a deep theoretical explanation of why the middle layers are the reasoning engine is still missing.

For practitioners, the message is clear: If you are RL-tuning an LLM and have limited compute, start by focusing on the middle layers.