From Reasoning Traces to Reusable Modules: Why RL is the Key to Compositional Generalization

Executive Summary

TL;DR: Reasoning in Large Language Models (LLMs) isn't just about imitating "golden" solutions—it's about learning a library of reusable mental tools. This paper reveals that Supervised Fine-Tuning (SFT) provides the "raw materials" (skills), but Reinforcement Learning (RL) acts as the "industrial separator," decomposing entangled training traces into modular, atomic skills and routing logic that can be recombined to solve unseen, complex problems.

The work establishes a new theoretical benchmark: The Hierarchical Latent Selection Model, proving that RL’s value lies in exploring "off-support" combinations that SFT fundamentally cannot teach.

The Problem: The Entanglement Trap

Current LLM training often relies on SFT to mimic specific chains-of-thought. However, SFT suffers from a major weakness: Statistical Entanglement.

If a model only sees the skill "Addition" paired with the routing "Carry the result to Step 3," it treats them as a single, fused routine. When a new problem requires "Addition" but needs to "Carry the result to Step 5," the SFT-trained model fails because it never learned to separate the operation from the flow. This is why SFT-only models excel at mimicking but struggle with Out-of-Distribution (OOD) compositions.

Methodology: The Architecture of Reasoning

The researchers model reasoning as a hierarchy of discrete latent selections.

1. Atomic Skills (Operations): Local transformations like adding a number or substituting a variable.
2. Routing Mechanisms (Flow): The logic that decides which previous intermediate result to use next.

The Latent Selection Hierarchy

The authors prove (Theorem 3.1) that under certain conditions, like Neighborhood Coverage, these latent modules are identifiable. SFT data usually has a "hidden support" problem—it doesn't provide enough variation to prove a module is independent. RL solves this by sampling diverse rollouts and rewarding success, which enriches the model's exposure to "identification-critical events."

Experimental Proof: Decomposition in Action

Using synthetic string transformation tasks, the authors tested how models handle increasingly deep compositions (Level $L$).

Key Finding: RL Beats Atomic Supervision

One might think training a model on "atomic tasks" (Level 1) is the best way to teach skills. Surprisingly, training on compound traces (SFT) followed by RL exploration leads to much stronger generalization in deeper tasks than just training on isolated atoms.

The SFT-RL Data Formula

The most striking discovery is the relationship between SFT and RL data sets:

• Overlapping data ($C_{SFT}\supset C_{RL}$): Weakest generalization.
• Disjoint data ($C_{SFT}\cap C_{RL}=\varnothing$): Strongest generalization.

This suggests that RL's power is wasted on re-learning what SFT has already covered. RL should be used to push the model into the "void" where new combinations of skills exist.

Real-World Evidence: Qwen-3 Case Study

The authors validated their theory on open-source models (Qwen3-4B). They tokenized these models' reasoning traces and found that RL-tuned models exhibited higher diversity in short skill n-grams (2-3 steps). This confirms that RL encourages the local recombination of skills—the hallmark of modular reasoning—rather than just memorizing longer, fixed templates.

Deep Insights & Conclusion

Takeaways for AI Practitioners

• Curriculum Matters: Stop treating SFT and RL as independent stages. Use SFT to define the "vocabulary" of your model (all skills/routers) and use RL to teach the "grammar" (how to combine them in new ways).
• Verification is the Catalyst: Verifiable rewards (exact match, code execution) are essential because they provide the signal needed to disentangle skills from noise.

Limitations

The study focuses on synthetic and mathematical reasoning where rewards are binary and clear. Extending this to open-ended creative writing or subjective reasoning remains a challenge, as the "verifier" in those domains is much noisier.

Final Thought: Reasoning isn't about knowing the answer; it's about knowing how to rearrange the pieces you already have. RL is the engine that teaches the model how to move those pieces.