From Boards to Bytes: Deciphering the Scaling Interface Between Pretraining and RL

TL;DR

Researchers from NYU and Columbia have turned to the 64 squares of a chess board to solve one of the most expensive mysteries in AI: how much should we pretrain versus how much should we use Reinforcement Learning? By establishing a Joint Pretraining-RL Scaling Law, they demonstrate that while pretraining sets the ultimate "performance ceiling," the amount of pretraining data determines how fast the model learns from feedback.

The "Experience" vs. "Prior" Dilemma

In the current LLM landscape, we see two diverging philosophies. One side (the "Prior" camp) believes scaling massive human-generated corpora is the key to intelligence. The other (the "Experience" camp), inspired by successes like AlphaZero, argues that models should learn from environmental interaction and verifiable rewards.

In the real world, RL for LLMs is almost always "cold-started" from a pretrained base. This leads to a critical compute allocation problem: If you have a fixed budget of FLOPs, where do you stop pretraining and start RL?

Methodology: Chess as the Ultimate Lab

Studying this in natural language is a nightmare—data is messy, and RL rewards are often subjective. Chess provides a "clean room" environment:

• Actions are discrete: Move tokens (e.g., Pe2e4).
• Rewards are verifiable: Win/Loss or Stockfish engine scores.
• Reasoning is measurable: The authors use "synthetic reasoning traces" (prefix trees of possible future moves) to simulate a Chain-of-Thought (CoT).

The Joint Scaling Law: Prediction through Loss

The paper’s core contribution is a mathematical bridge between stages. They find that:

1. The Starting Line (Reference Reward): The performance of a model after a fixed amount of RL is highly predictable based on its pretraining validation loss. Lower loss = higher post-RL potential.
2. The Learning Speed (Slope): The rate at which a model improves during RL grows linearly with the logarithm of the pretraining tokens. Essentially, the more a model "knows" about the world's distribution, the faster it interprets rewards.

The Formula for Success

The authors propose a local scaling fit: $R(C_{RL},N,T)=f(L_{pt}(N,T))+g(N,T)({\log }_{10}{C}_{RL}-{\log }_{10}{C}_{ref})$ where $f(L)$ represents the performance ceiling derived from loss, and $g(N,T)$ represents the slope driven by data scale.

What Does RL Actually Change?

Is RL just "sharpening" what the model already knows? The authors performed a "surgical" analysis of the move policy across difficulty levels:

• On Easy Puzzles: RL mostly engages in Ground-truth Amplification. It takes a move the SFT policy already liked and makes it dominant.
• On Hard Puzzles: RL performs Tail Discovery. It finds correct moves that had $<5\%$ probability in the base model and brings them to the top.

However, RL isn't a silver bullet. It also suffers from Wrong-mode Amplification on difficult tasks, where it reinforces the wrong reasoning path if it can't find the correct one, explaining why RL often improves pass@1 but fails to significantly move the needle on pass@k.

Transfer to Math

To prove this isn't just a chess quirk, the team tested the law on a 1B OLMo-2 language model trained on math data. The pattern held: pretraining loss predicted post-RL performance, and the learning slope scaled with training tokens.

Key Takeaways for the AI Industry

• RL Strategy: As total compute increases, you should spend a larger percentage of that budget on RL.
• The Initialization Limit: RL is not a substitute for poor pretraining. If the base model hasn't "seen" enough, the RL gains will be shallow and slow.
• Future Work: The "Wrong-mode Amplification" suggests that we need better SFT data or RL objectives that prevent the model from confidently going down the wrong path when faced with difficulty.

Conclusion

This study offers a quantitative compass for researchers navigating the pretraining-to-post-training pipeline. By treating pretraining loss as a predictor of RL success, we can finally begin to treat AI training not as an art of "vibe-checking," but as a rigorous science of compute optimization.