Requential Coding: Decoding why Large Models are Actually "Simpler"
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
This paper introduces "Requential Coding," a novel model compression framework that encodes generative models by compressing their training trajectory rather than raw parameters. By utilizing Relative Entropy Coding (REC) and a teacher-student distillation setup, it achieves state-of-the-art compression rates for LLMs, reaching approximately 1 bit per parameter.
Executive Summary
TL;DR: Researchers from NYU and CMU have introduced Requential Coding, a breakthrough in model compression that discards the obsession with parameter counts. Instead, it compresses the process of learning. By having a student model "guess" its training data and only encoding the teacher's corrections, the authors demonstrate that billion-parameter models are far more compressible than we thought—often reaching sub-1-bit per parameter.
Background Positioning: This work bridges the gap between Information Theory and Deep Learning. It moves beyond post-training quantization (PTQ) and standard Prequential Coding to provide a rigorous, non-vacuous theoretical foundation for why LLMs generalize so well as they scale.
The Core Tension: Parameters vs. Information
Why do larger models generalize better despite having billions of parameters to "memorize" data? Traditional compression methods give us two unsatisfying answers:
- Quantization (PTQ): Tells us a model is just its weights. If you have 70B parameters, you pay for 70B parameters, even if most contribute nothing.
- Prequential Coding: Tells us a model is its data. But data is noisy. If you train on high-entropy images, you pay for every random pixel, even if the model only learns the concept of a "cat."
Requential Coding resolves this by asking: How many bits are needed to tell the student what the teacher knows that the student hasn't figured out yet?
Methodology: The Student, The Teacher, and the PRNG
The brilliance of Requential Coding lies in Relative Entropy Coding (REC).
The Workflow
- Shared Randomness: Both the encoder and decoder share a random seed.
- Candidate Proposing: The student model generates thousands of potential "next training batches" based on its current state.
- Teacher Selection: The teacher model (which sees the real data) looks at these candidates and picks the one that best matches the true data distribution.
- Index Encoding: We don't send the data. We only send the index of the candidate the teacher picked.
If the student and teacher perfectly agree, the index is always '1' (0 bits). If they disagree, the cost is proportional to the KL Divergence between them.
Figure 1: The Requential Coding workflow. The student generates candidates; the teacher picks one via REC; the code only stores the selection index.
Revolutionary Insights
1. Scaling Makes Models "Simpler"
Counter-intuitively, as models get bigger, their requential code length per parameter decreases. This proves that larger models are more sample-efficient—they "guess" the right distribution more often, requiring fewer corrections from the teacher.
2. State-of-the-Art Generalization Bounds
By plugging the requential code into a PAC-Bayes bound, the authors achieved the first tight, non-vacuous generalization guarantees for billion-parameter LLMs. As shown in the results, the generalization gap actually vanishes as models grow in the compute-optimal regime.
Figure 2: Requential codes certify that models generalize better with scale, outperforming 4-bit quantization baselines.
3. Measuring "Learnability"
The method provides a new metric for data quality. By measuring how many bits it takes to "teach" a model a dataset, the authors found that text is rich in learnable structure, whereas images contain vast amounts of random, unlearnable information (entropy).
Critical Analysis & Conclusion
Takeaways
Requential Coding suggests that the current path to AGI via scaling is theoretically sound. It confirms that we aren't just building massive lookup tables; we are discovering increasingly simple, compressible representations of reality.
Limitations
- Computation: The encoding process is currently expensive, requiring the generation of many proposals.
- Lossy vs. Lossless: It technically compresses a "student" that is a distilled version of the teacher, not the original teacher itself.
Future Outlook: If we can optimize the REC process, Requential Coding could become the standard for not just understanding LLMs, but potentially for ultra-efficient model transmission and deployment.