How much can you actually shrink a neural network without wrecking performance?
The short answer: a lot. A 2025 study on language models found that a combined knowledge distillation approach cut the number of trainable parameters by 99% compared to full fine-tuning, while still achieving 97% of the original model's answer quality (measured by ROUGE-L and Perplexity scores) [1]. That means a model that originally required, say, 100 GB of memory could be squeezed down to roughly 1 GB of trainable parameters, with only a 3% drop in how well it answers questions.
For image-classification models, the compression is also substantial. A 2022 analysis of MobileNetV1 showed that widthwise compression (making each layer narrower) achieved a 42.27% compression rate, and layerwise compression (removing entire layers) hit 32.42% [4]. When knowledge distillation was applied to these compressed models, accuracy jumped by over 4.71% for the widthwise version, meaning distillation didn't just preserve performance—it actively improved it compared to training the smaller model from scratch [4].
What are the catches? When does distillation fall short?
Knowledge distillation isn't a magic bullet—it has real limitations. The same 2025 language-model study noted that standard distillation methods suffer from "inaccurate knowledge transfer, long learning process, [and] accumulation of errors in long sequences" [1]. In other words, if you're working with very long documents or conversations, a simple teacher-student setup can gradually drift off course. The researchers had to combine two advanced techniques—selective teacher intervention and low-rank adaptation—to fix this, and even then the student model only reached 98% of the quality of a low-rank adaptation method alone [1].
Another catch: the compression rate isn't uniform across all layers. A 2025 paper introduced "counterclockwise block-by-block" distillation, which assigns higher compression rates to deeper layers of the network [2]. This suggests that shallow layers (which handle basic features) may need to stay larger to preserve accuracy, while deeper layers (which handle more abstract patterns) can be squeezed harder. So you can't just shrink everything equally and expect good results.
Finally, the choice of what knowledge to transfer matters a lot. A 2022 study on graph neural networks found that preserving the global structure of how the teacher model organizes data (using contrastive learning) consistently outperformed older methods that only preserved local connections [3]. The right distillation objective can make or break the compression.
Which distillation method works best for your situation?
There's no single best method—it depends on your task and constraints. For language models handling long sequences, the 2025 study recommends combining selective teacher intervention (where the teacher steps in to correct the student when their predictions diverge too much) with low-rank adaptation (replacing large weight matrices with smaller, low-rank ones) [1]. This combo reduced GPU memory usage by 75% and cut inference time by 30% compared to full fine-tuning, while keeping quality high [1].
For image classification, the 2022 analysis suggests that widthwise compression (making each layer narrower) responds better to distillation than layerwise compression (removing entire layers), because the narrower model still has all the layers to learn from [4]. If you're working with graph data (like social networks or molecular structures), a 2022 study recommends using contrastive learning to align the student's internal representations with the teacher's, as this preserves both local and global relationships better than older methods [3].
A 2021 survey of the entire field confirms that distillation is a mature and widely used technique, but it also notes that challenges remain—especially around choosing the right teacher-student architecture and training scheme [5]. The bottom line: start with a method that matches your data type (text, image, graph), then test a combined approach if you need extreme compression.
Sources used in this answer
Optimizing knowledge distillation models for language models
Combining selective teacher intervention and low-rank adaptation reduced trainable parameters by 99% while retaining 97% of full fine-tuning quality on long sequences, with 75% less GPU memory and 30% faster inference.
Counterclockwise block-by-block knowledge distillation for neural network compression
Counterclockwise block-by-block distillation, which assigns higher compression rates to deeper network layers, improved distillation performance on Tiny-ImageNet-200 and CIFAR-10.
On Representation Knowledge Distillation for Graph Neural Networks
Graph Contrastive Representation Distillation (G-CRD) consistently boosted lightweight graph neural network performance across 4 datasets and 14 architectures, outperforming local-structure-preserving methods.
Analysis of Model Compression Using Knowledge Distillation
Widthwise compression of MobileNetV1 achieved 42.27% compression; applying knowledge distillation improved accuracy by over 4.71% compared to training the compressed model without distillation.
Knowledge Distillation: A Survey
A comprehensive survey confirms knowledge distillation is an effective model compression technique, but challenges remain in teacher-student architecture design and training schemes.