Why can't small language models handle multi-step reasoning?
The biggest evidence gap for small language models is their limited ability to perform complex, multi-step reasoning. A systematic review of 68 studies found that on benchmarks requiring deep reasoning, SLMs like CodeT5-Large achieve only 57.58% accuracy, while large models like GPT-4 reach 92.13% [1]. This isn't just a minor shortfall—it's a fundamental limitation of having fewer parameters (often under 10 billion), which makes it harder to maintain logical chains across multiple steps.
But this gap isn't fixed. Targeted training with high-quality reasoning data can dramatically close it. In the medical domain, researchers created a family of SLMs called Meerkat by fine-tuning 7B- and 8B-parameter models on 441,000 training examples, including chain-of-thought reasoning paths extracted from 18 medical textbooks. The result: Meerkat-7B outperformed its counterparts by 22.3% across six exam datasets, and on the NEJM Case Challenge, it scored 16 (beating the human average of 13.7) [2]. This shows that with the right data, SLMs can learn sophisticated reasoning for specific domains.
Another approach, called EvidenceMap, teaches a tiny 66M-parameter model to explicitly analyze evidence—evaluating support, logical connections, and summarizing content—before feeding that analysis to a 3B-parameter generative model. This method outperformed a standard retrieval-augmented generation (RAG) system using an 8B LLM by 19.9% in reference-based quality and 5.7% in accuracy [3]. The key insight: teaching SLMs how to reason about evidence, rather than just memorizing facts, can partially overcome their parameter limitations.
What other capabilities are missing in small language models?
Beyond reasoning, small language models have a clear weakness in in-context learning—the ability to learn from examples given in the prompt without retraining. A comprehensive study of over 60 publicly available SLMs (including Microsoft Phi and Google Gemma) found that while state-of-the-art SLMs now outperform 7B-parameter models on general tasks, their in-context learning remains limited [4]. This means they struggle to adapt on the fly to new tasks or instructions without fine-tuning, which is a key capability for flexible deployment.
Efficiency is another area where SLMs have significant room for improvement. The same study identified optimization opportunities like dynamic task-specific routing (choosing the right model for each task), model-hardware co-design, and vocabulary/KV cache compression [4]. These techniques could make SLMs much faster and more practical for real-world use. For example, combining Low-Rank Adaptation (LoRA) with quantization can reduce latency from 500ms to 50ms on mobile devices—a 10x speedup—while maintaining over 95% efficiency gains for privacy-preserving AI [1].
There's also a gap in planning capabilities for tasks like smart home device control. While large language models can handle abstract instructions and dynamic home configurations zero-shot, small models initially show limited ability. However, researchers have shown that by generating synthetic training data using a large language model and then fine-tuning the small model on that data, SLM performance for device control can be significantly improved [5]. This suggests that the planning gap is not inherent but stems from a lack of appropriate training data.
Can better training data really make up for smaller model size?
Yes—and the evidence strongly points to data quality, not quantity, as the key. The systematic review found that well-curated domain corpora of just 1–50 GB can offset smaller model capacity, with data quality outweighing volume. In low-resource settings, addressing data imbalance yielded 10–25% performance gains [1]. This is a crucial finding because it means organizations don't need massive datasets to build effective SLMs—they need the right data.
The Meerkat study provides a concrete example: by using high-quality chain-of-thought reasoning paths from medical textbooks (not just raw text), they created a training dataset that boosted SLM performance by over 20% on medical exams [2]. Similarly, the EvidenceMap approach succeeded by teaching a tiny model to analyze evidence in a structured way—supportive evaluation, logical correlation, and content summarization—rather than just feeding it more data [3]. These results converge on the same conclusion: for SLMs, how you train matters more than how much you train.
However, there's a catch. While data quality can compensate for model size in specialized domains (like medicine or law), the evidence shows that for broad, general-purpose reasoning, large models still hold a significant advantage [1]. The gap is narrowed, not closed. So the practical takeaway is: if your task is narrow and well-defined, a well-trained SLM can match or even exceed a general-purpose LLM at a fraction of the cost. But for open-ended, multi-domain reasoning, you'll still need the bigger model.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2024 to 2026, 5 from 2024 or later, 1 in Q1 journals — selected as the most relevant from 5 studies that passed quality screening, drawn from 65 papers retrieved from a database of over 500 million.
Sources used in this answer
Small Language Models: A Systematic Review of Computational Trade‐Offs, Privacy Advantages and Deployment in Intelligent Systems
A systematic review of 68 studies found that fine-tuned SLMs achieve 85–90% accuracy on domain-specific tasks at 10–25% of LLM computational costs, but complex reasoning remains a major gap (e.g., CodeT5-Large 57.58% vs. GPT-4 92.13%). Data quality outweighs volume, with imbalance mitigation yielding 10–25% gains in low-resource settings.
Small language models learn enhanced reasoning skills from medical textbooks
In a study using 441K training examples from medical textbooks, Meerkat-7B outperformed counterparts by 22.3% across six exam datasets and scored 16 on the NEJM Case Challenge, surpassing the human average of 13.7, demonstrating that targeted chain-of-thought training can dramatically improve SLM reasoning.
EvidenceMap: Learning evidence analysis to unleash the power of small language models for biomedical question answering.
EvidenceMap, which teaches a 66M-parameter model to analyze evidence (support, logic, summarization) before feeding to a 3B generative model, outperformed a RAG system with an 8B LLM by 19.9% in reference-based quality and 5.7% in accuracy, showing structured evidence analysis can compensate for model size.
Demystifying Small Language Models for Edge Deployment
A comprehensive study of over 60 SLMs (e.g., Phi, Gemma) found that state-of-the-art SLMs now outperform 7B models on general tasks but have limited in-context learning and significant optimization potential in dynamic routing, model-hardware co-design, and compression.
Enabling Device Control Planning Capabilities of Small Language Model
Researchers showed that by generating synthetic device control planning data using a large language model and fine-tuning a small model on that data, SLM performance for smart home device control can be significantly improved, indicating the planning gap is due to lack of appropriate training data.