Small language model
Small language models (SLM) or compact language models are artificial intelligence language models designed for human natural language processing including language and text generation. Small language models typically range have less than forty billion parameters.[1] This make them feasible to train or host entirely on consumer electronics such as personal computers, laptops, or smart devices.[2][3][4][5][6]
A large language model typically contains hundreds of billions of training parameters, with some models exceeding a trillion parameters. This enables the model to encode vast amounts of information, improving the generalizability and accuracy of its outputs. However, training and hosting such models demands enormous computational resources, rendering it infeasible for an individual to do so using a single computer and graphics processing unit.
Most contemporary small language models use the same architecture as a large language model, but with a smaller parameter count and sometimes lower arithmetic precision. Similar to LLMs, SLMs are optimized via knowledge distillation, pruning, and quantization.
Models
[edit]Notable families with some models under ten billion parameters include:[3][1]
- Qwen by Alibaba Cloud
- Gemma by Google DeepMind
- Granite by IBM
- Nemotron by Nvidia
- Muse Glimmer by Meta Superintelligence Labs
- Mistral Small by Mistral AI
- SmolLM by Hugging Face family of SMLs, including SmolLM2 models with 135 million, 360 million, and 1.7 billion parameters.[7]
Language model with small pre-training dataset
[edit]Traditional AI language systems need enormous computers and vast amounts of data.
Pre-training matters, even tiny models show significant performance improvements when pre-trained performance increases with larger pre-training datasets. Classification accuracy improves when pre-training and test datasets share similar tokens.
Shallow architectures can replicate deep model performance through collaborative learning.[8]
See also
[edit]References
[edit]- 1 2 "Small Open Source Models | Artificial Analysis". artificialanalysis.ai. Retrieved 2026-08-17.
- ↑ Rina Diane Caballar (31 October 2024). "What are small language models?". IBM.
- 1 2 John Johnson (25 February 2025). "Small Language Models (SLM): A Comprehensive Overview". Huggingface.
- ↑ Kate Whiting. "What is a small language model and how can businesses leverage this AI tool?". The World Economic Forum.
- ↑ "SLM (Small Language Model) with your Data". Microsoft. 11 July 2024.
- ↑ Ciaramella, Alberto; Ciaramella, Marco (2024). Introduction to Artificial Intelligence: from data analysis to generative AI. Intellisemantic Editions. ISBN 9788894787603.
- ↑ Wiggers, Kyle (July 16, 2024). "Hugging Face's SmolLM models bring powerful AI to your phone, no cloud required". VentureBeat. Retrieved August 10, 2026.
- ↑ Gross, Ronit D.; Tzach, Yarden; Halevi, Tal; Koresh, Ella; Kanter, Ido (2026). "Tiny language models". Physica A: Statistical Mechanics and Its Applications. 681 131102. arXiv:2507.14871. doi:10.1016/j.physa.2025.131102.
 | This statistics-related article is a stub. You can help Wikipedia by adding missing information. |
 | This article about natural language processing is a stub. You can help Wikipedia by adding missing information. |
