AI Explainedby Milad Fahmy

AI Explained

Small language models (SLMs)

Bigger isn't always better. Small language models with a few billion parameters are catching up, and they're cheaper, faster and can run on your phone.

Milad FahmyNov 3, 20262 min read

On this page
  1. What counts as small?
  2. How good can a small model be?
  3. Why they got so good
  4. SLMs and agents
  5. Other advantages
  6. The catch
  7. Where to start

Bigger isn't always better. Small language models are catching up fast, and for many jobs they're the smarter choice.

What counts as small?

A small language model, or SLM, has a few billion parameters or fewer. The biggest frontier models are many times larger. That makes SLMs cheaper and faster to run, and small enough for a laptop or even a phone.

How good can a small model be?

Microsoft's Phi models are a good example of how far they've come:

  • phi-1 (2023) had just 1.3 billion parameters and scored 50.6% on HumanEval, a coding benchmark.
  • phi-3-mini (2024), with 3.8 billion parameters, rivaled Mixtral 8x7B and GPT-3.5 on benchmarks, scoring 69% on MMLU (a broad knowledge test), while being small enough to run on a phone.

Why they got so good

The phi-1 paper's title says it: Textbooks Are All You Need. The trick was data quality: "textbook quality" code and exercises, not just anything from the web. Add more training, and often distillation from bigger models, and small models punch far above their size.

SLMs and agents

A 2025 NVIDIA position paper argues small models are "sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems".

Its idea is to mix models: small ones handle the routine, repetitive steps (parse this, classify that, call this tool), and a big one steps in for open-ended conversation and hard reasoning.

Other advantages

  • Privacy: run one on your device, and your data stays there.
  • Cheap fine-tuning: make one great at your exact task.
  • Speed: fast answers, even without a big GPU.

The catch

Small models know less about the world, and struggle more with long, hard reasoning. Benchmarks aren't your app, so test a small model on your own tasks before you switch.

Where to start

Popular families include Phi, Gemma, Llama and Qwen, with versions from under a billion to a few billion parameters. A good path: prove the idea with a big model, then try a small one, and keep it if the quality holds.

Small model, big savings.

Sources

#small language models#slm#phi#local ai#ai agents

Watch