Something odd happened on the way to bigger language models: the useful ones got smaller. While headlines chased systems trained on unimaginable piles of text, engineers quietly began shipping modest models that fit on a phone, answer in a fraction of a second, and cost almost nothing to run.

You have probably met one without noticing. The sentence-completion on your phone's keyboard. The tool that turns a rambling customer call into three tidy bullet points. The scanner in a stockroom that reads a part number off a battered label. These are small language models, and they are spreading for a simple reason: for a great many jobs, "good enough at one thing, right now, on cheap hardware" beats "brilliant at everything, expensive, and far away."

What "Small" Actually Means

A language model is software that has learned statistical patterns in text. Its size is usually described in parameters — the internal settings it adjusts while learning. A frontier system may have hundreds of billions of them. A small model might have a few hundred million to a few billion.

That gap sounds like a quality gap, and partly it is. But parameters are also weight: memory, electricity, chips, milliseconds. A model with a billion parameters can run on a modern phone or a modest server. A model a hundred times larger needs a data centre and a queue.

The better analogy is a reference library versus a pocket guide. The library answers almost anything, eventually, if you have the time and the budget. The pocket guide was written for one job — identifying birds, say — and fits in your jacket.

Why Companies Are Adopting Them

Four pressures pushed small models from curiosity to default.

  • Cost per request. A narrow task run on a small model can cost a fraction of the same task on a giant one. At millions of requests a day, the difference shows up on the invoice.
  • Speed. Answers arrive in milliseconds rather than seconds. That changes what you can build: live captions, instant filtering, a suggestion that appears before you finish typing.
  • Privacy and control. A model running on the device can process a document without the document leaving the room. For hospitals, law firms, banks and public bodies, that is often the whole argument.
  • Availability. On-device models keep working on a plane, in a basement, or on a factory floor with patchy Wi-Fi.

There is a technical trick behind much of this too. Instead of training a small model from scratch, teams take a large one and teach it to imitate — a process called distillation. The big model does the expensive thinking once; the small model learns the pattern and handles the daily traffic.

Where the Big Systems Still Win

Small models are specialists, and specialists are fragile outside their specialty. Ask one to plan a five-step project, switch between three languages, reason through an unfamiliar legal question, or write in a voice it has never seen, and the gap becomes obvious.

Big systems carry broader world knowledge and handle messy, open-ended requests. They also cope better when the input is strange: a badly scanned invoice, a rambling email, a question phrased in a way nobody anticipated.

The rule most teams settle on: use the smallest model that clears your accuracy bar for a defined task, and keep a larger one available for the odd jobs that don't fit. Route by difficulty, not by habit.

What It Looks Like in Practice

The pattern repeats across industries. A model is asked to do one bounded thing, over and over, and a person checks the output when it matters.

  • Insurance: sorting claims into categories before a human reviews them.
  • Retail: reading shelf photos for out-of-stock items.
  • Fieldwork: turning a technician's voice notes into a fixed template.
  • Customer support: drafting a reply for an agent to edit, not send.

Notice how many of these keep a person in the loop. That is deliberate. Small models are cheap enough to run everywhere, which also means they can be wrong everywhere — quietly, at scale, and in ways that look plausible.

The Trade-offs Nobody Puts on the Slide

A small model has less room to be graceful. When it doesn't know, it may still answer confidently. It follows instructions less reliably, especially long or contradictory ones. It drifts when your data changes: new product names, new slang, a new invoice format.

Then there is maintenance. A model you fine-tune is a model you have to watch. Teams that do this well keep a test set of real examples, re-run it after every change, and track where the output goes wrong. Teams that don't hear about problems from customers first.

If the model touches decisions about money, health, employment or legal matters, treat its output as a draft and have a qualified professional review it. Cheap text generation is not a licence to skip judgement.

How to Tell What You're Dealing With

When a vendor or an internal team pitches you a model, five questions cut through most of the noise.

  1. Does it run on the device, on your servers, or on someone else's? That answers most privacy questions before you ask them.
  2. What exactly is it doing — classifying, extracting, summarizing, drafting? A vague answer usually means the task hasn't been defined.
  3. How accurate is it on your examples, not on a demo? Ask for results on a sample of your own data.
  4. What happens when it is wrong? Is there a human step, a fallback, a way to escalate?
  5. Who maintains it when the world changes?

A Practical Starting Point

You don't need a strategy document to try this. Pick one repetitive text task your team already does by hand — tagging tickets, summarizing calls, tidying addresses. Collect fifty real examples. Write down what a correct answer looks like, including the awkward edge cases. Then test the smallest option available and see how close it gets.

If it clears the bar, keep it narrow, measure it monthly, and leave a person in the loop for anything that leaves your building. If it doesn't, the honest conclusion may be that this job needs a bigger system, a better-defined task, or a human. All three are legitimate answers.

The shift toward smaller models isn't a retreat from ambition. It is a sign that the technology has matured enough to be judged like any other tool: by what it costs, where it runs, and whether it actually does the job.

Photo: Júlio Riccó / Pexels