Nearly every architecture we audit contains the same inefficiency: an expensive frontier model is called for tasks a small model handles just as well.
It is not a judgement failure. It is that things were built that way when only one option existed.
The economics
Small models running on your server or the user’s device have a property no API can match: they cost exactly zero per query after the initial download.
Against the budget tiers of the API providers, at low figures per million tokens, that looks like a small difference. At scale it stops looking small. A process classifying 200,000 documents a month accrues cost linearly forever; the same process on a self-hosted model has a fixed infrastructure cost and a marginal of zero.
And that is with inference prices on the floor. The argument is not that the API is expensive: it is that for a certain class of task you are paying for capability you don’t use.
What they are actually good for
A small model does not reason like a frontier one, and pretending otherwise is the fastest way to fail. But there is a very wide band of enterprise work that needs no deep reasoning:
- Classifying. This ticket is billing, this email is a complaint, this document is a delivery note.
- Extracting. Pulling fifteen fields off an invoice, an order number from an email, dates from a contract.
- Routing. Deciding which team, which workflow or which large model should handle each case.
- Templated drafting. Structured replies where format matters more than creativity.
- Filtering and prioritising. Separating urgent from what can wait.
At a mid-sized company, this is usually the bulk of the volume.
The pattern we recommend
A router in front. A cheap small model decides whether the case is routine or complex. It handles the routine; the complex escalates to the frontier model.
Advantages beyond cost:
- Latency. A local model answers in milliseconds, with no network round trip.
- Availability. If your provider has an outage, 80% of your traffic keeps working.
- Privacy. Data resolved by the local model never leaves your network, which settles half a dozen legal objections at once.
- Independence. You stop being exposed to price or policy changes on the part that carries most volume.
What to budget for
This is not free, only cheap:
- Hardware, or a GPU instance, and someone who can maintain it.
- An evaluation set to prove the small model performs on your cases. Without it, this is an act of faith.
- An upgrade plan. Open models improve every few months.
- A clear escalation path: what happens when the router gets it wrong and sends up something it shouldn’t, or vice versa.
How to start
Take the highest-volume process you have in production. Pull 200 real cases. Run them against the large model you use today and against a small open model. Compare.
If the quality difference is small, you have just found the saving. If it is large, you have learned something useful about your use case for the price of an afternoon.