Most teams pay frontier-model prices for work that does not need a frontier model. The way to cut AI API costs is not a cheaper vendor, it is deciding which steps in a pipeline require judgment and which are just volume.
Last week we tested that on our own infrastructure. A server we already run, mostly for databases, took over one step of a pipeline that had been billing on every call.
The short version
- Mixture-of-experts models changed the hardware math: Qwen3-30B-A3B activates ~3B parameters per token, so it runs on CPU alone at 14.5 tokens per second.
- We moved one step, topic labelling, onto an existing server. The scoring step, where a wrong answer is expensive, stayed on the API.
- The split that keeps holding: cheap local model for volume, frontier model for judgment. Most pipelines have far more volume than judgment.
- The local model is an optimisation, not a dependency: on timeout the step falls back to the API and nobody notices.
Why a 30B model runs on a machine with no usable GPU
The server has a good CPU, 64GB of RAM, and a GPU old enough that current CUDA builds will not talk to it. By the usual rule of thumb that makes it useless for running models locally.
That rule assumes a dense model, where every parameter is involved in producing every token. Mixture-of-experts models break the assumption: Qwen3-30B-A3B holds 30B parameters but activates roughly 3B of them per token.
The practical result is 14.5 tokens per second on CPU alone, about 18GB on disk and roughly 20GB of RAM while the model is loaded. That is slow for a chat interface and perfectly adequate for a queue of small classification jobs that nobody is watching in real time.
The split that cuts AI API costs: volume downstairs, judgment upstairs
We did not move the pipeline. We moved one step of it.
The pipeline runs every 6 hours and produces thousands of small labelling calls: read an item, assign it a topic from a known set. Each call is trivial in isolation, and the volume is what makes it expensive.
| Volume steps | Judgment steps | |
|---|---|---|
| Example | Topic labelling, field normalisation | Scoring, ranking, prioritisation |
| Call shape | Thousands of tiny calls | Fewer calls, more context each |
| Cost driver | Call count | Context size and stakes |
| Cost of a wrong answer | One mislabelled row | A bad decision acted on downstream |
| Where it runs now | Local model on our own server | Frontier model over the API |
The test that decided it was not a benchmark, it was a sample of real inputs. The local model correctly normalised a messy scraped job title to "VP, Growth Marketing", which is exactly the class of small, bounded judgment the step actually requires.
Scoring stayed on the API, and that was not a close call. A mislabelled topic is a row someone can fix; a badly scored item propagates into what the system surfaces and what people spend attention on.
“Most pipelines have far more volume than judgment. The bill usually reflects the opposite.”
How to tell a volume step from a judgment step
A step belongs on local hardware when it has all four of these properties, and a step missing any one of them is not worth the operational surface it adds.
It runs on a schedule rather than in front of a person, so latency measured in seconds instead of milliseconds is invisible to everyone.
It calls the model many times with small inputs, because the saving scales with call count rather than with model size. One expensive call a day saves nothing worth managing.
Its output is constrained: a label from a known set, a normalised string, a field extracted from a document. Constrained output can be validated mechanically, without a second model reviewing the first.
And a wrong answer is visible and cheap, correctable by a person looking at one row rather than by unwinding a decision that was already acted on.
Topic labelling matched all four. Scoring matched only the first. That was the entire selection process, and it took less time than provisioning the model did.
Why the second migration is easier than the first
Most of the cost of the first step is not the step. It is the fallback path, the monitoring, the decision about what happens on a reboot, and the argument about whether any of this belongs on hardware you own.
All of that is fixed cost. Once the routing exists and something is watching the box, moving a second step is a configuration change rather than a project, which is why the honest way to evaluate the first migration is against the two or three steps you would move after it, not against the one step in front of you.
The next experiment is another volume step, picked on the same four properties. Which one, and whether it actually moves, is not decided yet.
Failure has to be cheap
The step falls back to the API if the local model times out. That single line of routing is what makes the whole thing safe to run.
Without it, a memory-pressured server or an unloaded model becomes an outage in a client-facing pipeline, and the savings on a labelling step are nowhere near worth that trade. With it, a local failure costs money instead of correctness.
Pro tip
The honest limit: this is not "self-host your AI"
Moving one scheduled step to a local model is a narrow optimisation, not a strategy for running your AI in-house.
The model is slower than the API by a wide margin, it is unsuitable for anything interactive, and the quality gap on genuinely hard reasoning is real rather than a rounding error. We did not benchmark our way to that conclusion; we routed around it by leaving the hard work where it was.
There is also an operational cost. A model on your own server is a thing that can run out of memory, sit unloaded after a reboot, or quietly degrade while the scheduler reports success. If a company has no one who notices that, the API bill is the cheaper option and it is not close.
What to do with this on Monday
Take one pipeline and split its model calls into two lists: the ones where a wrong answer is one bad row, and the ones where a wrong answer is a bad decision.
If the first list is where most of your call volume sits, you have a routing problem rather than a pricing problem, and it is fixable this quarter without changing vendors or rewriting the pipeline.
Key takeaways
- Mixture-of-experts models made CPU-only inference practical: a 30B model can activate ~3B parameters per token and run without a GPU.
- Split model calls by the cost of a wrong answer, not by how impressive the task sounds.
- Move scheduled, high-volume, constrained-output steps first; keep anything a person acts on directly on the frontier model.
- Write the API fallback before the migration so a local failure costs money instead of correctness.
- A local model adds an operations surface. If nobody is watching it, the API bill is the cheaper choice.
Related reading: the same separation-of-roles thinking shows up in why nothing should grade its own output, and the MX record check on 80,590 domains is another case where the free, boring step beat the interesting one. If you want this kind of routing decision made inside your operations rather than left to a default, that is the work we do inside companies.
Common questions
Can you run a 30B parameter model without a GPU?
Yes, if it is a mixture-of-experts model. Qwen3-30B-A3B activates about 3B parameters per token, so it produced 14.5 tokens per second on CPU alone on a server whose GPU was too old for current CUDA builds.
Which AI steps should move to a local model and which should stay on an API?
Move the high-volume, low-judgment steps: classification, labelling, normalisation, extraction of a known field. Keep the steps where a wrong answer is expensive and hard to spot, such as scoring, ranking, and anything a person will act on directly.
What happens if the local model is slow or down?
Route around it. Our labelling step falls back to the API on timeout, so a local failure costs money rather than correctness. A local model is a cost optimisation, not a dependency you want to be blocked by.