When people ask why a self-hosted AI model is so slow, the answer is usually something other than its tokens per second. Ours took 223 minutes to answer one question last week, and when we pulled the timeline apart, the model's own speed was the smallest of three stacked causes.
The system is an internal assistant we run on our own hardware: it reads mail, prepares briefings, and keeps notes between sessions. A question went in during the morning. The answer arrived after lunch.
The short version
- 223 minutes for one answer, on a 24-core CPU box with no GPU, running a 177B mixture-of-experts model.
- Two large models were sharing the machine. Removing one gave the survivor a single slot with its full 262k context instead of two half-sized ones.
- The session compacted its own context twice in 80 minutes, and every compaction throws away the cache, so the next step re-reads everything.
- A background job burned 95 minutes of slot time on 14 requests nobody was waiting for, because a cancelled request on this server keeps running to completion.
Cause one: two large models, one machine
The box is a 24-core CPU server with no usable GPU. It was hosting Qwen3.8-Flash-Next, a 177B-parameter mixture-of-experts model, 90 GB on disk, alongside a 320B-parameter GLM model.
Sharing does not work the way it does with web services. Each model holds its weights in memory and the available context has to be divided between them, so the machine was running two slots of 131k context each rather than one slot of the full 262k.
Removing the larger model from rotation left the remaining one generating at 7.4 tokens per second and reading prompts at roughly 50. Those are not fast numbers. They are honest ones, and they are what the hardware actually does when nothing else is competing for it.
Cause two: the session kept re-reading its own memory
The assistant had gone looking for a document and pulled it in 12,000-token chunks. That is the correct behaviour for reading a large file and the wrong behaviour for a conversation that has to stay in a context window.
Context reached 116k at 15:02 and compacted down to 42k. It grew back to 97k by 16:20 and compacted again. Each compaction throws away the cached state, which means the next step re-reads what it just summarised, at 50 tokens per second.
Where the 223 minutes went
15:02
First compaction
Context hit 116k after the document was pulled in 12k-token chunks, and was cut back to 42k.
15:00 to 16:50
Background jobs queueing
14 timed-out requests kept generating, occupying 95 minutes of slot time.
16:20
Second compaction
Context back up to 97k, cut again, cache discarded again.
16:50
Still unanswered
The question put in that morning had produced nothing. The autopsy started here.
Put the two numbers together and the cost stops being abstract. Reading 42,000 tokens of freshly compacted context back in at roughly 50 tokens per second is about 14 minutes of the machine doing nothing but catching up on what it had already read.
That happened twice. The compaction itself is cheap; the re-read it forces is not, and on a hosted model it is invisible because the throughput hides it.
This is the failure that generalises best to a company's own internal AI work. A system that manages its memory badly does not fail loudly; it spends its time reorganising instead of answering, and from the outside it just looks slow.
Cause three: cancelled requests that were not cancelled
The third cause was the one nobody would guess. A background job writes the assistant's notes, and it calls the same local model to do it. Those calls time out after 5 minutes and retry three times.
The timeout only ends the waiting, not the work. On llama-server, the inference server behind this box, a non-streaming request keeps generating after the client hangs up, so every abandoned call carried on producing an answer that would never be read, holding a slot the whole time.
Between 15:00 and 16:50 that came to 95 minutes of slot time across 14 requests nobody was waiting for. The foreground question was queued behind a backlog of work that had already been given up on.
The one-line difference between a cancelled request and an abandoned one
This was not read in a changelog, it was tested live against the router: a non-streaming request keeps running after the client disconnects, while a streaming one is cancelled the moment the socket closes. The server has no other signal that the caller has gone.
So the fix was to make every background call stream, whether or not anything consumes the stream token by token. A timeout now actually stops the work rather than just stopping the waiting.
The second half of the fix is a pre-check: both background pollers now ask the inference server whether a slot is free before they queue anything. A background job that cannot get a slot should skip its turn, not join a queue in front of a person.
Neither change required touching the model, the prompts or the hardware. Both were in the plumbing between them.
The honest limit: none of this made it fast
After all three fixes the assistant answers in minutes instead of hours, and 7.4 tokens per second is still 7.4 tokens per second. Removing waste is not the same as buying speed.
What the exercise actually settled was a scope question. This machine is good at always-on background work: triage, grading, summarising overnight, deciding what deserves a human's attention in the morning. Slowness is free when nobody is waiting.
Anything a person is sitting in front of goes to a hosted model. That is the same split we described when moving volume work onto our own server, now with the failure mode attached: the split holds, and the moment you let a foreground task cross the line you get 223 minutes.
A GPU changes this arithmetic entirely, and we do not have one on this box. Read the numbers here as the shape of the problem, not as a benchmark.
If you are debugging your own box, the order that would have saved us the afternoon is: count how many models hold weights in memory, then count how many callers can queue against them, then check whether a cancelled call actually stops generating, and only then look at tokens per second. The first three are configuration questions with yes or no answers. The fourth is the one everybody starts with and the only one you cannot fix without buying hardware.
The reason that order works is that the first three produce the same symptom as the fourth. A queue you cannot see and a model that is genuinely slow both look like waiting.
Key takeaways
- Before blaming the model's speed, check whether anything else is competing for the same inference slots.
- Two models on one machine split the context budget as well as the memory, so consolidating onto one can be a speed improvement.
- Every context compaction discards the cache and forces a re-read, so a session that manages memory badly spends its time re-reading rather than answering.
- On a local inference server, a timed-out non-streaming request keeps generating. Make background calls stream so a cancellation actually cancels.
- Keep foreground work off a CPU-only local model: it is well suited to background jobs where nobody is waiting.
If you are standing up internal AI infrastructure, the useful move is to instrument the queue before tuning the model, because the first bottleneck is almost never the one on the spec sheet. That is also the argument in why AI adoption stalls on operations rather than on capability, and it is the kind of thing we look for first when we work inside a company's systems.
Common questions
Why is a self-hosted AI model so slow?
Usually not because of the model's tokens per second. In our case the three causes were two large models sharing one set of slots, a session that kept compacting and re-reading its own context, and background jobs whose abandoned requests kept running to completion.
Does cancelling a request free up a local inference slot?
Only if the request streams. On llama-server a non-streaming request keeps running after the client hangs up, so a timed-out background call still occupies a slot until it finishes generating an answer nobody will read.
What should run on a local model and what should not?
Anything a person is waiting on should not. Local inference on CPU is well suited to background triage, grading and summarising overnight, where slowness costs nothing.