A 430 GB Model Runs at 0.027 Tokens a Second, and the CPU Is Not the Problem

Abhishek DashUpdated November 11, 202611 min read
A lone tall server tower in a vast dark data center hall with a vertical amber light bar rising like a thermometer, blue data streams flowing across the floor, and a CPU cooler spinning behind a glass panel.

In short

On a 12 vCPU Xeon E5-2620 v3 with 47 GiB of RAM and no GPU, Colibri v2.1.0 ran eight MoE models from 19.4 GB to 429.3 GB. Throughput fell from 0.646 tokens a second on Qwen3.6-35B-A3B to 0.027 on GLM-5.2 744B, a 24 times gap, with a log-log slope of -1.10. Expert disk reads took 65 to 96 percent of request time on every model, so this is a batch machine.

Key takeaways

  • The best valid result was 0.646 tokens a second on Qwen3.6-35B-A3B, at 4.1 minutes for 128 tokens.
  • The worst was 0.027 tokens a second on GLM-5.2 744B, which took a median 28.7 minutes for a 32 token reply.
  • A twenty times increase in on-disk size cost a 24 times drop in throughput, a log-log slope of -1.10 with r of -0.92 across seven points.
  • Expert disk reads accounted for 65 to 96 percent of engine-reported time on every model; expert matmul accounted for single digits to low teens.
  • Qwen3-Coder-30B returned 0 tokens on all four rotating requests, so its primary metric is missing while its warm 0.689 and cold 0.717 figures stand.
  • MiMo-V2.6 logged 0.07 tokens a second end-to-end; on the decode-only basis used for the other models it is 0.118 and outranks Qwen3.8-Flash-Next.

A 23.0 GB model ran at 0.646 tokens a second on this machine. A 429.3 GB model, on the same machine with the same engine, ran at 0.027, one token every 36.8 seconds. Twenty times the bytes on disk bought a 24 times slower machine.

The number that explains the spread is not either endpoint. It is 65 to 96 percent: the share of engine-accounted request time colibri v2.1.0 attributes to reading routed expert weights off disk, on every one of the eight text models. Expert matmul, the arithmetic the CPU actually does, sits in the single digits to low teens. The CPU is not the constraint. On these checkpoints it is barely in the room.

The test machine is a KVM guest with 12 vCPU of Intel Xeon E5-2620 v3 at 2.40 GHz, 47 GiB of RAM, no swap, one 1.5 TB virtio disk that also carries the root partition, and no GPU. AVX2 and FMA, no AVX-512. The engine is colibri v2.1.0, commit 0a5504a, built for the native architecture, CPU backend only.

What ran, and in what order

Eight text checkpoints and one image model, one at a time. Each was downloaded, benchmarked, and deleted before the next began, which is why fetching and deleting weights dominated total wall clock for the large models.

Each text model went through three phases inside one long lived engine process. Cold evicts the page cache and takes the first request. Warm-identical repeats the same prompt with caches retained, a one-sample upper bound. Rotating runs four different prompts with the engine alive, and its median is the primary result.

Output caps were 128 new tokens for the two 20 GB models, 64 for the mid sized ones, 32 for the three largest. Decode rates survive that; total request times do not. Speed is re-derived here as completion tokens divided by request seconds minus time to first token, because the engine's logs do not all use that definition. The one that does not is handled below.

The leaderboard

Model Size GB Rotating tok/s TTFT s Median request
Qwen3.6-35B-A3B 23.0 0.646 47 4.1 min
Qwen3-Coder-30B-A3B 19.4 failed - 59 s, 0 tokens
DeepSeek V4 Flash REAP 150B 84.7 0.322 68 4.5 min
DeepSeek V4 Flash 284B 166.9 0.288 79 3.1 min
MiMo-V2.6 Flash 309B 171.8 0.118 180 7.4 min
Qwen3.8-Flash-Next 185.6 0.090 174 15.0 min
GLM-5.3 744B 419.3 0.030 504 25.9 min
GLM-5.2 744B 429.3 0.027 559 28.7 min

Spread between runs was tight, with a logged sigma of 0.000 to 0.019 tokens a second, so the ordering is robust. The exception is the bottom pair: 0.030 and 0.027 are not separable, and the report says so rather than ranking them.

Two things matter more than the table. One is what 0.646 tokens a second means in practice, about 1.5 seconds per token and 4.1 minutes for a 128 token answer, which is the best this box offers. The other is that time to first token is not a rounding error. GLM-5.2 spent a median 559 seconds reaching the first token on a prompt of 29 to 37 tokens. For anything interactive, that is the number a user feels.

Where the time goes

The phase breakdown is identical in shape across eight checkpoints from five architectures. Expert disk I/O is the largest slice everywhere; expert matmul is a rounding error beside it. On GLM-5.3 the disk counter logged 5,884 seconds inside a request whose median wall time was about 1,700, which is the tell that these counters sum across loader threads rather than sit on a stopwatch. Read proportions, not seconds. What the proportions say is that the arithmetic is waiting on storage, and that more cores or clock would move almost nothing.

Memory is the other half of the picture. The engine keeps dense weights resident and streams routed experts, which is why Qwen3.6 holds 3.2 GB of RSS for a 23 GB model while the DeepSeek models hold about 36 GB and the GLMs 34.5 to 36.1. GLM leaves roughly 94 percent of experts on disk, and both 744B checkpoints auto-lowered their cache cap to fit: 16 down to 15 for GLM-5.3 at a projected 45.7 GB peak, 16 down to 13 for GLM-5.2 at 46.0. With no swap configured, a little more pressure would have meant an OOM kill rather than a slower run. Qwen3.8 logged that 164 experts per layer would fit in 38.8 GB at 4.69 MiB each, but ran with a cap of 16.

Speed falls faster than size grows

Throughput against on-disk size on log-log axes gives a slope of -1.10 and r of -0.92 across seven points. Twenty times the size costs 24 times the speed. The two exponents being close in magnitude is the point: decay runs slightly faster than growth, so the largest models are worse than a naive linear trade predicts.

Seven points from five architectures is indicative, not a law, and size is a proxy. The driver is expert bytes fetched per generated token. DeepSeek V4 pulled 294 to 320 GB of direct expert reads across a whole run, roughly 0.8 GB per token on REAP and 1.5 GB on the full 284B, including prefill, the cold request and the warm one. That overstates steady state, but it is why 166.9 GB lands at 0.288 while 419.3 GB lands at 0.030.

Hit rate tells you less than it looks

DeepSeek V4 Flash REAP has the best measured hit rate in the study at 77.8 percent and runs at 0.322 tokens a second. Qwen3.6-35B-A3B has 50.3 percent and runs at 0.646, about twice as fast. A 27 point hit rate advantage loses to an architecture with cheap misses: Qwen3.6 activates 8 of 256 experts per layer at a small logged expert size.

Within a family the metric behaves. REAP at 77.8 percent beats the full 284B at 61.5 percent by 12 percent while downloading half the bytes. Across families it stops predicting anything.

The planner is the other surprise. It projected 5 to 35 percent hit rates for the large models and measured 20 to 78, because expert routing is skewed enough that a modest cache absorbs far more traffic than a uniform model predicts. The one undershoot was Qwen3.8 at 19.8 percent against a projected 26.

Cold and warm decode sit within about 25 percent of each other on all eight models, because steady state misses set the pace, not first touch loading. The warm-run gain is in prefill: on GLM the warm request reuses the KV prefix and time to first token falls from about 600 seconds to 40 or 47.

Speculation did not rescue anything. Qwen3.8's multi-token head produced 1.7 to 2.4 tokens per forward with 20 to 73 percent acceptance and still ran at 0.090, because each forward is roughly 20 seconds of disk waiting. GLM-5.2's gate chose 174 plain steps against 12 verifies, and GLM-5.3 has no multi-token head.

Two results that need a footnote

Qwen3-Coder-30B-A3B has no valid primary result. All four rotating requests returned 0 tokens after roughly 59 to 66 seconds, with a 0 percent hit rate, an all-zero profile, and no speculative forwards from turn three. Its warm 0.689 and cold 0.717 figures come from valid single requests, and 0.717 is the best cold number in the study. The rotating median does not exist. Quoting 0.00 as its speed is wrong, and quoting 0.69 as a rotating figure is equally wrong.

The likely cause is the engine rather than the model. The Coder ran on the qwen36 engine, Qwen3.6's engine, although its geometry differs: 128 experts, no shared expert, no DeltaNet. Whether that mismatch causes the persistent mode stall is untested. It is a reproducible failure worth an upstream report, and it leaves the smallest model in the study with no score.

MiMo-V2.6 Flash 309B has the opposite problem: its number is real but measured differently. The logged 0.07 tokens a second is end-to-end, completion tokens over total request time, while every other model logs decode only. Recomputed on the same basis, MiMo is 0.118, warm 0.128 and cold 0.126. That moves it above Qwen3.8-Flash-Next at 0.090, and the published ranking that puts it below is an artefact of two definitions in one table. Its 20.3 GB peak RSS is the lowest of the large models, which is why it never approached the memory ceiling.

Downloads moved faster than the table suggests

The XET downloader measured 87 to 116 MB/s across six large models, against roughly 25 MB/s for the standard library path, which is 3.5 to 4.6 times faster. The archive README quotes about 96 MB/s for XET and 25 for stdlib, and the measured range brackets it.

The effect is on provisioning, not inference. Decode speed does not care which downloader fetched the weights. What changes is how long a 429 GB checkpoint takes to arrive: 63 minutes for GLM-5.2, 81 for GLM-5.3, which is where most of the serial run time went.

The image model, and the shape of the box

Qwen-Image-2.1 ran one request: a 768 by 512 PNG at 8 steps from the prompt "a red cube on a table", returning HTTP 200 with a 573,967 byte file in 862.5 seconds, about 108 seconds per denoising step if text encoding is ignored. Peak RSS was near 13 GB and CPU averaged 732 percent of the 1,200 percent available. One sample, no repeats, no quality assessment: a timing, not a distribution. The first attempt hung for 41 minutes in foreground mode, and the log does not say what it was waiting on.

What this machine is for

Batch work, and only batch work. Qwen3.6-35B-A3B is the only checkpoint with a valid primary result above 0.5 tokens a second, and the sensible use is batch generation, evals and overnight jobs at about 1.5 seconds per token. The two 20 GB models are the only ones a person could sit in front of, and 4.1 minutes for 128 tokens is reading, not conversation. The 309B and 744B runs should be struck from any serving plan: at 0.027 to 0.118 tokens a second they are a floor for how bad disk streamed mixture-of-experts gets on this storage, not a service.

The hardware conclusion is narrower than it looks: a faster CPU would not fix this, because matmul is 1 to 12 percent of request time. The levers are storage and RAM. The report proposes an NVMe re-run and a raised cache cap on Qwen3.8, and notes that expert streaming capping throughput should transfer to faster hardware even though the numbers will not.

Where this could be wrong

One machine, one virtio disk, one kernel, and a run window of 9 to 10 November 2026. No NVMe was tested, so the claim that storage is the fix comes from the phase profile rather than a controlled comparison. The direction is well supported; the size of the gain is not measured.

The archive contains no model weights. Everything here comes from eight datapoint reports, eight engine stderr logs, a results table and a serial download log, so the runs cannot be re-scored without re-downloading hundreds of gigabytes. Throughput is derived from the engine's own per-request logs rather than an external timer, and the phase counters are summed across loader threads and therefore exceed wall clock. Read them as proportions.

At least one run is invalid. Qwen3-Coder-30B produced 0 tokens on 4 of 4 rotating requests, and its primary metric is missing rather than zero. MiMo-V2.6's logged figure used a different definition from every other row. Six engines appear in one ranking, with different loaders, kernels and quantisation, so model and engine effects are blended and the ordering is not a clean comparison of architectures. The rotating sample is four prompts, the warm sample is one, and while spread was between 0.000 and 0.019 tokens a second, nothing here supports a confidence interval. The GLM pair is explicitly not separable, and output caps differ by model size, which keeps decode comparable and makes request times not.

The archive carries no external links, so there is no source ledger beyond the report and its logs. Every figure above is traceable to a table, a per-model note or an appendix row in it, and the archive's file names are the citation.

On this page

Sources

  1. Colibri CPU-only Benchmark ReportBenchmark analysis, 2026

Frequently asked questions

Why is every model slow on this server?

The engine streams routed expert weights from disk, and it reports expert disk reads at 65 to 96 percent of accounted request time on all eight models. Expert matmul is single digits to low teens. The virtio disk, not the 12 vCPU Xeon, sets the pace for every model tested.

Does a faster CPU fix this?

No. With matmul at 1 to 12 percent of request time, a faster CPU has almost nothing to speed up. The report's recommendation is to change storage or add enough RAM to hold the hot expert set. An NVMe re-run was proposed but not done, so the size of that gain is untested.

Can any of these models be used interactively?

Only the two 20 GB Qwen3 models are close. Qwen3.6-35B-A3B produces about 1.5 seconds per token and needs 4.1 minutes for 128 tokens, which is readable but not conversational. Everything above 80 GB on disk is batch only: a 32 token GLM reply takes 26 to 29 minutes, most of it prefill.

Why does DeepSeek V4 REAP lose to Qwen3.6 despite a better hit rate?

REAP measured the best expert cache hit rate in the study at 77.8 percent, against 50.3 percent for Qwen3.6, yet ran at about half the speed. Hit rate alone does not set throughput. Qwen3.6 activates 8 of 256 experts per layer and its misses are cheap, while DeepSeek V4 pulled roughly 0.8 to 1.5 GB from disk per generated token.

What happened to Qwen3-Coder-30B?

All four rotating requests returned 0 tokens after roughly 59 to 66 seconds, with a zero hit rate, an all-zero profile and no speculative forwards from turn three. Its warm 0.689 and cold 0.717 figures come from single valid requests and are usable. The primary rotating median does not exist, and the report says not to quote it as either 0.00 or 0.69.