A 48 GB Model Cold-Starts in 3 Seconds on a 2014 Xeon. llama.cpp Took 3 Minutes.

Abhishek Dash8 min read
A glowing red data crate connected by a ribbon cable to a CPU under a tall cooler on a workshop bench, with a progress bar on a monitor behind.

In short

On a 12-thread Xeon E5-2620 v3 with 47 GiB of RAM, a SATA SSD and no GPU, Janas beat llama.cpp on prefill for all five models and on the engine decode benchmark for four, using 1.2 to 2.0 times less RAM. The 80B ran at 7.6 tokens a second with multi-token prediction at 24.5 GB peak RSS and no swap. llama.cpp needed 187.9 seconds to become ready and 41.6 GB for the same model.

Key takeaways

  • The test machine was a 12-thread Xeon E5-2620 v3 at 2.4 GHz, 47 GiB of RAM, a SATA SSD and no GPU.
  • On the engine's 256-token decode benchmark Janas beat llama.cpp on four of five models; in the server harness it won three, tied the 30B at 6.94 against 6.98, and lost the dense 27B.
  • Janas was faster on prefill for all five models, by 1.14 to 1.75 times.
  • The 80B model cold-started in 3.06 seconds on Janas against 187.9 seconds on llama.cpp, and peaked at 24.5 GB RSS against 41.6 GB.
  • Peak disk utilisation on Janas runs was 84 to 100 percent, which makes the SATA link the binding constraint for the 80B.

A 48.4 GB model became ready in 3.06 seconds on a machine with a SATA SSD. The same model took llama.cpp 187.9 seconds to become ready on the same machine. The gap is the more useful number.

The test box is a 12-thread Intel Xeon E5-2620 v3 at 2.4 GHz, with 47 GiB of RAM, a 1.5 TB SATA SSD and no GPU. It has AVX2 but not AVX-VNNI. It is a 2014 server part, and the disk is the slowest component in the chain. The machine was otherwise idle: a co-located game server was stopped for the runs, and the background services were left running but quiet.

Five models, two engines, three rounds each, one job at a time. The engines were Janas v2026.10.30.1 and llama.cpp b12032. The models were the closest catalog entries to the plan's five tiers, each in the same quant family for both engines.

What Janas is betting on

Janas is a CPU-first inference engine for mixture-of-experts models. Its bet is that a large MoE model does not need to sit entirely in RAM. It can stream experts from disk as the router selects them, keeping a cache of the hot ones, so a 48 GB model can run on a machine with less memory than the model.

The least favourable hardware for that bet is a slow disk with a small cache. A SATA SSD at roughly 500 to 700 MB/s is that hardware. The plan that set up this test expected Janas to lose on it, and wrote a decision rule that said so: if the 30B model fell below 10 tokens a second, switch to llama.cpp because it handles SATA better.

The rule was reasonable. The measurement did not cooperate.

Method

Four measurements per model, in this order:

  1. janas-bench with a 512-token prompt, 256-token generation and three rounds. This is the engine's own tuned measurement.
  2. Janas server, with three streamed requests using the same prompt, max_tokens=256 and reasoning disabled. This captured cold start, time to first token, generation speed, peak RSS from /proc, peak disk read from /proc/diskstats, and peak disk utilisation from iostat.
  3. llama-bench with the same 512/256 shape and 12 threads.
  4. llama.cpp server, with the same three streamed requests and the same monitors.

Reasoning was disabled in every server run. The Qwen3.x models burn roughly 880 reasoning tokens before answering, which would otherwise be counted as generation and make the comparison meaningless.

Each model's files were deleted after its runs. The machine returned to 91 GB used and 1.4 TB free.

The results

tg256 is the engine's own benchmark. Gen is generation speed in the server harness. Cold is time to ready. RSS is peak resident memory.

Model Engine tg256 tok/s Gen tok/s Cold s Peak RSS GB
Qwen3.5-9B dense Janas 4.3 7.55 5.17 5.74
Qwen3.5-9B dense llama.cpp 4.05 4.00 9.26 8.56
Qwen3.8-27B dense Janas 1.1 2.10 12.96 16.97
Qwen3.8-27B dense llama.cpp 1.15 1.17 14.29 20.23
Qwen3-30B-A3B MoE Janas 7.4 6.94 1.75 15.04
Qwen3-30B-A3B MoE llama.cpp 6.86 6.98 35.72 30.81
Qwen3-Next-80B-A3B MoE Janas 5.5 6.64 3.06 24.54
Qwen3-Next-80B-A3B MoE llama.cpp 3.90 3.62 187.92 41.62
Gemma-4-26B-A4B MoE Janas 4.3 5.96 8.43 16.62
Gemma-4-26B-A4B MoE llama.cpp 3.56 3.59 26.16 24.92

The two columns disagree on the 30B because they measure differently. tg256 is the engine's greedy benchmark; Gen is the streamed server harness. The 30B is a Janas win in tg256 at 7.4 against 6.86, and a tie in the harness at 6.94 against 6.98.

The dense 27B is the exception. llama.cpp edged it on raw decode, 1.15 against 1.1 tokens a second, and the server harness gave Janas 2.10 against 1.17 because of multi-token prediction. The engine's own benchmark, which does not use MTP, is the fairer raw comparison, and there llama.cpp is 4 percent ahead. That is the one model where the plan's premise held.

In the engine's decode benchmark, Janas won the other four. In the server harness, it won the 9B, 80B and Gemma, tied the 30B at 6.94 against 6.98, and lost only the 27B. Prefill went to Janas on all five, from 1.14 times on the 27B to 1.75 times on the 80B. Cold start went to Janas on every model, from 1.1 times on the 27B to 61 times on the 80B. Peak RSS was 1.2 to 2.0 times lower on every model.

The 80B is the headline. Janas held a 30.1 GiB expert cache, streamed the rest from disk, and never swapped. Its peak read hit 644.7 MB/s at 97.8 percent disk utilisation. llama.cpp needed 41.6 GB of resident memory on a 47 GiB machine and 187.9 seconds to mmap the 48.5 GB file over SATA.

The time-to-first-token wrinkle

The server harness puts one number in the other direction: mean time to first token was lower on llama.cpp for most models. The 80B averaged 17.45 seconds on Janas against 8.55 on llama.cpp.

That average is misleading. Janas's first run on each model includes the expert cache fill, and the 80B's first run took 49.4 seconds. The next two took 1.62 and 1.38 seconds. The steady-state first token is about 1.6 seconds. The harness records the mean of three, so the cold run dominates it.

llama.cpp's first token is quicker once the model is loaded because it is memory-mapped and the operating system can fault pages in on demand. The catch is that getting to the point where it can serve took 187.9 seconds.

Neither number is wrong. They measure different things: warm response latency versus time to a usable server.

Applying the plan's decision rule

The plan said: if the 30B model gets less than 10 tokens a second, the SATA disk is too slow, so switch to llama.cpp.

The 30B got 7.4 tokens a second. The first half of the rule triggered. The second half did not survive: llama.cpp got 6.86 on the same model, and 3.90 on the 80B against Janas's 5.5.

The rule assumed llama.cpp handles SATA better. On this machine it moved less data but was slower to start, slower on prefill and slower on the engine's decode benchmark for four of five models and a tie on the 30B in the server harness. The threshold was fine. The premise was wrong.

What to run

The 80B with multi-token prediction is the primary recommendation. It cold-starts in about 3 seconds, peaks at 24.5 GB RSS, leaves roughly 20 GB for the co-located game server, and holds 7.6 to 9.1 tokens a second with a steady-state first token around 1.6 seconds. That is a genuine 80B model on a CPU-only box with a SATA disk.

The speed-first alternative is the 30B at 7.4 tokens a second raw, 15 GB RSS and a 1.75 second cold start. It is the better chat experience if latency matters more than model size.

Avoid the dense 27B. At 1.1 tokens a second raw, 2.1 with MTP, it is not interactive on this CPU. It is also the only model where llama.cpp is ahead, which makes it a poor default for either engine.

The hardware recommendation is narrower than the plan expected. An NVMe upgrade is the only change that would move the 80B materially, because peak disk utilisation hit 84 to 100 percent on every Janas run and the 80B streams experts continuously. A faster CPU would help dense models, but the 27B's problem is the 2.4 GHz clock and the missing AVX-VNNI, not the disk.

Where this could be wrong

This is one machine, one kernel, one run date. The CPU is a 2014 part, the disk is SATA, and the results should not be generalised to an NVMe workstation or a modern server.

The model mapping differs from the plan. The plan's model names were written from memory and do not exist in the catalog, so the nearest real entries were used. The tiers are comparable, the exact models are not.

The server harness averages three runs, and the first run carries the cache fill. That makes mean time to first token pessimistic for Janas. The steady-state numbers are reported separately for that reason.

Multi-token prediction is engine-specific. The tg256 column does not use it; the server generation numbers do where the model ships a prediction block. The two should not be mixed.

Janas is experimental software with a weekly release cadence. The numbers are tied to v2026.10.30.1 and would need a re-run after a major release.

Raw reports, harness JSON and logs exist for every model. The counts above were not copied from a vendor page. They were measured on the test machine, and the machine was otherwise idle.

On this page

Sources

  1. Janas releasesJanas, 2026
  2. llama.cppggml-org, 2026

Frequently asked questions

What is Janas?

A CPU-first inference engine for mixture-of-experts models. Its bet is that a large MoE model can stream experts from disk instead of holding every weight in RAM, so a 48 GB model can run on a machine with less memory than the model. The benchmark tests that claim on hardware with a slow disk, which is the least favourable case for it.

Why did the plan expect llama.cpp to win?

The plan assumed a SATA SSD would hurt Janas more than llama.cpp, because Janas streams experts from disk while llama.cpp can use mmap. The measured result inverted that. Janas was faster to start, faster on prefill, and faster on prefill, and ahead on generation in three of the five server runs, with the 30B a tie and the dense 27B the only loss. llama.cpp moved less data, but it also took far longer to become ready.

Is 7.6 tokens a second usable?

For chat, it is readable but not fast. The 30B model is the speed-first choice at 7.4 tokens a second raw with a 1.75 second cold start. The 80B with multi-token prediction reaches 7.6 to 9.1 tokens a second in steady state, which is usable for drafting and batch work and slow for interactive coding.

Would an NVMe drive fix it?

It is the only change that would move the 80B materially. Peak disk utilisation hit 84 to 100 percent on every Janas run, and the 80B streams experts continuously. The CPU also caps dense models, because it has AVX2 but no AVX-VNNI and runs at 2.4 GHz. A faster disk would not make the dense 27B interactive.