I Built a Fully Local Voice AI Stack on a CPU With No GPU: Here Are the Real Numbers
Voice assistants are the part of local AI that actually matters to normal people. Not another terminal chatbot: a thing you talk to, that answers out loud, that never phones home. So when a pitch crosses my desk for a "2.27B parameter voice stack, zero API calls, 150 tokens per second, data goes nowhere", I do what I have learned to do with every pitch now: I build it and measure it.
The stack in question: Qwen3.5-2B as the language model, Moonshine Base for speech to text, Kokoro for text to speech, llama.cpp doing the serving. Target hardware: a CPU-only server. No GPU anywhere in the build. This post is the full audit, what worked, what the marketing gets wrong, and what a fully local voice stack really costs you in disk, RAM, and seconds.
The build
Three models, one server:
| Component | Model | Job |
|---|---|---|
| Language | Qwen3.5-2B (Q4_K_M and BF16 GGUF) | Generation |
| Speech to text | Moonshine Base EN (58M params, MIT) | Transcription |
| Text to speech | Kokoro-7M-Distill (7.5M params, Apache 2.0) | Synthesis |
| Serving | llama.cpp v0.4.1, CPU build, 4 threads | LLM inference with prefix caching |
The host is an old 12-core Xeon E5-2620 v3 at 2.4GHz with 47GB RAM and no GPU, which is exactly the class of machine most people have lying around. Everything ran pinned to 4 of the 12 cores at low priority so the server's other work was untouched, which makes the numbers you are about to see a floor, not a ceiling.
The named model assets total about 1.45GB with Q4 text weights (1.28GB LLM plus Moonshine's 141MB plus Kokoro's 30MB), or about 3.95GB with the BF16 LLM. The whole temporary install, both LLM variants plus speech models plus PyTorch plus the compiled server, was 7.5GB, and even that got cleaned up afterward. For a complete voice assistant, that is nothing.
The LLM numbers
llama-bench, three repetitions, four threads, thinking disabled for the pipeline:
| Quantization | Prompt 512 | Prompt 2048 | Generation 128 |
|---|---|---|---|
| Q4_K_M | 45.7 tok/s | 44.7 tok/s | 11.05 tok/s |
| BF16 | 42.3 tok/s | 42.4 tok/s | 3.01 tok/s |
Note the asymmetry: quantization barely touches prompt processing (5-8%) but Q4 generation is 3.67x faster than BF16. On CPU, quantization is the correct default, not a compromise.
The 150 tok/s claim, audited
The pitch I was auditing promised 150 tokens per second. Here is what measurement says:
- Q4 single-stream: 11.05 tok/s, 13.57x slower than claimed
- BF16 single-stream: 3.01 tok/s, 49.78x slower than claimed
- Q4 with four concurrent requests: 22.24 tok/s aggregate, still 6.74x slower
Could a GPU, a different engine, a different quant, or bigger batching hit 150? Maybe. That is exactly the problem: the claim ships with no hardware, engine, quantization, context, batch, or concurrency details, so it is a vibe, not a spec. My 11 tok/s is a reproducible number on stated hardware. The 150 is nothing you can plan around. When a number has no config behind it, treat it as fiction until proven otherwise.
Concurrency: four streams on a 2013 CPU
This is the result that surprised me. With four parallel slots and continuous batching on the same 4 threads:
| Concurrent requests | Per-stream | Aggregate | End-to-end throughput |
|---|---|---|---|
| 1 | 11.50 tok/s | 11.50 tok/s | 2.70 tok/s |
| 2 | 8.08 tok/s | 16.16 tok/s | 2.92 tok/s |
| 4 | 5.56 tok/s | 22.24 tok/s | 3.11 tok/s |
Per-stream speed falls as concurrency rises, but aggregate decode nearly doubles and end-to-end throughput actually rises with more concurrent users, because prompt evaluation (the expensive part on CPU) overlaps with someone else's decode. Four people talking to one voice assistant on a CPU that predates the AI boom is genuinely workable. This is not a million-user architecture. For a household, a small team, or a private platform, it is exactly right.
The prefix cache finding
The most interesting number in the whole benchmark. I built a deliberately voice-assistant-shaped workload: two prompts sharing 2,048 identical leading tokens (think system prompt plus conversation history, which is what every voice session actually looks like) with only 516 new trailing tokens.
- Token-weighted cache hit rate: 79.875%
- Prompt evaluation dropped from 58.0s to 12.0s on reuse: 4.82x faster
That lands almost exactly on the theoretical 4.97x ideal, losing only a few percent to cache overhead. The 80% figure being "workload-dependent" matters, but voice assistants are precisely the workload it depends on: every turn re-sends the persona and system prompt, so a voice assistant with prefix caching gets this speedup in production, not in a synthetic test.
The speech numbers, both directions
Moonshine Base EN transcribed a 44.37-second recording in a median 7.37 seconds, 6.02x real-time, with a peak footprint of about 740MB. Kokoro-7M-Distill synthesized 8.85 seconds of audio in a median 0.73 seconds of compute, 12.10x real-time, in under 820MB of memory. Speech is a solved problem on CPU; both directions are comfortably faster than real time, which is the property that makes an interactive voice loop possible at all.
One honest wrinkle: Moonshine's five offline transcription runs were not textually identical, so do not describe the quantized model as deterministic. And the one-second-clip test produced an empty transcript, which is not a quality failure, it is a reminder that very short utterances need endpointing logic.
The full loop
The end-to-end pipeline: Moonshine transcribes, Qwen thinks (with thinking disabled for latency), Kokoro speaks, and then Moonshine transcribes Kokoro's own output back as an intelligibility check.
| Stage (Q4 pipeline) | Median |
|---|---|
| Speech to text | 7.60s |
| LLM | 6.99s |
| Text to speech | 1.01s |
| Total for a 44.37s recording | 15.60s |
That is 2.84x faster than real time for the whole loop, with a total memory footprint around 3GB across processes. The round-trip word error rate against the LLM's generated reply was 4.0%, meaning the TTS produced audio the local transcriber could read back almost perfectly. The BF16-text variant ran the same loop in 19.91s, slower at every LLM-bound stage.
A fun quality wrinkle from the logs: one BF16 run misidentified the source of a passage as A Christmas Carol instead of A Tale of Two Cities, then got it right on a later run. Small model, quantized stack, occasional confident nonsense, budget accordingly.
What "fully local" actually means
After staging the assets, I ran the complete inference pass under a syscall trace watching network activity. Both the LLM server and the speech process made exactly zero non-loopback connections. Everything stayed on 127.0.0.1.
That is the strong version of the privacy claim, and it is real: no audio, prompts, or transcripts went to any third-party API during inference. But the same audit found the fine print, and the fine print is usually where the marketing hides:
- Setup needed the network (models, packages, one surprise spaCy download on first TTS use)
- "Zero API calls" is not literally true; components talk over local HTTP, and audio sent from your phone to the server has left your phone
- The server itself may retain audio and transcripts in logs, memory, or databases unless you explicitly disable that
- Hosting, backups, and package distribution are still subprocessors of a sort
Local inference removes the AI vendor from the picture. It does not remove your obligations, your logs, or your network. That distinction is the difference between a defensible privacy claim and a compliance hand-wave.
The claim card
| Claim | Verdict |
|---|---|
| "2.27B parameters" for the stack | Misleading: that is the Qwen checkpoint alone; text-only deployment plus speech is about 1.95B |
| "4.7GB on disk" | Conditionally correct for named model files, excluding runtime and any LoRA |
| "Zero API calls" | Misleading literally: local-only after staging, setup used the network |
| "150 tokens/second" | Not reproduced: 13.6x off single-stream, 6.7x off at 4-way concurrency |
| "80% prefix cache hit rate" | Reproducible, workload-dependent (79.875% on a shared-prefix workload) |
| "Data goes nowhere" | Overstatement: nothing left the box during inference, but logs and client transmission still exist |
| "Runs on an RTX 3060 laptop" | Plausible for Q4 text only; the speech runtime is CPU-oriented and BF16 is tight on 6GB |
The verdict
A fully local voice stack on a CPU-only box is a working thing with real throughput: 11 tok/s of generation single-stream, 22 aggregate, 6x real-time ears, 12x real-time speech, 3GB of RAM, 1.45GB of disk. Total per-token cost: whatever the electricity costs, which on this hardware is rounding error.
The claims that survived: CPU-only operation, the transcription and synthesis speeds, the concurrency behavior, the prefix-cache effect, and the network isolation. The claims that did not survive: the 150 tok/s headline, the parameter-count framing, and the blanket privacy phrasing. If someone pitches you a local voice stack, these are now the numbers to check their pitch against. Mine are reproducible; the setup, the pinned model revisions, and the benchmark commands are all documented, and the whole thing rebuilt from scratch in a temporary directory in one afternoon.
The part I keep thinking about: every component here, the 2B brain, the 58M-param ears, the 7M-param voice, is small enough that the whole stack fits in less RAM than a Chrome tab pile. The voice assistant that stays in your house is no longer a research project. It is an afternoon.