Ternary Bonsai 2 27B Benchmarked: A 6GB Model That Actually Codes

Ternary Bonsai 2 27B makes aggressive claims for a model you can fit on a single consumer GPU: a 27B network at 2.13 bits per weight, a 5.95GB ternary packing claiming 98.2% of full-precision intelligence, 262k context on weak GPUs through hybrid attention, and agentic tool calling intact. Claims like that usually fall apart the moment you actually run them. So I ran them, on a pair of Tesla T4s, which are exactly the kind of weak, old GPUs this model claims to serve.

What ternary weights actually are#

Normal language models store every weight as a full 16-bit number, so each parameter can take any of 65,536 values. Ternary quantization throws that away completely: every weight is one of exactly three values, negative one, zero, or positive one. Groups of 128 weights share a single FP16 scale factor that says how big their steps are.

The math works out to about 1.72 bits per weight, a 9x compression against the roughly 54GB that the full-precision model would need. The trick is that Bonsai 2 is not a careless post-training squeeze. It is derived from Qwen3.8 27B with the architecture unchanged, the weights are rotated with a Hadamard transform before the ternary assignment so the rounding damage lands where it hurts least, and the result covers the whole model end to end: embeddings, attention projections, MLPs, and the output head. Only 26 million of 24 billion parameters (0.1%) stay in higher precision.

What makes this release interesting is less the 5.9GB itself and more the comparison against ordinary 2-bit quants. PrismML's own table, which my results are consistent with: the standard IQ2_XXS 2-bit quant of the same base model drops to 84.1% of full precision. Bonsai 2 holds 98.2% (an 84.78 thinking-mode average against 86.32 for FP16 across a 20-benchmark suite). A conventional "2-bit" quant costs about a sixth of the model's intelligence. Compression that is trained for from the start costs about 2%. The first Bonsai from July sat at 95%, so the gap is also closing generation over generation.

The license is Apache 2.0. It runs on NVIDIA through a custom llama.cpp fork and on Apple devices through custom MLX kernels, with the packed weights consumed directly and never expanded back to FP16 in memory. PrismML claims up to 143 tokens per second on an RTX 5090. My hardware was more modest.

What I was testing#

The GGUF release ships two packings of the same 27B model (26.9B language parameters plus an optional vision tower):

Both files matched the model card's claimed byte sizes exactly, which is a good sign for a release. One catch to know upfront: stock llama.cpp cannot run these files. The PQ2_0 and PTQ1_0 formats need custom Hadamard kernels, so you need the PrismML llama.cpp fork. I used their CUDA 12.8 build on a Kaggle VM with 2x Tesla T4 (15GB each, Turing architecture), 4 vCPU, and 31GB RAM.

Throughput#

Official pp512/tg128 benchmarks with full GPU offload and flash attention on:

The packing choice is a clean trade. PQ2_0 is nearly 1.5x faster at prompt processing, which matters a lot for agents. PTQ1_0 wins on decode for memory-constrained cards and is the only way 8GB cards can play at all. On the T4s decode was a tie, but on newer Ada-class hardware the ternary packing officially pulls ahead, since its 17% lower weight traffic matters more when bandwidth is the bottleneck.

Adding a second T4 did almost nothing for prefill (+6%) but 52% for decode, which tracks: prefill is compute-distributed, decode is bandwidth-bound.

VRAM math that actually works#

The hybrid attention design keeps only 16 of 64 layers as full attention (one in every four blocks), the other 48 run linear attention whose state stays constant in size. That is what makes giant context windows practical at all: the KV cache, the part of memory that normally explodes with context length, grows at only 64KB per token, or roughly 6.3GB at 100k context. That gives a simple planning formula:

max_ctx ≈ (VRAM_GB - weights_GB - 1.5 overhead) × 1024^3 / 65536

Measured and validated against that formula:

The full 262,144-token context server loaded and served on 26.2GB across the two T4s. The weak-GPU claim holds for 12GB and above at useful context windows. 8GB cards cannot fully offload PQ2_0 (it needs 8.7GB minimum), so they get the ternary packing with small context or partial CPU offload.

The real exam: an agent building an app#

Benchmarks are one thing. I also gave it the exam I actually care about: driving a coding agent end to end. I pointed a local agent at a llama-server running PQ2_0 with a 64k context and a 2048 reasoning budget, and gave it a real task: build a 3-file todo webapp (HTML, CSS, JS) with add, complete, delete, filter, and localStorage persistence, then verify it compiles.

It succeeded in about 35 minutes across 15 to 20 agent steps.

• All three files written correctly on the first try, node --check clean
• Unprompted, it wrote its own 16-assertion functional test suite with DOM stubs
• When checks failed, it correctly blamed its own test stubs (missing addEventListener and createElement), never the working app code, and iterated to all 16 checks passing
• The final app was genuinely clean: try/catch storage, event delegation, textContent-only rendering, aria labels
• Native OpenAI-format tool calls through llama-server worked flawlessly, and context compaction fired mid-run without derailing anything

The pace is the weakness: two to four minutes per step, about 35 minutes for what a frontier model does in three.

Prefill is the tax nobody prices in#

Raw prefill throughput barely degrades: 293 tok/s at short prompts, still 285 at 8k, only 15% down at 32k. The problem is arithmetic, not bandwidth. Every agentic step re-prefills the whole conversation. At 40k context, that is about 160 seconds of pure prefill per step before a single token is generated, plus up to two minutes of thinking at 15 to 22 tok/s. Hence 3 to 5 minutes per step, observed consistently.

One hard lesson: the agent's own system prompt is around 36k tokens, so a 32k-context server cannot run it at all. 64k is the practical floor for agentic use, which needs 12.4GB of VRAM on PQ2_0. If you run local agents, the mitigation list is: fewer, bigger steps per round trip, cap the reasoning budget, use server-side prefix caching (it does help, later steps were not slower than the first), and let compaction do its thing.

Capability spot checks#

Eight quick probes with a 400-token cap:

• GSM8K-style arithmetic: pass, clean steps
• AIME-style proof sketch: correct reasoning (no positive n exists, sum is 0), cut off by my token cap before the final line
• HumanEval-style is_prime: pass, correct trial division, code only as asked
• Trap rain water O(n): pass, correct two-pointer solution, verified the example output
• Instruction following (3 bullets starting with "Green"): over-deliberated the ambiguous prompt and never answered. My fault for capping tokens on a thinking model
• MuSR-style liar puzzle: reasoning complete, answer cut by the cap
• MMLU-style knowledge: pass
• Tool call JSON format: correct shape, cut mid-JSON by the cap

Score: 4 clean passes, 4 truncated by my harness, with every visible reasoning trace correct. The lesson generalizes to every thinking model: give them token room. 1024 tokens minimum, or lower the reasoning effort for trivial prompts, otherwise you are benchmarking your own token cap.

Verdict#

The claims survive contact with hardware. The 5.95GB file is real, 15 tok/s on a single weak GPU is real, the full 262k context on 30GB of VRAM is real, and the model retained thinking, tool calling, and correct reasoning across math, code, and logic. As a local coding model it works, with patience: a 12GB-plus card, a 64k context, capped reasoning budget, and tolerance for 3 to 5 minute agent steps.

The honest engineering impresses me more than the speed. A 27B model at 1.75 bits per weight that still passes its own written test suite, blames its test stubs instead of the app, and refuses to hallucinate file state. That is a 6GB file that behaves like a real coding assistant, just a slow one. On the right hardware (a 24GB card with PTQ1_0 at full 262k context), it stops being a curiosity and becomes a genuinely capable local agent.