Laya vs Jev: The First Byte-Identical Benchmark of System One Decision Models

There is a category of model that almost nobody benchmarks properly: the fast, small "System One" decision models, the ones that triage a support ticket, flag a toxic message, classify intent, or gate whether an agent should act. They run in milliseconds, they cost fractions of a cent, and they make most of the actual decisions inside production AI systems. Two of them matter right now: Laya, a small open encoder-style model, and Jev, the decision model served through the TypeSafe API. And here is the odd part: as far as I can tell, nobody had ever run both on identical inputs. The Laya author never had Jev API access, and Jev's published numbers come from different prompts, so every comparison so far has been apples to oranges.

So I built the harness. Same 751 states, same questions, same seed, and the runner verifies question SHA hashes before comparing anything, byte for byte. Laya is the convaiinnovations/laya English checkpoint on a local MacBook M2 CPU. Jev is pinned jev-1.13.0 through the TypeSafe API. The entire Jev run cost $0.008. Everything is open in the repo: github.com/instax-dutta/sysone-bench.

The headline table#

Suite	n	Laya	Jev	Qwen-PCD	Best
triage (curated)	160	0.800	0.888	0.825	Jev
guardrails (curated)	60	0.883	0.967	0.900	Jev
moderation (curated)	90	0.833	0.989	0.700	Jev
agnews (4 labels)	100	0.940	0.910	0.730	Laya
emotion (6 labels)	100	0.540	0.550	0.540	tie
banking77 (12 intents)	96	0.802	0.906	0.500	Jev
mnli (3-way NLI)	60	0.983	0.867	0.733	Laya
sst5 (score, 5 levels)	60	0.367	0.617	0.617	Jev/Qwen tie
multilingual intent (5 langs)	25	0.360	1.000	0.880	Jev

The third column needs explaining. Qwen-PCD is stock Qwen2.5-1.5B-Instruct under parallel constrained decoding, the same technique as a well-known RLCD repo that ships no fine-tuned weights, running locally in MLX on the M2, with score questions asked as enums over rubric levels. I added it as a control: what does a general-purpose small LLM do against two purpose-built decision models?

What the numbers actually say#

Jev leads the curated decision suites, and the lead is not uniform. The gap concentrates in fine-grained multi-class judgment: 6-way intent went 0.725 versus 0.975, toxicity 0.767 versus 1.000, multilingual intent (Hindi, Spanish, French, German, Arabic) 0.360 versus a perfect 1.000. Those are the questions where a wrong answer has a specific wrong shape, and Jev is near-perfect at them.

Laya wins where the question is comparison-shaped. MNLI, the 3-way natural language inference suite, went to Laya 0.983 versus 0.867, and agnews to Laya 0.940 versus 0.910. A bidirectional encoder apparently suits premise-hypothesis comparison better than the generative approach. That detail matters if you are picking a model for RAG-relevance or dedup-style work.

Emotion is the wall nobody climbed. Three models, 0.54 to 0.55, with calibration error around 0.3. Six overlapping affect labels with single-word criteria under-specify the question itself. When all models fail identically, the benchmark is measuring the label scheme, not the models.

The Qwen-PCD column produced the most instructive failure: first-token logit slicing degrades as option count grows. It held its own on 4-6 way questions and tied Jev on sst5 scoring, but collapsed to 0.500 on banking77's 12-way enum. Constrained decoding is a real technique with a real scaling cliff.

Calibration and the gate#

Accuracy is half the story for decision models; confidence calibration is the other half, because production systems gate on it. Both models ship well-calibrated on this suite (expected calibration error 0.03-0.08 on curated suites), with Jev lower everywhere. Gating at 0.85 confidence: Laya answers 69% of questions at 95.3% accuracy, Jev answers 83% at 98.8%. At a 0.95 threshold: Laya 46% coverage at 96.5%, Jev 66% at 99%. Laya gates almost as well, which matters more than raw accuracy if your architecture defers uncertain calls to a bigger model.

Cost and latency#

This is where the comparison gets lopsided in the other direction. Laya is $0 self-hosted after an 808MB download, running at 375-476ms per call on an M2 CPU. Jev through the API measured 885-1068ms per call end to end from my region (above the 70-500ms claimed, worth knowing if you are latency-sensitive; fine for batch triage, too slow for inline request paths), at a cost of $0.008 for the entire 192K-token run at $0.042 per million input tokens. Eight thousandths of a dollar. The pricing story of small decision models is genuinely surreal.

The Router fix and what it shows#

One follow-up worth having in the record. Laya ships a Router mode that preloads both English and multilingual checkpoints, so I re-ran all 751 states through it. Every English number reproduced exactly (good sign for reproducibility), and multilingual intent jumped from 0.360 to 0.840. The router keys on script rather than language, so Spanish, French, and German stayed on the English checkpoint and Jev still wins the multilingual column outright. But 0.840 versus 0.360 is a 2.3x improvement from one design decision, and it tells you where the Laya project is heading.

Limits, stated plainly#

The curated suites are hand-labeled by one author, me. The public subsets are random draws at sizes 25-100, so confidence intervals are wide, and the guardrails gap in particular is not statistically significant. One Jev version, one day, one region. The score questions on the Laya side underperform in ways its own benchmarks doc predicts honestly. This is a first measurement, not a final verdict, which is exactly why the harness checks hashes and ships everything in the open: re-run it when jev-latest moves.

The takeaway#

For everyday triage, guardrails, and moderation with mostly binary questions plus confidence gating, self-hosted Laya sits within 7-16 points of Jev at literally zero marginal cost, and gates almost as well. If your workload includes 6-way intent classification, toxicity precision, or non-English inputs, Jev earns its per-token price, and by a wide margin on multilingual. And if you thought a 1.5B general instruct model with constrained decoding could split the difference, it can, right up until the option count passes about six.

What I like most about this category is what the numbers imply: most of the "AI decisions" in production systems do not need a frontier model, and the frontier's price is not the price of intelligence at this layer. A $0.008 benchmark run producing 751 graded states says the quiet part out loud: the cheapest useful intelligence got very cheap, very fast. The full harness, run files, and fairness proofs are at github.com/instax-dutta/sysone-bench if you want to re-run or extend it.