Spark-X2.5 Audited: What a 4B Coding Agent Can Actually Do on Two T4s

Spark-X2.5 shows up with a big card for a small model: a 4B and a 1.7B pair trained on Huawei Ascend hardware, native 1M-token context through hybrid attention, a native agent tool protocol, and headline scores like SWE-Bench Pro 44.4 and AIME 90.7. Cards like that deserve a line-by-line check, because the gap between the card and the model is where most disappointments hide. So I audited both cards against the artifacts, then stress-tested the one thing a card cannot tell you: can these things actually drive a coding agent over multiple steps.

Short version: the architecture and integration claims are real, the headline benchmark numbers are not reproducible without H100-class hardware (and I say so up front), the 4B is a genuinely usable small coding agent, and the 1.7B talks about acting more often than it acts.

Checking the card line by line#

The test rig: a Kaggle VM with two Tesla T4s (15GB each), 4 vCPU, 31GB RAM, BF16 through transformers with the card-recommended sampling (temperature 1.0, top_p 0.95), thinking mode on by default.

An environment landmine first: the VM image ships transformers 5.0.0, and it crashes on Spark's custom modeling_spark.py. The card pins 4.57.1, and the card is right. Believe model cards when they pin exact versions.

Card claim	Verdict
Hybrid attention, 1 full + 3 sliding-window layers	Verified: repeating SSSF pattern in config, 4B has 36 layers (9 full/27 SWA), 1.7B has 28 (7/21)
Native 1M-token context	Verified structurally (max_position_embeddings 1048576, RoPE theta 5M); functionally tested only to 5.6k on this hardware
Exact dimensions for both sizes	Verified, exact match in config.json
Pretraining scale, Ascend training, RL post-training	Not verifiable from public artifacts, taken on trust
Agent tool protocol	Verified: native tool-call shape out, tool results in, OpenAI-style tools schema, thinking blocks. Both models pass the single tool-call test
llama.cpp support	Verified in source: master contains native spark2.5 support. transformers path tested working
200+ languages	Plausible: 131k vocab, lossless round-trip on six languages plus code, correct Chinese smoke answers. Count not enumerated
Headline scores (SWE-Pro 44.4, AIME 90.7...)	Not reproduced. That needs H100-class VRAM and the full eval harnesses

That last row is the honesty anchor of this post. My tests are smaller, but they run end to end on the actual models, which is the kind of verification a model card cannot give you.

The measured basics#

Metric	1.7B	4B
Load time to CUDA	20s	8.5s
VRAM idle (BF16)	3.4GB	8.2GB
VRAM under agentic load	~4.8GB, stable	~10-12GB, OOMs in long sessions
Throughput on T4, thinking on	~16 tok/s	~12 tok/s

The 1.7B fits a single T4 with room to spare. The 4B only misbehaves once a session gets long: eager attention transients spike past 15GB during long agentic runs, so for real work it wants CPU offload, shorter context windows, or a bigger GPU. Load times are a non-issue either way, 20 seconds for the 1.7B and 8.5 for the 4B.

The real exam: agentic coding#

Three tasks, same for both models, each driving a native tool loop (read, list, write, edit, run_bash) with a 20-step cap:

Task	1.7B	4B
Implement median() to pass 5 pytest tests	pass, 5 steps	pass, 4 steps, zero errors
Debug an off-by-one across 2 files, 7 tests	pass, 3 steps	pass, but 14 steps and it never ran the test suite itself
Build an LRU cache from a spec, from scratch	fail: 3 tool steps plus 17 turns of talking instead of acting	pass, 5 steps

Agentic score: 1.7B goes 2/3, 4B goes 3/3.

The 4B's pass list is what makes it worth taking seriously: it implements, it debugs, it builds from a written spec, it uses the file tools correctly, and it runs the tests when it is done. The failure modes are specific too. It converges slowly when diagnosing (14 steps on the debug task) and left a verification gap at the end, so pair it with a harness that auto-runs tests instead of trusting it to check its own work.

The 1.7B's failure is the more interesting data point. On the from-scratch task it spent 17 consecutive turns writing prose about what it was going to do instead of emitting a single tool call. That is the difference between an assistant and an agent, visible in one trace: it narrates action instead of taking it. Fine for short implement-and-debug bursts. Do not give it open-ended multi-file work.

Micro-benchmarks backed the pattern up. A single tool call was textbook-clean on the 4B. Instruction following with exact constraints failed on both (asked for exactly 3 sentences, got 23 and 29; thinking models crowd out fixed length constraints). Single-turn codegen at a 1024-token budget went 1/5 for the 1.7B and 2/5 for the 4B, though most of those failures were fence fragments from duplicate code blocks rather than wrong code, and a needle-in-haystack retrieval at 5.6k tokens passed on the 1.7B.

The thinking-verbosity tax#

Both models default to long thinking traces, and this is the section I wish every model card shipped with.

Thinking crowds out answers. Trivial prompts drew 179 thinking tokens from the 1.7B against 54 from the 4B, and on constrained-output tasks the traces ate the response budget and broke the constraints.

Thinking models also emit multiple code blocks, a draft inside the trace plus the final answer. Naive parsers that grab the first block under-score these models badly. My first scoring pass gave 0/5 and 1/5 to outputs that scored 1/5 and 2/5 once the parser took the last block instead. If you evaluate thinking models with a first-block parser, you are measuring your parser.

And because the thinking draft and the final answer both look like ordinary fenced code, prose fragments end up inside fences and turn into SyntaxErrors in a naive harness.

The fixes are cheap: give each step a token budget of 1024 or more, parse the last block, and switch thinking off for trivial steps.

Deployment notes, from pain encountered#

• Pin transformers to 4.57.1. Version 5.x crashes on the custom modeling code
• The 4B plus a 15GB card plus long context equals OOM: eager attention spikes over 2GB of transients. Per-step history caps, emptying the CUDA cache per step, and CPU offload all worked as mitigations
• bitsandbytes 8-bit does not work with this architecture (the custom eager attention path is not implemented for 8-bit types). For memory-constrained boxes, use the Q4 GGUF, not 8-bit tricks
• The claimed integrations (Codex, Claude Code, Hermes-style harnesses) rest on a real native protocol, which I verified by parsing real generations. What I did not test is harness integration quality, so treat "works with your agent" as likely but unverified

Limits, stated plainly#

Small task set (three agent tasks plus six micros), one run each at temperature 1.0, so expect plus-or-minus one task of run-to-run variance; one problem passed on a repeat run for the 1.7B. No rival baseline ran on this VM, so the numbers are absolute, not relative. The 1M context claim is structurally verified and functionally tested only to 5.6k, the ceiling of this hardware class. And the headline card numbers were not reproduced, by design: that requires a different hardware tier and the official harnesses.

Verdict#

The 4B is the smallest model I have tested that completes implement, debug, and build-from-spec agent loops with clean native tool calls. It needs offload on 15GB hardware, a harness that runs tests for it, and tool-and-test constraints instead of prose instructions. The 1.7B is a short-horizon assistant: quick on simple fixes, incapable of open-ended multi-file work, and the weakest instruction follower of the pair. Both are honest small models with loud cards; the cards' structural claims are accurate, and the parts that need big iron to verify are the parts to hold at arm's length.

For deployment: 1.7B where memory and latency dominate and horizons are short, 4B for real agent loops if you have the VRAM or a Q4 quant, and in both cases give the thinking traces room and parse the last code block.