Qwen3.8-27B Benchmark on NVIDIA DGX Spark GB10

1. Hardware Constraints & Theoretical Ceiling#

The GB10 is bandwidth-bound for single-stream dense-model decode. With 128 GB of unified memory and ~225-273 GB/s effective bandwidth, the numbers work out like this:

• 27B parameter model at NVFP4 (W4A4): ~20 GB read per forward step
• Theoretical ceiling: 225 GB/s ÷ 20 GB ≈ 11-12 tok/s without speculative decoding
• With speculative decoding accepting 3.3-4.9 tokens per verify step: 34-47 tok/s achievable

Put another way, the DGX Spark runs at about 92% of the physical wall in optimal configurations. The headroom that's left comes from better draft models (higher acceptance rates), not from faster engines.

2. Engine & Configuration Matrix#

Prefill throughput (14k prompt):

• SGLang + DSpark: ~1,570 tok/s
• llama.cpp: ~600 tok/s

Concurrent throughput (8 streams):

• SGLang + DSpark: ~94-109 tok/s aggregate
• vLLM: 289-313 tok/s aggregate (strength is batching, not single-stream)

NVFP4 vs FP8 Quantization (vLLM 0.27.1, 16 concurrent requests)#

NVFP4 is consistently 30–34% faster than FP8 on vLLM 0.27.1. The reason is lower memory traffic and smaller weight tensors. The gap widens in memory-bandwidth-bound decode scenarios.

3. Speculative Decoding: DSpark vs MTP#

Which speculative decoding method you want depends on what you're doing with the model:

Why DSpark wins on code/math: DSpark drafts a 7-token block in one pass using a small 1.4B drafter, while MTP drafts one token at a time. DSpark acceptance is higher for structured content (3.3-4.9 vs ~3-3.5).

Why MTP wins on prose: Free-form text has lower acceptance for block drafting (1.25-1.8), making the single-token MTP head more efficient. MTP acceptance on prose: ~0.35-0.52 per token vs DSpark's ~0.09-0.26.

4. Multi-Node Scaling (TP=2)#

The DGX Spark has no NVLink between nodes — TP=2 crosses ConnectX-7 (200 Gb/s RDMA). Despite that, scaling is meaningful:

Scaling factor: 1.26x-1.51x on 27B at 6-10K context (the regime agentic workloads actually live in). At longer contexts, scaling improves because the interconnect overhead becomes negligible relative to compute.

Key insight: For dense models like 27B, multi-node TP on Ethernet is practical and delivers near-linear scaling for decode. The ConnectX-7 link (~546 GB/s aggregate on 2 nodes) doubles effective bandwidth.

5. Quality Preservation#

NVFP4 quantization is bitwise-lossless for speculative decoding by construction. The DSpark draft model is trained against the quantized target, so the speculative decode path does not introduce quality degradation. MixedInt4-AutoRound recipes achieve 99.5% MMLU recovery while fitting comfortably on a single DGX Spark.

6. Practical Recommendations#

For Coding / Agentic Workloads#

Use SGLang + NVFP4 + DSpark:

docker run --gpus all --shm-size 32g -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=" --ipc=host \
  lmsysorg/sglang:qwen38-27b \
  sglang serve --trust-remote-code \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --mem-fraction-static 0.50 \
  --attention-backend flashinfer \
  --chunked-prefill-size 8192 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark \
  --host 0.0.0.0 --port 30000

For Long-Context / Prose Workloads#

Use vLLM + NVFP4 + MTP or llama.cpp + MTP for better free-form text acceptance. Keep --mem-fraction-static at 0.50 or lower to avoid CUDA graph capture failures.

For Maximum Single-Node Speed#

Use the veloGB10 engine (Rust-based, GB10-specific kernels). It achieves ~40–42 tok/s single-node (Qwen3.6-27B baseline) and ~45–53 tok/s on TP=2, at the cost of a less mature serving ecosystem. See sf-stav/veloGB10 on GitHub.

For Reproducible Deployment#

Use the hasso5703/dgx-spark-qwen38 one-command Docker setup (SGLang + NVFP4 + DSpark) for a pinned, reproducible baseline.

7. Context Length & Memory#

At NVFP4 with FP8 KV cache, a full 1M-token sequence costs ~33 GB of KV. Two full-length 1M requests fit simultaneously on the 128 GB unified memory.

8. Known Pitfalls#

1. --mem-fraction-static 0.95 will freeze the DGX Spark. The 128 GB is unified; reserving 113 GB leaves the host with nothing. CUDA graph capture fails silently and decode falls back to eager mode. Use 0.50 or lower.
2. Prefix caching breaks with speculative decoding. Cache hit rates drop to 0% and tool calls may fail. Disable prefix caching when using DSpark or MTP.
3. DSpark acceptance varies wildly by content. Math/code: 3.3-4.9 tok/step. German prose: 1.25-1.8 tok/step. Always benchmark with your actual workload, not synthetic benchmarks.
4. vLLM strengths (batching, paged KV) do nothing for single-stream. At batch=1 the GB10 is fully bandwidth-bound. Engine overhead dominates — llama.cpp has less overhead than vLLM for single-user scenarios.
5. All sm_121 kernels are young. Performance gaps between engines will shift with driver and framework releases. Pin container image digests for reproducible benchmarks.