DFlash2 on a DGX Spark: benchmarking Qwen3.8-27B (NVFP4)
I’ve been evaluating DFlash2 speculative decoding for Qwen3.8-27B (NVFP4) on a single NVIDIA DGX Spark (GB10) in the homelab — the successor to the Qwen3.6-27B DFlash setup that was my daily driver, kept behind the same OpenAI-compatible contract the agents already use.
The stack
- Model:
RadixArk/Qwen3.8-27B-NVFP4 - Draft model:
z-lab/Qwen3.8-27B-DFlash2, 8 draft tokens - Server: SGLang (patched
qwen38-dflash2:v1.2.2image with the NVFP4lm_headfix), FlashInfer attention,--mem-fraction-static 0.5 - Host: DGX Spark GB10, 121 GiB unified memory,
aarch64 - Contract preserved: the same OpenAI-compatible endpoint the agents already use, so nothing downstream had to change
First boot takes ~9 minutes for torch compile and CUDA graph capture, so don’t panic when health checks lag.
DFlash2 replaces MTP (which used 3 speculative tokens). The upstream repo claims roughly 2–2.5x over its own vLLM+MTP baseline, with decode reported anywhere from 24 to 53 tok/s depending on workload — community numbers I wanted to reproduce locally.
First measurements
Three back-to-back runs, same workload: 20 requests, concurrency 2, ~512 input / 128 output tokens each.
| Run | Wall (s) | Req/s | Output tok/s | TTFT p50 (ms) | Latency p50 (ms) |
|---|---|---|---|---|---|
| 01 | 19.907 | 1.005 | 128.60 | 229.3 | 1990.0 |
| 02 | 19.919 | 1.004 | 128.52 | 229.5 | 1990.0 |
| 03 | 19.930 | 1.004 | 128.45 | 230.8 | 1994.0 |
Reading the numbers
The wall time is no accident: 20 requests at concurrency 2 is 10 sequential waves of 2, and 10 × ~1.99 s ≈ 19.9 s. Each request breaks down as ~229 ms to first token, then ~1761 ms to stream out 128 tokens — about 72.7 tokens/s per stream. Two streams in flight gives an aggregate of ~128.5 output tokens/s, comfortably above the community decode range for this recipe and a clear step up from the Qwen3.6-27B DFlash box it replaces.
The most reassuring part isn’t the speed, it’s the consistency: p95 sits within half a percent of p50 on every run, and the three runs barely differ. No thermal throttling, no contention, fully reproducible.
What this run does not tell me
Concurrency 2 measures latency, not peak throughput. At two streams the GB10
is idle a good chunk of the time, so 128 tok/s is a floor, not a ceiling. The
next step is a concurrency sweep — 4, 8, 16, 32 — watching for the knee where
output_tokens_per_second stops climbing and TTFT starts to rise. That knee is
the real capacity of this box.
Still on the list: regression-testing agent tool calls, structured output and
long-context behaviour (the recipe runs --allow-auto-truncate, and DFlash2
doesn’t play with YaRN context extension), then a like-for-like benchmark
against the old MTP stack. If it regresses, docker compose down and the MTP
setup comes back.
More runs to follow as I push the concurrency up.