Releases·Tool Release
An UltraFast Qwen3.8 Flash recipe: 74 tok/s, 212 tok/s aggregate on one DGX Spark
A developer published an open vLLM serving recipe for Qwen3.8 Flash on Nvidia's DGX Spark (GB10), reaching 74 tok/s single-stream and 212 tok/s aggregate across 8 streams — well above the 35–45 tok/s typical for single-Spark setups. The speed comes from serving-path optimizations, including a 65K-id draft vocab and W4A16/FP8/INT8 quantization, while quality matches the original checkpoint within noise.
6 engagement1 sourceMon, Sep 28, 2026, 08:09 PM
AI Overview
- The open-source vLLM recipe hits 74.1 tok/s single-stream and 212.2 tok/s at 8 streams on a GB10 DGX Spark, with full 262K context
- Cold prefill reaches 4,016 tok/s at 16K tokens versus 1,171 on the forked Saren-Arterius AutoRound recipe, a 243% increase
- Speed comes from the model's MTP head at 3.65-3.94 tokens per verify step, a draft vocab cut from 248K to 65K ids, and W4A16 AutoRound MoE quantization
- Decode steps dropped from 68.3 ms to 52.3 ms on an agent coding workload; a cached coding prompt starts replying in about 0.57 s
- Quality matches the original checkpoint: 93.1-93.3% on a fixed 492-question suite, with 0.06-point top-1 drift inside a 0.15-point noise band
- The model uses about 71 GiB with a 16 GB KV pool, drawing 35-37 W median while generating and peaking near 70 W, with no throttling
- The 65K draft vocab, built from English and code, drafts CJK output slower, though quality stays unaffected since the full model verifies every token
Related
Products·launched
Qwen3.8-Flash-Next quantization recipe for single GB10/DGX Spark using Intel AutoRound and vLLM
A community recipe and repo for running Qwen3.8-Flash-Next on a single DGX Spark (GB10) device. It combines Intel AutoRound int4 quantization with additional uncalibrated int8/fp8 quantization of specific layers, served via vLLM with an fp8 ngram table offloaded to SSD or an RDMA server. It lets local-LLM hobbyists fit the model on constrained hardware at usable speeds.
A community recipe and repo for running Qwen3.8-Flash-Next on a single DGX Spark (GB10) device. It combines Intel AutoRound int4 quantization with additional uncalibrated int8/fp8 quantization of specific layers, served via vLLM with an fp8 ngram table offloaded to SSD or an RDMA server. It lets local-LLM hobbyists fit the model on constrained hardware at usable speeds.
2 engagement1 sourceredditMon, Aug 31, 2026, 02:04 PM
Releases·Tool Release
Developer's INT8 vLLM fork hits 972 tok/s TG on Qwen3.8 27B with 4x MI100
A developer released a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a GPTQ INT8 quant with DFlash2. On a 4x MI100 rig costing $6.5k, it achieves 972 tok/s time-to-first-token and 5,680 tokens/s per prompt, up from 15 tok/s on stock vLLM. This enables older INT8-centric hardware to run modern models efficiently.
A developer released a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a GPTQ INT8 quant with DFlash2. On a 4x MI100 rig costing $6.5k, it achieves 972 tok/s time-to-first-token and 5,680 tokens/s per prompt, up from 15 tok/s on stock vLLM. This enables older INT8-centric hardware to run modern models efficiently.
57 engagement1 sourceredditWed, Aug 26, 2026, 08:54 PM
Releases·Tool Releasefeatured
Developer pushes Qwen3.8-27B to 381 tps on RTX 3090 with DFlash2 drafting
A developer iteratively optimized an inference engine for Qwen3.8-27B on an RTX 3090, reaching 381 tokens per second when the model reproduces its own context, and 133 tps on real chat prompts. The engine uses fp8 KV cache, DFlash2 drafting, and lookup-augmented drafting, achieving 15 of 16 tokens accepted per verify step on document-quoting workloads.
A developer iteratively optimized an inference engine for Qwen3.8-27B on an RTX 3090, reaching 381 tokens per second when the model reproduces its own context, and 133 tps on real chat prompts. The engine uses fp8 KV cache, DFlash2 drafting, and lookup-augmented drafting, achieving 15 of 16 tokens accepted per verify step on document-quoting workloads.
Updated Thu, Aug 20, 2026, 08:15 PM — Latest update pushes single-request speed to 381 tps on context reproduction and 133 tps on real chat prompts.
300626 engagement3 sourcesredditThu, Aug 20, 2026, 12:00 AM
Social·Community
Swift 1.5 on veloGB10, ~110 tok/s on 2× DGX Spark: xhigh beats base Flash-Next at medium on vLLM
A Reddit user benchmarks UkisAI's Swift 1.5 (a reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 engine across two DGX Sparks, hitting ~110 tok/s single-stream decode and, at xhigh effort, the best score yet on their private agentic coding battery. They also patch the velo loader so 12 of 14 community EXL3 packs (which shard the 51B n-gram table into 128 tensors) load as-is, and file the issue upstream.
A Reddit user benchmarks UkisAI's Swift 1.5 (a reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 engine across two DGX Sparks, hitting ~110 tok/s single-stream decode and, at xhigh effort, the best score yet on their private agentic coding battery. They also patch the velo loader so 12 of 14 community EXL3 packs (which shard the 51B n-gram table into 128 tensors) load as-is, and file the issue upstream.
23 engagement1 sourceredditMon, Oct 5, 2026, 03:04 PM
Releases·Tool Release
Dev's vLLM MXFP4 build runs NVFP4 Qwen3.8-27B on AMD R9700 at 5,809 tok/s prefill, 276 tok/s decode
A developer extended their MXFP4 vLLM build on the Radiance image with online NVFP4-to-MXFP4 conversion, letting NVFP4-quantized models use the MXFP4 fast path on AMD R9700 GPUs. Running Unsloth's Qwen3.8-27B NVFP4 quant, they measured 5,809 tok/s prefill and 276 tok/s decode via their BetterBench harness.
A developer extended their MXFP4 vLLM build on the Radiance image with online NVFP4-to-MXFP4 conversion, letting NVFP4-quantized models use the MXFP4 fast path on AMD R9700 GPUs. Running Unsloth's Qwen3.8-27B NVFP4 quant, they measured 5,809 tok/s prefill and 276 tok/s decode via their BetterBench harness.
69 engagement1 sourceredditTue, Sep 15, 2026, 08:51 PM