Releases·Tool Release
Go-LLM repo documents serving Qwen3-Flash-Next with vLLM on 2x RTX 3090 via patched GGUF fork
A developer published Go-LLM, a public guide/config/scripts repo for serving Qwen3-Flash-Next with vLLM, covering an official vLLM nightly path on Blackwell hardware and a patched vLLM 0.29.0 GGUF fork on dual RTX 3090s. The post includes benchmark comparisons between Q4_K_XL and IQ4_XS quantizations. Practitioners gain concrete launch flags and quantization tradeoffs for running this model outside datacenter hardware.
6 engagement1 sourceTue, Sep 15, 2026, 09:27 AM
AI Overview
- Go-LLM repo (no weights) documents two serving paths: vLLM nightly on Blackwell (DGX Spark/SM121, NVFP4) and vLLM 0.29.0 GGUF fork on 2x RTX 3090 (sm_86)
- Benchmarks on the dual-3090 rig: Q4_K_XL hits 854-1007 tok/s prefill; IQ4_XS decodes ~40.5-40.9 tok/s, about 27% faster but ~2.5x slower prefill
- Both quantizations stay coherent at 128K context under best configs; IQ4_XS time-to-first-token ran ~2 minutes on long-context runs
- Author retracts an earlier ~6665 tok/s prefill figure as non-reproducible and says prior Q4_K_XL long-context degradation reports used non-best configs
- Stock PyPI vLLM will not serve the model on consumer cards, per the guide, which lists required patches and launch flags
Related
Releases·Tool Release
Qwen3.8-Flash-Next on 6x3090 / 6x4090 without NVLink: prefill 8-10x faster and long-context decode 2-3x faster than stock llama.cpp, binaries included
A developer published llama.cpp-multigpu, a performance fork of llama.cpp shipping binaries and Docker images that speeds up long-context inference of Qwen3.8-Flash-Next on 6x3090/6x4090 rigs without NVLink. Patched builds reach 2,111-7,403 t/s prefill at 250k context and 27-30 t/s per-session decode across 5 concurrent 262k sessions, versus stock's 2.3 t/s per session. The fork is temporary, CUDA-only, and validated by measurement rather than code review, so no upstream PRs are planned.
A developer published llama.cpp-multigpu, a performance fork of llama.cpp shipping binaries and Docker images that speeds up long-context inference of Qwen3.8-Flash-Next on 6x3090/6x4090 rigs without NVLink. Patched builds reach 2,111-7,403 t/s prefill at 250k context and 27-30 t/s per-session decode across 5 concurrent 262k sessions, versus stock's 2.3 t/s per session. The fork is temporary, CUDA-only, and validated by measurement rather than code review, so no upstream PRs are planned.
59 engagement1 sourceredditWed, Oct 7, 2026, 05:06 PM
Social·Community
4x RTX 3090 PCIe 4.0 x16 - advice? Qwen 3.8 Next Flash?
A Reddit user is planning a server upgrade from two RTX 3090 cards to four on a Threadripper Pro 5955WX with 128GB DDR4, asking how best to use the added VRAM. They currently serve Qwen 3.8 27b Q8 via vLLM to a few users and are weighing whether to stay with the current model, run smaller models alongside, or move to Qwen 3.8 Next Flash at Q4/Q5 quantization. The thread highlights practical tradeoffs self-hosters face when scaling multi-GPU inference on consumer hardware.
A Reddit user is planning a server upgrade from two RTX 3090 cards to four on a Threadripper Pro 5955WX with 128GB DDR4, asking how best to use the added VRAM. They currently serve Qwen 3.8 27b Q8 via vLLM to a few users and are weighing whether to stay with the current model, run smaller models alongside, or move to Qwen 3.8 Next Flash at Q4/Q5 quantization. The thread highlights practical tradeoffs self-hosters face when scaling multi-GPU inference on consumer hardware.
62 engagement1 sourceredditSat, Sep 19, 2026, 10:49 AM
Social·Community
Same Ling model, different long-context curve: INT4/vLLM vs Q5/llama.cpp on one Spark
A community benchmark on a 128GB DGX Spark compares Ling-3.0-flash served via vLLM (official INT4) against llama.cpp (Q5_K_M) with identical prompts. vLLM wins on short prompts, but llama.cpp sustains roughly 33 tok/s at 45K–90K tokens while vLLM falls to 4.6–7.9 tok/s — a relevant tradeoff for practitioners choosing serving stacks for long-context local inference.
A community benchmark on a 128GB DGX Spark compares Ling-3.0-flash served via vLLM (official INT4) against llama.cpp (Q5_K_M) with identical prompts. vLLM wins on short prompts, but llama.cpp sustains roughly 33 tok/s at 45K–90K tokens while vLLM falls to 4.6–7.9 tok/s — a relevant tradeoff for practitioners choosing serving stacks for long-context local inference.
7 engagement1 sourceredditThu, Sep 10, 2026, 04:52 AM
Social·Community
Swift 1.5 on veloGB10, ~110 tok/s on 2× DGX Spark: xhigh beats base Flash-Next at medium on vLLM
A Reddit user benchmarks UkisAI's Swift 1.5 (a reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 engine across two DGX Sparks, hitting ~110 tok/s single-stream decode and, at xhigh effort, the best score yet on their private agentic coding battery. They also patch the velo loader so 12 of 14 community EXL3 packs (which shard the 51B n-gram table into 128 tensors) load as-is, and file the issue upstream.
A Reddit user benchmarks UkisAI's Swift 1.5 (a reasoning-efficient fine-tune of Qwen3.8-Flash-Next) on sf-stav's veloGB10 engine across two DGX Sparks, hitting ~110 tok/s single-stream decode and, at xhigh effort, the best score yet on their private agentic coding battery. They also patch the velo loader so 12 of 14 community EXL3 packs (which shard the 51B n-gram table into 128 tensors) load as-is, and file the issue upstream.
23 engagement1 sourceredditMon, Oct 5, 2026, 03:04 PM
Releases·Tool Release
Developer releases R9V custom RDNA4 kernels, tripling Qwen3.8-Flash-Next speed on dual R9700s
A developer released R9V, a set of hand-tuned kernels for AMD RDNA4 (R9700) GPUs, plus quantized model packages on Hugging Face. Applied to vLLM-Radiance, the kernels push Qwen3.8-Flash-Next to 78 tok/s TG256 on two R9700s, roughly 3x faster than before. The release targets local-inference practitioners on AMD hardware and includes an early-stage inference engine for dense models.
A developer released R9V, a set of hand-tuned kernels for AMD RDNA4 (R9700) GPUs, plus quantized model packages on Hugging Face. Applied to vLLM-Radiance, the kernels push Qwen3.8-Flash-Next to 78 tok/s TG256 on two R9700s, roughly 3x faster than before. The release targets local-inference practitioners on AMD hardware and includes an early-stage inference engine for dense models.
49 engagement1 sourceredditMon, Aug 31, 2026, 01:13 AM