Dev's vLLM MXFP4 build runs NVFP4 Qwen3.8-27B on AMD R9700 at 5,809 tok/s prefill, 276 tok/s decode

Social·Communityfeatured

Local LLM inference: Qwen3.8 on 5090s, 7900 XTX at 262k context, dual R9700s, 4x RTX 3060, and Atlas 300I Duo cards

A wave of r/LocalLLaMA posts reports working local deployments of Qwen3.8-27B and Qwen3.8-Flash-Next on hardware from an iPhone-tethered MacBook to Huawei Ascend cards and an eBay 8x V100 server. Common levers are MTP/speculative decoding, low-bit quants, and runtime forks (llama.cpp, vLLM, exllamav3, Ninfer). Together they map where local, full-context agentic inference is actually usable.

A wave of r/LocalLLaMA posts reports working local deployments of Qwen3.8-27B and Qwen3.8-Flash-Next on hardware from an iPhone-tethered MacBook to Huawei Ascend cards and an eBay 8x V100 server. Common levers are MTP/speculative decoding, low-bit quants, and runtime forks (llama.cpp, vLLM, exllamav3, Ninfer). Together they map where local, full-context agentic inference is actually usable.

Releases·Tool Release

153 tok/s on 1x AMD Radeon R9700 running Qwen3.8 27b NVFP4, 470 tok/s @ 8 conc requests, Prefill @ 3,619 tok/s

A developer released optimizations for single AMD Radeon R9700 cards in their vLLM fork (radiance-vllm-mxfp4), roughly doubling performance across the board. Benchmarks with Unsloth's Qwen3.8 27B NVFP4 show up to 153 tok/s single-stream decode, 471 tok/s at 8 concurrent requests, and ~3,619 tok/s prefill.

A developer released optimizations for single AMD Radeon R9700 cards in their vLLM fork (radiance-vllm-mxfp4), roughly doubling performance across the board. Benchmarks with Unsloth's Qwen3.8 27B NVFP4 show up to 153 tok/s single-stream decode, 471 tok/s at 8 concurrent requests, and ~3,619 tok/s prefill.

Releases·Tool Release

akopytko releases Blackwell-native NVFP4 quant of Qwen3.8 27B, 50% faster than Q4

A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.

A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.

Releases·Tool Release

Developer releases R9V custom RDNA4 kernels, tripling Qwen3.8-Flash-Next speed on dual R9700s

A developer released R9V, a set of hand-tuned kernels for AMD RDNA4 (R9700) GPUs, plus quantized model packages on Hugging Face. Applied to vLLM-Radiance, the kernels push Qwen3.8-Flash-Next to 78 tok/s TG256 on two R9700s, roughly 3x faster than before. The release targets local-inference practitioners on AMD hardware and includes an early-stage inference engine for dense models.

A developer released R9V, a set of hand-tuned kernels for AMD RDNA4 (R9700) GPUs, plus quantized model packages on Hugging Face. Applied to vLLM-Radiance, the kernels push Qwen3.8-Flash-Next to 78 tok/s TG256 on two R9700s, roughly 3x faster than before. The release targets local-inference practitioners on AMD hardware and includes an early-stage inference engine for dense models.

A user running vllm-radiance on dual AMD R9700 GPUs asks whether better inference options exist for Qwen 3.8 27B FP8 on RDNA4, reporting strong throughput (5000 tok/s prefill, 130+ tok/s code generation) and seeking alternatives, particularly for the unsupported Qwen 3.8 Flash Next model.