Qwen3.8-Flash runs 1.7× faster locally with MTP, reaching 170 tokens/s on RTX PRO 6000

Releases·Tool Release

Run Qwen3.8+Flash-Next and tiny models on Apple Silicon up to 3x faster with MLX.fast fork ishizuki

A developer shared ishizuki, a GitHub project enabling faster local inference of Qwen3.8+Flash-Next and small models on Apple Silicon, claiming up to 3x speedups. The author says they previously topped MLX.fast leaderboards on chips below M5 and is inviting community contributions.

A developer shared ishizuki, a GitHub project enabling faster local inference of Qwen3.8+Flash-Next and small models on Apple Silicon, claiming up to 3x speedups. The author says they previously topped MLX.fast leaderboards on chips below M5 and is inviting community contributions.

Social·Community

Community pushes Qwen3.8-Flash-Next to ~240 tok/s decode on a single RTX 6000 Pro via patched sglang

A Reddit user reproduced jpezzulli's patched sglang build reaching 178 tok/s on Qwen3.8-Flash-Next NVFP4, then used further optimizations to hit nearly 240 tok/s decode on a single RTX 6000 Pro Max-Q at 300W. The patch applies to the main sglang library and targets the RadixArk Hugging Face checkpoint, approaching the model's ~280 tok/s theoretical bandwidth limit.

A Reddit user reproduced jpezzulli's patched sglang build reaching 178 tok/s on Qwen3.8-Flash-Next NVFP4, then used further optimizations to hit nearly 240 tok/s decode on a single RTX 6000 Pro Max-Q at 300W. The patch applies to the main sglang library and targets the RadixArk Hugging Face checkpoint, approaching the model's ~280 tok/s theoretical bandwidth limit.

Developments·Benchmark

Qwen3.8-Flash-Next-oQ4e-mtp: 45 tok/s on M4 Max, 25 tok/s on M2 Ultra for local inference — llm-bench.io

llm-bench.io published local inference benchmarks for a quantized Qwen3.8-Flash-Next build on Apple Silicon, reporting 45 tok/s on an M4 Max and 25 tok/s on an M2 Ultra. The build is quantized to oQ4e with multi-token prediction and delivers speeds similar to Qwen3.8 27B on Apple Silicon.

llm-bench.io published local inference benchmarks for a quantized Qwen3.8-Flash-Next build on Apple Silicon, reporting 45 tok/s on an M4 Max and 25 tok/s on an M2 Ultra. The build is quantized to oQ4e with multi-token prediction and delivers speeds similar to Qwen3.8 27B on Apple Silicon.

Releases·Model Releasefeatured

Nvidia serves Qwen 3.8 2.4T at 288k tokens/s on GB300 NVL72

Nvidia published a blog demonstrating serving the Qwen 3.8 2.4T parameter model on its GB300 NVL72 system, achieving over 4K tokens per second per GPU and 288k tokens/s aggregate in FP8. The post highlights Day-0 performance without additional tuning, with further optimizations expected via NVFP4 precision.

Nvidia published a blog demonstrating serving the Qwen 3.8 2.4T parameter model on its GB300 NVL72 system, achieving over 4K tokens per second per GPU and 288k tokens/s aggregate in FP8. The post highlights Day-0 performance without additional tuning, with further optimizations expected via NVFP4 precision.

Releases·Tool Release

akopytko releases Blackwell-native NVFP4 quant of Qwen3.8 27B, 50% faster than Q4

A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.

A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.