Qwen3.8-Flash-Next on MLX-serve, 1m context is released!

Releases·Tool Release

Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant

A developer released Slipstream, an open-source C++ Metal inference engine for Apple Silicon with native SSD expert streaming and speculative drafting. It runs the 95.5 GiB Qwen3.8-Flash-Next checkpoint at 41–52 tok/s on a 64GB M5 Pro Mac, a 1.76x speedup over a llama.cpp fork, with decode staying at 33–44 tok/s out to 130k context. An optional Swift KV-sparse variant of the model raises overall benchmark accuracy by 2.8% with no decode-speed cost.

A developer released Slipstream, an open-source C++ Metal inference engine for Apple Silicon with native SSD expert streaming and speculative drafting. It runs the 95.5 GiB Qwen3.8-Flash-Next checkpoint at 41–52 tok/s on a 64GB M5 Pro Mac, a 1.76x speedup over a llama.cpp fork, with decode staying at 33–44 tok/s out to 130k context. An optional Swift KV-sparse variant of the model raises overall benchmark accuracy by 2.8% with no decode-speed cost.

Products·launched

Qwen3.8-Flash-Next quantization recipe for single GB10/DGX Spark using Intel AutoRound and vLLM

A community recipe and repo for running Qwen3.8-Flash-Next on a single DGX Spark (GB10) device. It combines Intel AutoRound int4 quantization with additional uncalibrated int8/fp8 quantization of specific layers, served via vLLM with an fp8 ngram table offloaded to SSD or an RDMA server. It lets local-LLM hobbyists fit the model on constrained hardware at usable speeds.

A community recipe and repo for running Qwen3.8-Flash-Next on a single DGX Spark (GB10) device. It combines Intel AutoRound int4 quantization with additional uncalibrated int8/fp8 quantization of specific layers, served via vLLM with an fp8 ngram table offloaded to SSD or an RDMA server. It lets local-LLM hobbyists fit the model on constrained hardware at usable speeds.

Social·Community

Qwen Flash Next MTP work restarted

A developer restarted multi-token prediction (MTP) support for Qwen3.8 Flash Next in llama.cpp, pointing users to ggml-org's GGUF quants on Hugging Face and an open pull request. The work remains in progress.

A developer restarted multi-token prediction (MTP) support for Qwen3.8 Flash Next in llama.cpp, pointing users to ggml-org's GGUF quants on Hugging Face and an open pull request. The work remains in progress.

Releases·Model Release

WinterMix releases 3-bit Qwen3.5-122B MLX build with long-context coherence

A developer released WinterMix, a 3-bit quantization of Qwen3.5-122B-A10B in native MLX, sized at 59 GiB. The build claims best-in-class long-context coherence, outperforming UnSloth's Q3_K_XL GGUF in perplexity at 20k+ context, thanks to a new annealing process on reasoning traces.

A developer released WinterMix, a 3-bit quantization of Qwen3.5-122B-A10B in native MLX, sized at 59 GiB. The build claims best-in-class long-context coherence, outperforming UnSloth's Q3_K_XL GGUF in perplexity at 20k+ context, thanks to a new annealing process on reasoning traces.

Releases·Tool Release

Qwen3.8-Flash-Next MTP merged into ik_llama.cpp main: 45→90 tok/s on 5090+128GB, works down to 12GB 4070

ik_llama.cpp merged qwen4exp multi-token prediction support for Qwen3.8-Flash-Next (PR #2369), authored by the poster and tested by four reviewers on their own hardware. The model drafts its own next tokens with a 2.6B MTP head and verifies them, keeping output identical while lifting throughput from 45 to 90 tok/s on a 5090 with 128GB RAM. The merge matters because community threads previously believed MTP for this model only existed as an unsloth fork PR.

ik_llama.cpp merged qwen4exp multi-token prediction support for Qwen3.8-Flash-Next (PR #2369), authored by the poster and tested by four reviewers on their own hardware. The model drafts its own next tokens with a 2.6B MTP head and verifies them, keeping output identical while lifting throughput from 45 to 90 tok/s on a 5090 with 128GB RAM. The merge matters because community threads previously believed MTP for this model only existed as an unsloth fork PR.