Releases·Tool Release
Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant
A developer released Slipstream, an open-source C++ Metal inference engine for Apple Silicon with native SSD expert streaming and speculative drafting. It runs the 95.5 GiB Qwen3.8-Flash-Next checkpoint at 41–52 tok/s on a 64GB M5 Pro Mac, a 1.76x speedup over a llama.cpp fork, with decode staying at 33–44 tok/s out to 130k context. An optional Swift KV-sparse variant of the model raises overall benchmark accuracy by 2.8% with no decode-speed cost.
54 engagement1 sourceThu, Oct 1, 2026, 08:21 PM
AI Overview
- Developer releases Slipstream, a compiled C++ Metal engine with SSD expert streaming and MTP+PLD speculation, cutting prefill staging latency 28% and lifting tool-calling decode from 5.6 to 45+ tok/s
- Across 3,086 live requests, Slipstream decode stays at 33–44 tok/s out to 130,000 tokens on the 48-layer DeltaNet hybrid model on an M5 Pro (64 GB)
- Optional Swift-KV-sparse variant scores 70.3% vs 67.6% over 145 eval problems (GPQA Diamond +8.6%) at 44.4 tok/s, using spliced Q8 donor backbones
- Slipstream serves an OpenAI-compatible API, requires iogpu.wired_limit_mb=59392 on 64 GB Macs, and needs no CUDA/ROCm ports yet; author invites community porting help
- Slipstream's primitives (512-route MoE streaming, QSA attention, Metal n-gram gathers) target day-one local Qwen4 support; credits Splash (Incoai) design foundation
Related
Products·building
Qwen Flash Next Q4 running on Mac Mini M5 64GB with SSD streaming
A side project running Qwen Flash Next Q4 locally on a Mac Mini M5 64GB, with performance tuning for prompt processing and decode speeds. It uses an expert carousel buffer and dual external SSDs for parallel reads to boost throughput, targeting local LLM enthusiasts who want fast inference on consumer hardware.
A side project running Qwen Flash Next Q4 locally on a Mac Mini M5 64GB, with performance tuning for prompt processing and decode speeds. It uses an expert carousel buffer and dual external SSDs for parallel reads to boost throughput, targeting local LLM enthusiasts who want fast inference on consumer hardware.
14 engagement1 sourceredditMon, Oct 5, 2026, 06:44 AM
Social·Community
Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked!
A developer optimizing Qwen Flash Next on a Mac Mini M5 64GB via SSD streaming implemented an n+2 lookahead predictor, suggested by the LocalLLama community, that loads experts two layers ahead instead of one. The change delivered +8% throughput on chat and +4.2% on code, stacking on existing optimizations.
A developer optimizing Qwen Flash Next on a Mac Mini M5 64GB via SSD streaming implemented an n+2 lookahead predictor, suggested by the LocalLLama community, that loads experts two layers ahead instead of one. The change delivered +8% throughput on chat and +4.2% on code, stacking on existing optimizations.
119 engagement1 sourceredditTue, Oct 13, 2026, 09:57 AM
Social·Community
Qwen3.8-Flash-Next breaks 94% on cupel benchmark in early local tests
A developer reports early local testing of Qwen3.8-Flash-Next on an M4 Max 128GB Studio, achieving over 94% on the cupel benchmark—the first model this year to do so. The model uses the qwen4_exp architecture, which is not yet fully supported by oMLX, requiring workarounds.
A developer reports early local testing of Qwen3.8-Flash-Next on an M4 Max 128GB Studio, achieving over 94% on the cupel benchmark—the first model this year to do so. The model uses the qwen4_exp architecture, which is not yet fully supported by oMLX, requiring workarounds.
195 engagement1 sourceredditThu, Aug 27, 2026, 12:34 PM
Releases·Tool Release
llama.cpp v0.4.0 adds Qwen3.8-Flash-Next and Nemotron-3-Puzzle-75B-A9B support, lazy tensor loading, ggml 0.23.0
llama.cpp shipped v0.4.0, adding initial support for Qwen3.8-Flash-Next and Nemotron-3-Puzzle models. The release also brings on-demand tensor reading, per-slot server context limits, video input options, and a ggml update to 0.23.0 featuring major sparse flash attention and RDMA work.
llama.cpp shipped v0.4.0, adding initial support for Qwen3.8-Flash-Next and Nemotron-3-Puzzle models. The release also brings on-demand tensor reading, per-slot server context limits, video input options, and a ggml update to 0.23.0 featuring major sparse flash attention and RDMA work.
6 engagement1 sourcegithubFri, Sep 4, 2026, 07:56 PM
Social·Community
Reddit user benchmarks Qwen3.8 27B and Flash-Next at 20k–262k context on RTX 5090 + 3090 with llama.cpp
Practitioners shared llama.cpp benchmarks and tuning results for Qwen3.8-27B and Qwen3.8-Flash-Next GGUF quantizations on consumer hardware. A detailed dual-GPU (RTX 5090 + 3090) test found the 5090 alone beats the 2-GPU split for 27B under 128k context, identified a lazy-mode prefill regression on RAM-offloaded MoE, and showed MTP speculative decoding at draft 3 lifts decode ~12%. A second user ran Flash-Next on an M4 Pro Mac at 131k context after llama.cpp flag tuning.
Practitioners shared llama.cpp benchmarks and tuning results for Qwen3.8-27B and Qwen3.8-Flash-Next GGUF quantizations on consumer hardware. A detailed dual-GPU (RTX 5090 + 3090) test found the 5090 alone beats the 2-GPU split for 27B under 128k context, identified a lazy-mode prefill regression on RAM-offloaded MoE, and showed MTP speculative decoding at draft 3 lifts decode ~12%. A second user ran Flash-Next on an M4 Pro Mac at 131k context after llama.cpp flag tuning.
Updated Sun, Sep 27, 2026, 08:32 PM — A Mac M4 Pro user reports Flash-Next outperforming the dense 27B after llama.cpp flag tuning, holding 131k context witho
152 engagement2 sourcesredditMon, Sep 21, 2026, 08:39 PM