Releases·Tool Release
Qwen3.8-Flash-Next at 1M context on Strix Halo: 38 tok/s decode, 18 min prefill (halogen 0.12.0)
The halogen-flash-server maintainer released 0.12.0 fixing throughput degradation at long context, responding to user feedback. Serving Qwen3.8-Flash-Next at 1M context on a Ryzen AI Max+ 395 (128 GB), decode at 1,004,581 tokens rose from 27.3 to 38.3 tok/s and 1M prefill dropped from 21.2 to 17.9 minutes cold. The release enables 1M-context serving on a single consumer machine via a YaRN rope-scaling configuration.
7 engagement1 sourceSat, Sep 19, 2026, 09:49 PM
AI Overview
- Halogen 0.12.0 lifts decode at 1,004,581 tokens from 27.3 to 38.3 tok/s on Ryzen AI Max+ 395 with 128 GB, using the default speculative drafter.
- 1M prefill rises from 790 to 937 tok/s, cutting cold prefill from 21.2 to 17.9 minutes; 258k decode reaches 45.0 tok/s and prefill 1,114 tok/s.
- Running 1M context requires HALOGEN_ROPE_YARN=4 and HALOGEN_CTX=1048576 flags on the halogen podman image and the 128 GB machine.
- Cached follow-up turns at 1M reach first token in about 0.55 s; the ten-prompt 32k served mean was unchanged in 0.12.0.
Related
Releases·Model Release
Yamz Labs runs Qwen3.8-Flash-Next (125B MoE) on a single Strix Halo mini PC at 44-59 tok/s, open-weights and open engine
Yamz Labs released 95 GB EXL3 quantized weights for Qwen3.8-Flash-Next, a 125B MoE model with 6B active parameters, plus a new open version of its Kyojin inference engine built on ExLlamaV3. On a single AMD Strix Halo mini PC (Ryzen AI Max+ 395, 128 GB), it runs at 44-59 tok/s with speculative decoding and ~1,400 tok/s prefill. Practitioners get a full local inference stack for a large open model on consumer hardware, though Halogen 0.16.2 currently outperforms it on speed.
Yamz Labs released 95 GB EXL3 quantized weights for Qwen3.8-Flash-Next, a 125B MoE model with 6B active parameters, plus a new open version of its Kyojin inference engine built on ExLlamaV3. On a single AMD Strix Halo mini PC (Ryzen AI Max+ 395, 128 GB), it runs at 44-59 tok/s with speculative decoding and ~1,400 tok/s prefill. Practitioners get a full local inference stack for a large open model on consumer hardware, though Halogen 0.16.2 currently outperforms it on speed.
151 engagement1 sourceredditMon, Oct 5, 2026, 03:25 PM
Social·Community
Reddit users say hardware shortage pushes Local LLM hobbyists to tweak inference, quantization, and custom llama.cpp forks
A Reddit essay argues the current hardware shortage is driving a renaissance in the Local LLM community, forcing hobbyists to optimize inference engines, quantization, and architectures instead of buying more compute. The author cites forked llama.cpp builds and halogen-flash-server for Strix Halo that roughly doubled decode and multiplied prefill speed for Qwen 3.8 Flash Next, and likens the moment to the early-web era of forum-driven, end-to-end learning.
A Reddit essay argues the current hardware shortage is driving a renaissance in the Local LLM community, forcing hobbyists to optimize inference engines, quantization, and architectures instead of buying more compute. The author cites forked llama.cpp builds and halogen-flash-server for Strix Halo that roughly doubled decode and multiplied prefill speed for Qwen 3.8 Flash Next, and likens the moment to the early-web era of forum-driven, end-to-end learning.
1.1k engagement1 sourceredditSun, Sep 13, 2026, 09:56 AM
Developments·Benchmark
Community benchmark tests Qwen3.8-Flash-Next NVFP4 on local agentic coding
A community benchmark run compares the NVFP4-quantized Qwen3.8-Flash-Next against 27B-class and other local models on agentic coding tasks. Results are published on a public local-LLM benchmark site, with a companion post on r/BlackwellPerformance covering single-RTX Pro 6000 performance. It matters to practitioners choosing quantized models for local coding agents.
A community benchmark run compares the NVFP4-quantized Qwen3.8-Flash-Next against 27B-class and other local models on agentic coding tasks. Results are published on a public local-LLM benchmark site, with a companion post on r/BlackwellPerformance covering single-RTX Pro 6000 performance. It matters to practitioners choosing quantized models for local coding agents.
156 engagement2 sourcesredditFri, Aug 28, 2026, 05:30 PM
Releases·Tool Release
Qwen3.8-Flash-Next MTP merged into ik_llama.cpp main: 45→90 tok/s on 5090+128GB, works down to 12GB 4070
ik_llama.cpp merged qwen4exp multi-token prediction support for Qwen3.8-Flash-Next (PR #2369), authored by the poster and tested by four reviewers on their own hardware. The model drafts its own next tokens with a 2.6B MTP head and verifies them, keeping output identical while lifting throughput from 45 to 90 tok/s on a 5090 with 128GB RAM. The merge matters because community threads previously believed MTP for this model only existed as an unsloth fork PR.
ik_llama.cpp merged qwen4exp multi-token prediction support for Qwen3.8-Flash-Next (PR #2369), authored by the poster and tested by four reviewers on their own hardware. The model drafts its own next tokens with a 2.6B MTP head and verifies them, keeping output identical while lifting throughput from 45 to 90 tok/s on a 5090 with 128GB RAM. The merge matters because community threads previously believed MTP for this model only existed as an unsloth fork PR.
248 engagement1 sourceredditThu, Sep 3, 2026, 04:30 PM