Faster than Light in Air: 8-22 tg/s Qwen3.8-Flash-Next (Q4/Q4ish) on a 32GB M4 MacBook Air

Releases·Tool Release

MoEspresso 3 runs Qwen3.8-Flash-Next (125B) at 12-15 tok/s on a 2021 32GB M1 Max, scoring 84.3% on a 48-question benchmark

The author of MoEspresso released version 3 of their homegrown inference engine, which runs the 125B Qwen3.8-Flash-Next on a 2021 M1 Max with 32 GB unified memory at 12-15 decode tokens per second. The key technique is Cache-Prior, which biases decode toward experts already resident in memory, paired with IQ2_K quantization and SSD streaming when experts exceed memory. In the author's 48-question benchmark, the locally run model scores 84.3%, near hosted frontier models.

The author of MoEspresso released version 3 of their homegrown inference engine, which runs the 125B Qwen3.8-Flash-Next on a 2021 M1 Max with 32 GB unified memory at 12-15 decode tokens per second. The key technique is Cache-Prior, which biases decode toward experts already resident in memory, paired with IQ2_K quantization and SSD streaming when experts exceed memory. In the author's 48-question benchmark, the locally run model scores 84.3%, near hosted frontier models.

Products·building

PagedServe: paged LLM inference engine for Apple Silicon reducing attention memory

PagedServe is an open-source LLM inference runtime for Apple Silicon that uses fixed-size physical KV blocks with logical-to-physical block tables to avoid KV-cache fragmentation. It targets developers running LLMs on Macs, cutting temporary memory overhead in attention computation. The author reports 128x memory reduction for attention at a 2048-token context.

PagedServe is an open-source LLM inference runtime for Apple Silicon that uses fixed-size physical KV blocks with logical-to-physical block tables to avoid KV-cache fragmentation. It targets developers running LLMs on Macs, cutting temporary memory overhead in attention computation. The author reports 128x memory reduction for attention at a 2048-token context.

Releases·Tool Release

CompiledThoughts streams Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) off SSD at 9-10 tok/s on a 16 GB RTX 5060 Ti

CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.

CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.

Developments·Benchmark

Qwen3.8-Flash-Next-oQ4e-mtp: 45 tok/s on M4 Max, 25 tok/s on M2 Ultra for local inference — llm-bench.io

llm-bench.io published local inference benchmarks for a quantized Qwen3.8-Flash-Next build on Apple Silicon, reporting 45 tok/s on an M4 Max and 25 tok/s on an M2 Ultra. The build is quantized to oQ4e with multi-token prediction and delivers speeds similar to Qwen3.8 27B on Apple Silicon.

llm-bench.io published local inference benchmarks for a quantized Qwen3.8-Flash-Next build on Apple Silicon, reporting 45 tok/s on an M4 Max and 25 tok/s on an M2 Ultra. The build is quantized to oQ4e with multi-token prediction and delivers speeds similar to Qwen3.8 27B on Apple Silicon.

Products·launched

SSD-streamed MoE inference for Qwen3.6-35B-A3B on 16GB M1 Pro

A modified runtime (ds4) that runs the Qwen3.6-35B-A3B mixture-of-experts model on a 16 GB M1 Pro by streaming expert weights from SSD. It keeps a bounded cache of routed experts in unified memory and loads misses from disk, enabling inference on hardware that cannot hold the full model. Targets developers who want to run large MoE models locally on memory-constrained Apple Silicon machines.

A modified runtime (ds4) that runs the Qwen3.6-35B-A3B mixture-of-experts model on a 16 GB M1 Pro by streaming expert weights from SSD. It keeps a bounded cache of routed experts in unified memory and loads misses from disk, enabling inference on hardware that cannot hold the full model. Targets developers who want to run large MoE models locally on memory-constrained Apple Silicon machines.