Releases·Tool Release
CompiledThoughts streams Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) off SSD at 9-10 tok/s on a 16 GB RTX 5060 Ti
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.
105 engagement1 sourceSun, Sep 27, 2026, 10:13 PM
AI Overview
- CompiledThoughts' Inferred-Thoughts engine runs Qwen3.8-Flash-Next (176.9B params, 119 GiB NVFP4 GGUF) with 20 GiB in VRAM/RAM and 99 GiB on SSD
- Decode hits 9.06 tok/s (10.4 peak) at xhigh effort and 49.2 tok/s prefill on a 5.5k prompt, vs 4.9 tok/s for llama.cpp on the same machine
- VRAM holds 4.4 GiB dense weights plus 8.8 GiB hottest experts; 48.5 GiB routed experts and a 50.7 GiB n-gram table stay on Gen5 NVMe, ~270 MiB read per token
- Lookahead prefetch plus ~75% memory hit-rate on 480 experts/token (377 in memory, ~103 from SSD) enables the speed; NVFP4 matmuls run FP4xFP4 unpacked-free on tensor cores
- Also supports Qwen3.6-35B-A3B NVFP4 at 47.3 tok/s decode and 591 tok/s prefill on the same RTX 5060 Ti machine
- v1 ships an OpenAI-compatible server with tool calls and chat-template rendering, but is Blackwell (sm_120) only, Windows 11/WSL2 only, greedy decoding only
Related
Social·Community
Running Qwen 3.8 Next on 16 GB VRAM + 32 GB RAM: a llama.cpp setup guide for GPU-poor hardware
A Reddit user documents getting the large Qwen3.8-Flash-Next MoE model running on a low-end RTX 5060 Ti (16 GB) + 32 GB DDR5 system using a pruned REAP GGUF quant and llama.cpp flags. The N-gram/PLE embedding table (~29 GB) is lazily memory-mapped from SSD while experts, KV cache, and layers fit into VRAM and RAM, yielding ~18.3 tok/s decode at 64k context. The writeup offers a reproducible recipe for 'GPU-poor' local inference of models nominally far larger than the hardware.
A Reddit user documents getting the large Qwen3.8-Flash-Next MoE model running on a low-end RTX 5060 Ti (16 GB) + 32 GB DDR5 system using a pruned REAP GGUF quant and llama.cpp flags. The N-gram/PLE embedding table (~29 GB) is lazily memory-mapped from SSD while experts, KV cache, and layers fit into VRAM and RAM, yielding ~18.3 tok/s decode at 64k context. The writeup offers a reproducible recipe for 'GPU-poor' local inference of models nominally far larger than the hardware.
345 engagement2 sourcesredditMon, Sep 14, 2026, 08:24 AM
Social·Community
Running Qwen3.8-Flash-Next, a 125B MoE model, on one RTX 3090 Ti
Developer runs Qwen3.8-Flash-Next 125B MoE on a single RTX 3090 Ti
Developer runs Qwen3.8-Flash-Next 125B MoE on a single RTX 3090 Ti
0 engagement1 sourcerssSun, Oct 4, 2026, 01:25 AM
Releases·Tool Release
An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk
A developer extended llama.cpp's WebGPU backend to page MoE experts from disk into GPU slots, removing the need for per-model engines. On a MacBook Pro M4 Pro, this runs the 48.4 GB Qwen3-Next 80B-A3B GGUF in a Chrome tab at 2.9 tok/s, plus faster smaller MoE models. It is a proof of concept: prompt processing is slow, and it requires Chrome/Edge 137+ on Apple silicon.
A developer extended llama.cpp's WebGPU backend to page MoE experts from disk into GPU slots, removing the need for per-model engines. On a MacBook Pro M4 Pro, this runs the 48.4 GB Qwen3-Next 80B-A3B GGUF in a Chrome tab at 2.9 tok/s, plus faster smaller MoE models. It is a proof of concept: prompt processing is slow, and it requires Chrome/Edge 137+ on Apple silicon.
31 engagement1 sourceredditTue, Oct 13, 2026, 02:04 PM
Social·Community
Qwen 3.8 Flash Next (68GB IQ2_XS) hits 21 tok/s on RTX 3060 12GB + 16GB RAM via llama.cpp --moe-direct-io, bit-exact, no expert pruning
A developer implemented a --moe-direct-io feature in llama.cpp that predicts which MoE experts the next token needs and prefetches them, running Qwen 3.8 Flash Next (68GB IQ2_XS) at 20-24 tok/s on an RTX 3060 12GB with only 16GB DDR4 RAM. Output is bit-exact to stock llama.cpp, avoiding the expert-pruning tradeoff that made an earlier attempt unusable. The speedup is roughly 10-15x over stock mmap-based llama.cpp, which hit 1.4-2.1 tok/s on the same setup.
A developer implemented a --moe-direct-io feature in llama.cpp that predicts which MoE experts the next token needs and prefetches them, running Qwen 3.8 Flash Next (68GB IQ2_XS) at 20-24 tok/s on an RTX 3060 12GB with only 16GB DDR4 RAM. Output is bit-exact to stock llama.cpp, avoiding the expert-pruning tradeoff that made an earlier attempt unusable. The speedup is roughly 10-15x over stock mmap-based llama.cpp, which hit 1.4-2.1 tok/s on the same setup.
157 engagement1 sourceredditFri, Oct 9, 2026, 06:41 PM
Releases·Tool Release
Faster than Light in Air: 8-22 tg/s Qwen3.8-Flash-Next (Q4/Q4ish) on a 32GB M4 MacBook Air
A developer released Cherenkov, an open-source Apple Silicon inference engine that runs Qwen3.8-Flash-Next on a 32GB M4 MacBook Air at 8-22 tokens per second using only ~21GB of memory. It keeps a bounded working set of experts in unified memory and prefetches upcoming experts via SSD reads, claiming a record for memory-constrained inference.
A developer released Cherenkov, an open-source Apple Silicon inference engine that runs Qwen3.8-Flash-Next on a 32GB M4 MacBook Air at 8-22 tokens per second using only ~21GB of memory. It keeps a bounded working set of experts in unified memory and prefetches upcoming experts via SSD reads, claiming a record for memory-constrained inference.
61 engagement1 sourceredditThu, Sep 10, 2026, 09:31 PM