An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk
A developer extended llama.cpp's WebGPU backend to page MoE experts from disk into GPU slots, removing the need for per-model engines. On a MacBook Pro M4 Pro, this runs the 48.4 GB Qwen3-Next 80B-A3B GGUF in a Chrome tab at 2.9 tok/s, plus faster smaller MoE models. It is a proof of concept: prompt processing is slow, and it requires Chrome/Edge 137+ on Apple silicon.
- Diskformer.js pages MoE experts from OPFS into a fixed slot pool in llama.cpp's ggml-webgpu via a new buffer type and slot-mapped MUL_MAT_ID shaders, no per-model engine.
- On MacBook Pro M4 Pro (24 GB), Qwen3-Next 80B-A3B Q4_K_M (48.4 GB) loads in 48 s and decodes at 2.9 tok/s, reading ~359 MB per token from disk.
- OLMoE 1B-7B hits 22.4 tok/s and LFM2 8B-A1B 17.1 tok/s; Qwen3-30B-A3B (14.7 GB) also runs; paging on/off gives identical replies, 4–8 of 9 match Metal due to rounding.
- Requires Chrome or Edge 137+, tested only on Apple silicon; first load copies the full GGUF into OPFS, and prompt processing remains slow.
Entities
Related
LocalMind: browser-based LLM chat running models via WebGPU with MoE weights streamed from disk
LocalMind is a static web page that runs LLMs in the browser through WebGPU, with no server, account, or install. Its new engines stream mixture-of-experts weights from disk during generation, letting a browser tab run models larger than the machine's RAM. This targets people who want to run big local models on consumer machines like a 24 GB Mac.
LocalMind is a static web page that runs LLMs in the browser through WebGPU, with no server, account, or install. Its new engines stream mixture-of-experts weights from disk during generation, letting a browser tab run models larger than the machine's RAM. This targets people who want to run big local models on consumer machines like a 24 GB Mac.
llama.cpp PR #26563 caches hot MoE experts on GPU, boosting Qwen3.6-35B speed 1.68x on 8GB VRAM
A llama.cpp pull request (#26563) introduces a heatmap that tracks frequently used MoE experts and caches them in GPU VRAM while cold experts run on CPU. On Qwen3.6-35B-A3B with 8GB VRAM, the author reports speedups from 33.25 to 56.0 tok/s (Q2_M) and 17.34 to 35.93 tok/s (Q5_K_P). However, some models like Qwen3.5-122B-A10B and Laguna-S-2.1 were slower with caching enabled, indicating the technique is not universally beneficial.
A llama.cpp pull request (#26563) introduces a heatmap that tracks frequently used MoE experts and caches them in GPU VRAM while cold experts run on CPU. On Qwen3.6-35B-A3B with 8GB VRAM, the author reports speedups from 33.25 to 56.0 tok/s (Q2_M) and 17.34 to 35.93 tok/s (Q5_K_P). However, some models like Qwen3.5-122B-A10B and Laguna-S-2.1 were slower with caching enabled, indicating the technique is not universally beneficial.
Qwen 3.8 Flash Next (68GB IQ2_XS) hits 21 tok/s on RTX 3060 12GB + 16GB RAM via llama.cpp --moe-direct-io, bit-exact, no expert pruning
A developer implemented a --moe-direct-io feature in llama.cpp that predicts which MoE experts the next token needs and prefetches them, running Qwen 3.8 Flash Next (68GB IQ2_XS) at 20-24 tok/s on an RTX 3060 12GB with only 16GB DDR4 RAM. Output is bit-exact to stock llama.cpp, avoiding the expert-pruning tradeoff that made an earlier attempt unusable. The speedup is roughly 10-15x over stock mmap-based llama.cpp, which hit 1.4-2.1 tok/s on the same setup.
A developer implemented a --moe-direct-io feature in llama.cpp that predicts which MoE experts the next token needs and prefetches them, running Qwen 3.8 Flash Next (68GB IQ2_XS) at 20-24 tok/s on an RTX 3060 12GB with only 16GB DDR4 RAM. Output is bit-exact to stock llama.cpp, avoiding the expert-pruning tradeoff that made an earlier attempt unusable. The speedup is roughly 10-15x over stock mmap-based llama.cpp, which hit 1.4-2.1 tok/s on the same setup.
CompiledThoughts streams Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) off SSD at 9-10 tok/s on a 16 GB RTX 5060 Ti
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.
Qwen3-Next-80B on a 3090 with 16GB RAM: ~3x faster decode than stock llama.cpp by not waiting for every expert (patch + paper)
A developer published a llama.cpp research patch and preprint showing that, instead of waiting for NVMe reads of offloaded MoE experts, the engine substitutes already-resident experts under a gate-weight threshold rule. On Qwen3-Next-80B-A3B with a RTX 3090 and 16GB RAM, decode throughput rose from 31.8 to 94.5 tok/s versus stock llama.cpp, with modest quality loss.
A developer published a llama.cpp research patch and preprint showing that, instead of waiting for NVMe reads of offloaded MoE experts, the engine substitutes already-resident experts under a gate-weight threshold rule. On Qwen3-Next-80B-A3B with a RTX 3090 and 16GB RAM, decode throughput rose from 31.8 to 94.5 tok/s versus stock llama.cpp, with modest quality loss.



