Releases·Tool Release
Dev's expert-lookahead trick yields 10%+ speedup for MoE expert-offloading on low-memory Macs
A developer reports a 10%+ performance gain running MoE models like Qwen 3.8 flash on low-memory Macs by predicting upcoming experts and prefetching them from SSD via the slotstream tool. The technique runs the model's own router two layers ahead instead of a trained predictor, and a small correction model adds ~3-4% more.
32 engagement1 sourceTue, Sep 15, 2026, 03:23 PM
AI Overview
- Developer carloslfu's slotstream runs Qwen 3.8 flash's router 2 layers ahead on the current hidden state to prefetch experts to memory, adding 10%+ on top of existing SSD-streaming optimizations
- A trained small expert-prediction model lost to using the model's own router as predictor; a correction model now in training adds ~3-4% further improvement
- Docs for expert-lookahead are published on GitHub at carloslfu/slotstream
Related
Social·Community
MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?
A developer is running Qwen Flash Next (q4) on a Mac mini M5 with 64 GB of RAM using a custom SSD-streaming scheme for MoE experts that don't fit in memory, achieving 17.5 tokens/s decode and 390 tokens/s prompt processing. They are asking the community for ideas to reduce the 27% of decode time the GPU spends waiting for experts to stream from SSD, and have published their setup on GitHub.
A developer is running Qwen Flash Next (q4) on a Mac mini M5 with 64 GB of RAM using a custom SSD-streaming scheme for MoE experts that don't fit in memory, achieving 17.5 tokens/s decode and 390 tokens/s prompt processing. They are asking the community for ideas to reduce the 27% of decode time the GPU spends waiting for experts to stream from SSD, and have published their setup on GitHub.
29 engagement1 sourceredditMon, Oct 5, 2026, 10:50 AM
Social·Community
User tests MTP-based expert prefetch to cut MoE offload latency, 30→150-200 tok/s
A developer experimenting with Qwen3.6 35B A3B on a 12GB RTX 3060 found that using the model's MTP head to predict next-token expert usage can prefetch experts over PCIe, boosting throughput from 30 to 150-200 tokens/s. The technique addresses the PCIe bandwidth bottleneck in CPU/GPU offload setups.
A developer experimenting with Qwen3.6 35B A3B on a 12GB RTX 3060 found that using the model's MTP head to predict next-token expert usage can prefetch experts over PCIe, boosting throughput from 30 to 150-200 tokens/s. The technique addresses the PCIe bandwidth bottleneck in CPU/GPU offload setups.
150 engagement1 sourceredditThu, Jul 16, 2026, 06:47 PM
Social·Community
Developer reports 50% token-generation speedup by offloading hot MoE experts to VRAM in llama.cpp fork
A developer shared a llama.cpp fork that boosts token generation from 20 to 30 tokens per second (50%) for MoE models that don't fit in VRAM, by offloading only frequently used 'hot' experts to the GPU instead of whole layers. The approach exploits the observation that expert activation groups stay stable across coding workloads. It matters to practitioners running large MoE models locally on limited VRAM.
A developer shared a llama.cpp fork that boosts token generation from 20 to 30 tokens per second (50%) for MoE models that don't fit in VRAM, by offloading only frequently used 'hot' experts to the GPU instead of whole layers. The approach exploits the observation that expert activation groups stay stable across coding workloads. It matters to practitioners running large MoE models locally on limited VRAM.
169 engagement2 sourcesredditSat, Aug 29, 2026, 01:42 AM
Developments·Paper
Researcher details methodology for streaming MoE experts to edge devices
A researcher working on streaming large Mixture-of-Experts (MoE) models to edge devices like mobile phones reports a breakthrough in making such offloading sustainable and feasible. The core challenge addressed is predicting which experts will be needed in the near future to minimize I/O latency from flash storage.
A researcher working on streaming large Mixture-of-Experts (MoE) models to edge devices like mobile phones reports a breakthrough in making such offloading sustainable and feasible. The core challenge addressed is predicting which experts will be needed in the near future to minimize I/O latency from flash storage.
0 engagement1 sourceredditSat, Aug 1, 2026, 08:13 AM
Social·Community
Developer runs 300B DeepSeek-v4 on 32GB laptop via RAM-resident MoE streaming
A developer shares findings from running DeepSeek-v4 (DSv4) inference on a laptop with 32GB RAM by keeping everything RAM-resident except the MXFP4 experts, which are streamed from disk. They report that read speed is the main bottleneck, and propose optimizations like repacking for sequential reads, pipelining, and speculative expert prefetch.
A developer shares findings from running DeepSeek-v4 (DSv4) inference on a laptop with 32GB RAM by keeping everything RAM-resident except the MXFP4 experts, which are streamed from disk. They report that read speed is the main bottleneck, and propose optimizations like repacking for sequential reads, pipelining, and speculative expert prefetch.
23 engagement1 sourceredditSun, Aug 9, 2026, 10:14 AM