Releases·Tool Release
vLLM gains expert RAM offloading via tcclaviger, running DeepSeek-V4-Flash-Vision-Exp on four R9700s
A Reddit appreciation post credits contributor tcclaviger with bringing expert RAM offloading support to vLLM, making large frontier models runnable on modest local GPU setups. The author reports running DeepSeek-V4-Flash-Vision-Exp on four R9700 cards using tcclaviger's vLLM container and a shared Hugging Face model repository.
86 engagement1 sourceTue, Sep 29, 2026, 05:14 PM
AI Overview
- tcclaviger added expert RAM offloading support to vLLM, exposed via --enable-expert-offload and --expert-offload-mem flags
- A Reddit user ran the original DeepSeek-V4-Flash-Vision-Exp locally on four AMD R9700 GPUs with 160GB expert offload and tensor parallelism of 4
- The setup used tcclaviger's vLLM podman image, fp8 KV cache, 256000 max model length, speculative decoding (dspark, 3 tokens), and cudagraph capture up to size 16
- tcclaviger published a quantized model checkpoint, Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ, on Hugging Face
Related
Social·Community
DeepSeek-V4-Flash-Vision-Exp (285B MoE) on 10-12x RTX 3090 — spec decoding, vision
A community post describes running the full deepseek-ai/DeepSeek-V4-Flash-Vision-Exp model on consumer Ampere GPUs, using 10-12 RTX 3090s with a custom SM86-compatible vLLM build. It demonstrates that a 285B-parameter MoE vision model with FP4-quantized experts can operate entirely on second-hand consumer hardware.
A community post describes running the full deepseek-ai/DeepSeek-V4-Flash-Vision-Exp model on consumer Ampere GPUs, using 10-12 RTX 3090s with a custom SM86-compatible vLLM build. It demonstrates that a 285B-parameter MoE vision model with FP4-quantized experts can operate entirely on second-hand consumer hardware.
59 engagement1 sourceredditWed, Sep 9, 2026, 10:50 AM
Social·Community
User shares llama.cpp Docker setup for Qwen3.6 35B MoE on RTX 5060 Ti 16GB
A Reddit user posted their llama.cpp Docker configuration for running Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf on an RTX 5060 Ti 16GB with 16GB system RAM. They seek feedback on flags like -ngl 99 and the overall setup for a shared LLM endpoint.
A Reddit user posted their llama.cpp Docker configuration for running Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf on an RTX 5060 Ti 16GB with 16GB system RAM. They seek feedback on flags like -ngl 99 and the overall setup for a shared LLM endpoint.
13 engagement1 sourceredditMon, Jul 27, 2026, 01:30 PM
Products·building
Overspill: disk tier for FreeToken to run oversized MoE models on consumer hardware
Overspill is an experimental disk-storage tier for the FreeToken inference engine that lets MoE models larger than system RAM run on consumer rigs. It keeps experts on disk and streams them across disk/RAM/VRAM, inspired by Colibri's approach. It targets hobbyists running large MoE models like DeepSeek-V4-Flash on modest GPUs.
Overspill is an experimental disk-storage tier for the FreeToken inference engine that lets MoE models larger than system RAM run on consumer rigs. It keeps experts on disk and streams them across disk/RAM/VRAM, inspired by Colibri's approach. It targets hobbyists running large MoE models like DeepSeek-V4-Flash on modest GPUs.
47 engagement1 sourceredditSat, Sep 26, 2026, 05:41 PM
Releases·Tool Release
LayerStoRm open-sources MoE serving engine that streams experts over PCIe to run frontier-scale LLMs on consumer GPUs
LayerStoRm is an open-source MoE LLM serving engine that streams only the experts each token routes to from RAM over parallel PCIe transfers, enabling frontier-scale models on limited-VRAM consumer GPU systems. It targets systems with RTX 5090/5080-class GPUs and large system RAM, using a smart VRAM cache and tight packing techniques.
LayerStoRm is an open-source MoE LLM serving engine that streams only the experts each token routes to from RAM over parallel PCIe transfers, enabling frontier-scale models on limited-VRAM consumer GPU systems. It targets systems with RTX 5090/5080-class GPUs and large system RAM, using a smart VRAM cache and tight packing techniques.
3 engagement1 sourceredditTue, Aug 25, 2026, 03:38 AM
Social·Community
User runs DeepSeek-V4-Flash-0731 at full 1M context on single RTX 5090 with phase-adaptive DSpark K1/K2
A user reports running DeepSeek-V4-Flash-0731 at its native 1M context on a single RTX 5090, using phase-adaptive DSpark K1/K2 with dual CUDA graphs and routed MoE experts in system RAM. Achieves ~13.8 tok/s reasoning and ~17.0 tok/s final/code. An important edit warns to leave ~11% VRAM outside vLLM's model/KV budget to avoid crashes at long context.
A user reports running DeepSeek-V4-Flash-0731 at its native 1M context on a single RTX 5090, using phase-adaptive DSpark K1/K2 with dual CUDA graphs and routed MoE experts in system RAM. Achieves ~13.8 tok/s reasoning and ~17.0 tok/s final/code. An important edit warns to leave ~11% VRAM outside vLLM's model/KV budget to avoid crashes at long context.
16 engagement1 sourceredditTue, Aug 4, 2026, 09:39 PM