Dev runs full 510 GB DeepSeek-V4.1-Flash-1017 on a 64 GB RAM + RTX 3060 PC at up to 7.4 tok/s; releases PROMETHEVS engine
A developer released PROMETHEVS 0.2, an Apache-2.0 engine that streams the full original DeepSeek-V4.1-Flash-1017 weights (510 GB) through VRAM, RAM, and two NVMe drives on consumer hardware. Using a 2-bit expert copy with a compress-then-correct scheme, it reaches up to 7.4 tok/s with 98.3% agreement with the full model on a Ryzen 9 3950X + RTX 3060 12 GB machine.
- PROMETHEVS engine streams the 510 GB model across VRAM, RAM, and two NVMe drives on a Ryzen 9 3950X PC with 64 GB DDR4 and RTX 3060 12 GB
- Compress-then-correct method computes tokens on a 2-bit expert copy and recomputes uncertain ones with original FP4 weights, reaching 98.3% agreement on 960 audited tokens
- Speeds reach 7.4 tok/s warm on a code prompt versus 2.14 for exact full-model decoding; server runs 5–8 tok/s on code and 4.5–5.4 on reasoning prompts
- Release 0.2 (Apache-2.0) builds all stores from original weights against published hashes and serves an OpenAI-compatible endpoint with streaming and tool calls
- Limits: tested only on the author's PC, Windows plus one RTX 30/40 card, ~1.1 TB NVMe, not yet compared with DeepSeek's reference implementation
Related
Dev runs DeepSeek V4.1 Flash 1017 (IQ1_S) on RTX 5090 + 5060 Ti, hitting 63.5 tok/s decode
A developer got DeepSeek V4.1 Flash 1017 (Unsloth UD-IQ1_S quant) running on consumer hardware — an RTX 5090 plus RTX 5060 Ti — using Basalt, an inference engine they are developing. The setup stores all experts in RAM and uses the GPUs as an expert cache, reaching 63.5 tok/s decode versus ~20 tok/s on a llama.cpp baseline. It's a personal experiment showing that very large MoE models can run on single-GPU rigs, with more headroom on higher-VRAM or multi-channel DDR5 systems.
A developer got DeepSeek V4.1 Flash 1017 (Unsloth UD-IQ1_S quant) running on consumer hardware — an RTX 5090 plus RTX 5060 Ti — using Basalt, an inference engine they are developing. The setup stores all experts in RAM and uses the GPUs as an expert cache, reaching 63.5 tok/s decode versus ~20 tok/s on a llama.cpp baseline. It's a personal experiment showing that very large MoE models can run on single-GPU rigs, with more headroom on higher-VRAM or multi-channel DDR5 systems.
DeepSeek V4.1 Flash at 2.61 bpw + Engram: 26 t/s on dual RTX 3090s with dsx
A Reddit user quantized DeepSeek-V4.1-Flash to 2.61 bits per weight using the community dsx inference engine and published the weights on Hugging Face. On two RTX 3090s plus Engram lookup tables, it generates 26 tokens/s with 262,144-token context, making a large MoE model runnable on consumer hardware. Practitioners get a quality/speed data point and an alternative to existing 2-3 bpw builds.
A Reddit user quantized DeepSeek-V4.1-Flash to 2.61 bits per weight using the community dsx inference engine and published the weights on Hugging Face. On two RTX 3090s plus Engram lookup tables, it generates 26 tokens/s with 262,144-token context, making a large MoE model runnable on consumer hardware. Practitioners get a quality/speed data point and an alternative to existing 2-3 bpw builds.
tideLM serves oversized MoE models from two DGX Sparks via adaptive expert loading
TideLM is an open-source LLM inference engine that runs mixture-of-experts models too large for local memory by keeping frequently used experts resident in the unified memory of two DGX Sparks and streaming misses from NVMe. The resident expert set adapts to traffic over time, improving serving as usage accumulates. Benchmarks show 25.9 tok/s prose decoding for DeepSeek-V4.1-Flash-1017 at native MXFP4.
TideLM is an open-source LLM inference engine that runs mixture-of-experts models too large for local memory by keeping frequently used experts resident in the unified memory of two DGX Sparks and streaming misses from NVMe. The resident expert set adapts to traffic over time, improving serving as usage accumulates. Benchmarks show 25.9 tok/s prose decoding for DeepSeek-V4.1-Flash-1017 at native MXFP4.
Condor runs Qwen3.8-Flash-Next (180B MoE) on one 12-16 GB NVIDIA GPU with own engine and compressed weights
The Condor project released an open-source inference engine that runs the 180B-parameter Qwen3.8-Flash-Next MoE on one consumer 12-16 GB NVIDIA GPU with 32-64 GB of RAM. It uses its own compression scheme and CPU-GPU expert routing, with OpenAI- and Anthropic-compatible APIs, MCP support, and free personal use. Benchmarks on an RTX 4080 SUPER show GSM8K 148-149/150 and up to ~169 tok/s in fast mode.
The Condor project released an open-source inference engine that runs the 180B-parameter Qwen3.8-Flash-Next MoE on one consumer 12-16 GB NVIDIA GPU with 32-64 GB of RAM. It uses its own compression scheme and CPU-GPU expert routing, with OpenAI- and Anthropic-compatible APIs, MCP support, and free personal use. Benchmarks on an RTX 4080 SUPER show GSM8K 148-149/150 and up to ~169 tok/s in fast mode.
CompiledThoughts streams Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) off SSD at 9-10 tok/s on a 16 GB RTX 5060 Ti
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.



