Releases·Tool Release
Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors.
A solo developer released Flyweight, an open-source C++/CUDA GGUF inference runtime that runs MoE models larger than a single GPU's VRAM by keeping experts in system RAM with a hot set cached on-card. It ships OpenAI/Anthropic-compatible APIs, a chat UI, image generation, and quantized KV cache, with benchmarks from a 12 GB laptop GPU. The Apache-2.0 project is on PyPI and actively requesting contributors for hardware and model coverage.
66 engagement1 sourceFri, Sep 18, 2026, 07:39 AM
AI Overview
- Flyweight runs GGUF MoE experts on CPU with a VRAM hot set, auto-sizing at startup, with OpenAI/Anthropic-compatible APIs and a chat UI.
- On a 5070 Ti 12 GB laptop, it hits ~35 tok/s on Qwen3.8-Flash-Next IQ1_S and runs Z-Image 1024x1024 in ~15 s beside a 35B model.
- Kernels compile at runtime via NVRTC (no CUDA toolkit in wheel); KV cache offers f16/q8_0/TurboQuant 4-bit, tool calls are sampler-grammar-enforced, and thinking budget is a hard cap.
- Author Yair (yairpatch) disclosed writing much of the code with Claude Code, with all kernels parity-tested; Apache-2.0 on GitHub.
- Solicited contributions: Ampere/Ada GPUs, AMD CPUs, 8 GB cards, Windows/macOS/ARM testing, and CPU expert kernels for 1-3 bit IQ formats.
Related
Releases·Tool Release
LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)
LayerStoRm, a new experimental MIT-licensed inference engine, streams MoE experts from pinned host RAM to run models far larger than VRAM. It runs GLM-5.3-Flash (186 GiB, UD-Q4_K_XL) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM). This matters to practitioners running frontier-scale MoE models on consumer hardware without a CPU-inference speed penalty.
LayerStoRm, a new experimental MIT-licensed inference engine, streams MoE experts from pinned host RAM to run models far larger than VRAM. It runs GLM-5.3-Flash (186 GiB, UD-Q4_K_XL) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM). This matters to practitioners running frontier-scale MoE models on consumer hardware without a CPU-inference speed penalty.
23 engagement1 sourceredditMon, Sep 7, 2026, 01:14 AM
Releases·Model Releasefeatured
Unsloth releases Qwen3.8-27B Dynamic v3 GGUFs with 10% higher accuracy
Unsloth released new Qwen3.8-27B GGUFs using Dynamic v3.0, claiming over 10% higher accuracy on benchmarks like Div-300 and KLD. They also introduced 1-bit quants retaining 77% accuracy, enabling local runs on 8GB RAM. The update follows earlier support for running and fine-tuning Qwen3.8-27B locally.
Unsloth released new Qwen3.8-27B GGUFs using Dynamic v3.0, claiming over 10% higher accuracy on benchmarks like Div-300 and KLD. They also introduced 1-bit quants retaining 77% accuracy, enabling local runs on 8GB RAM. The update follows earlier support for running and fine-tuning Qwen3.8-27B locally.
Updated Fri, Aug 21, 2026, 02:15 PM — MagicQuant updated with Unsloth Dynamic v3, reporting substantial boost over v2.
42011.8k engagement6 sourcesgithub, reddit, xWed, Aug 19, 2026, 12:00 AM
Releases·Tool Release
NInfer port hits 45 tok/s Qwen3.8-27B on modded RTX 2080 Ti
A developer ported NInfer, a from-scratch C++/CUDA inference engine originally for Blackwell, to Turing GPUs, achieving 45 tok/s on a 22GB modded RTX 2080 Ti with Qwen3.8-27B using speculative decoding. The port targets sm_75 and is incompatible with unmodded 11GB cards.
A developer ported NInfer, a from-scratch C++/CUDA inference engine originally for Blackwell, to Turing GPUs, achieving 45 tok/s on a 22GB modded RTX 2080 Ti with Qwen3.8-27B using speculative decoding. The port targets sm_75 and is incompatible with unmodded 11GB cards.
21 engagement1 sourceredditFri, Aug 21, 2026, 06:16 AM
Products·launched
Run massive MoE models on mobile devices via CPU
BigEdgeOnMoE is an open-source codebase that enables running large Mixture-of-Experts models (up to 120B parameters) on standard smartphones or consumer PCs using only CPU. It achieves 6 tokens/second for Qwen 35B (Q4) on a mid-range phone with 12GB RAM, making large models accessible without specialized hardware.
BigEdgeOnMoE is an open-source codebase that enables running large Mixture-of-Experts models (up to 120B parameters) on standard smartphones or consumer PCs using only CPU. It achieves 6 tokens/second for Qwen 35B (Q4) on a mid-range phone with 12GB RAM, making large models accessible without specialized hardware.
20 engagement1 sourceredditThu, Jul 30, 2026, 03:34 PM
Products·launched
LocalMind: browser-based LLM chat running models via WebGPU with MoE weights streamed from disk
LocalMind is a static web page that runs LLMs in the browser through WebGPU, with no server, account, or install. Its new engines stream mixture-of-experts weights from disk during generation, letting a browser tab run models larger than the machine's RAM. This targets people who want to run big local models on consumer machines like a 24 GB Mac.
LocalMind is a static web page that runs LLMs in the browser through WebGPU, with no server, account, or install. Its new engines stream mixture-of-experts weights from disk during generation, letting a browser tab run models larger than the machine's RAM. This targets people who want to run big local models on consumer machines like a 24 GB Mac.
21 engagement1 sourceredditTue, Oct 6, 2026, 06:41 AM