Releases·Tool Release
ExLlamaV3 adds CPU offload of MoE experts, GLM-5.3-Flash and Qwen3.8-Flash-Next quants
Turboderp shipped a batch of ExLlamaV3 updates including CPU offload of MoE experts, ngram disk offload for Qwen3.8-Flash-Next, a GLM-5.3-Flash exl3 quant, and a new self-calibrated optimization technique for quantization. The releases target NVIDIA GPU users running large models locally and expand what fits in limited VRAM.
295 engagement1 sourceTue, Sep 1, 2026, 07:14 AM
AI Overview
- CPU offload of MoE experts lets users run larger MoE models than fit in VRAM alone
- Qwen3.8-Flash-Next exl3 quant adds ngram-based disk offload
- GLM-5.3-Flash released as an exl3 quant on Hugging Face
- New self-calibrated optimization technique documented in the repo's optimize.md
- Numerous other optimizations and improvements listed in the GitHub releases
- A Reddit poster recommends NVIDIA card owners retry ExLlamaV3 if they haven't lately
Related
Releases·Tool Release
Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine
Yamz-Labs released Kyojin, an inference engine built on ExLlamaV3 for AMD Strix Halo (gfx1151, ROCm), plus EXL3 quantized weight packs for GLM-5.3-Flash and MiMo-V2.6-Flash that each fit on a single 128 GB Ryzen AI Max+ 395 machine. The release brings 300B-class MoE inference to consumer mini PCs with measured prefill and decode speeds.
Yamz-Labs released Kyojin, an inference engine built on ExLlamaV3 for AMD Strix Halo (gfx1151, ROCm), plus EXL3 quantized weight packs for GLM-5.3-Flash and MiMo-V2.6-Flash that each fit on a single 128 GB Ryzen AI Max+ 395 machine. The release brings 300B-class MoE inference to consumer mini PCs with measured prefill and decode speeds.
154 engagement2 sourcesreddit, githubSat, Oct 3, 2026, 02:16 PM
Social·Community
Redditor plans €3,000+ budget CPU/GPU hybrid build to run Qwen3.8-Flash-Next at 4-bit locally
A Reddit user planning their first local inference PC asked for feedback on a hybrid CPU+GPU build centered on Qwen3.8-Flash-Next at 4-bit, after testing roughly 20 open models. Qwen3.8-Flash-Next with medium reasoning was the only model that completed their task without inventing config options when a harness forced doc lookups. The plan puts most model weights in DDR5 RAM with a used RTX 3090 handling the remainder, and the user questions whether the CPU, RAM capacity, and memory speed choices will work.
A Reddit user planning their first local inference PC asked for feedback on a hybrid CPU+GPU build centered on Qwen3.8-Flash-Next at 4-bit, after testing roughly 20 open models. Qwen3.8-Flash-Next with medium reasoning was the only model that completed their task without inventing config options when a harness forced doc lookups. The plan puts most model weights in DDR5 RAM with a used RTX 3090 handling the remainder, and the user questions whether the CPU, RAM capacity, and memory speed choices will work.
16 engagement1 sourceredditTue, Oct 6, 2026, 11:32 AM
Social·Community
Reddit user asks whether a custom inference engine like Strata exists for GLM-5.3-Flash local runs
A Reddit user asks whether anyone is building a custom inference engine for GLM-5.3-Flash, citing Strata's average 2x performance over llama.cpp with Qwen3.8-Flash-Next as precedent. The user reports GLM-5.3-Flash outperformed Claude and ChatGPT on their use case and wants to run it locally on a 3x3090 setup.
A Reddit user asks whether anyone is building a custom inference engine for GLM-5.3-Flash, citing Strata's average 2x performance over llama.cpp with Qwen3.8-Flash-Next as precedent. The user reports GLM-5.3-Flash outperformed Claude and ChatGPT on their use case and wants to run it locally on a 3x3090 setup.
30 engagement1 sourceredditTue, Oct 6, 2026, 03:38 PM
Social·Community
Reddit user: ExLlamaV3 quants beat llama.cpp on quality, KLD, and speed in personal tests
A Reddit user argues ExLlamaV3 (Exl3) is under-discussed relative to its quality, claiming its quants outperform llama.cpp in quality per size, KLD metrics, and speed on personal tests. The post notes the software appears limited to NVIDIA cards.
A Reddit user argues ExLlamaV3 (Exl3) is under-discussed relative to its quality, claiming its quants outperform llama.cpp in quality per size, KLD metrics, and speed on personal tests. The post notes the software appears limited to NVIDIA cards.
126 engagement1 sourceredditMon, Sep 7, 2026, 08:16 PM
Releases·Tool Release
ExLlamaV3 v1.0.0 released with major performance upgrades
After over a year of development, Turboderp released ExLlamaV3 v1.0.0, a production-ready inference engine for LLMs. The update removes flash-attention-2 and xformers dependencies, extends tensor-parallel support to most models including Gemma4, and introduces a new attention kernel with online cache quantization and dual input for SWA layers.
After over a year of development, Turboderp released ExLlamaV3 v1.0.0, a production-ready inference engine for LLMs. The update removes flash-attention-2 and xformers dependencies, extends tensor-parallel support to most models including Gemma4, and introduces a new attention kernel with online cache quantization and dual input for SWA layers.
370 engagement1 sourceredditWed, Jul 15, 2026, 07:17 AM