Releases·Tool Release
Dev releases open-source Dynamic Quantiser for making custom dynamic GGUF quants from llama.cpp files
A developer released Dynamic Quantiser, a free open-source tool that produces dynamic and custom quantizations of llama.cpp GGUF models at any target size without calibration data. It picks a quant level per tensor to minimize whole-model cosine deviation, which the author reports correlates 0.99 with KLD. Benchmarks against Unsloth show mixed results, with Unsloth stronger on pure text.
6 engagement1 sourceTue, Sep 22, 2026, 02:59 PM
AI Overview
- Random-Simulation's Dynamic Quantiser makes dynamic or custom quants of any llama.cpp-compatible gguf (bf16/f16/f32) at any target size, no calibration data needed
- It picks a quant level per tensor to minimise whole-model cosine deviation, which shows a 0.99 Pearson correlation with KLD
- It offers three algorithms: tensor-level dynamic quants, layer/group cosine minimisation, and disk-size minimisation for a fixed cosine deviation
- It optimises the ratio-of-sums objective with Dinkelbach's algorithm plus a Lagrangian inner loop and multi-start local search; after table build, solutions take seconds
- Benchmarks on Qwen 3.8 27b: it trails Unsloth on WikiText but beats Unsloth dynamic 3.0 on some quants using Bartowski's Mix calibration v5
- Limitations: AI-written code, Windows-only testing, ~25 min one-off table build on a Ryzen 5 3600 for the 53GB file, and no IQn_XS quants for inference-speed reasons
Related
Releases·Tool Release
Voodoo Dynamic Quant open-sourced under MIT license
The creator of Voodoo Quant, a dynamic quantization method claiming SOTA results at aggressive quant levels on smaller Qwen3.5 GGUF models, has released the full toolset under an MIT license on GitHub. The method uses gradient descent to train per-tensor scalar gates that select optimal quant levels, targeting both KL divergence against the BF16 reference and a filesize target. It competes with Unsloth Dynamic 3.0, which remains proprietary.
The creator of Voodoo Quant, a dynamic quantization method claiming SOTA results at aggressive quant levels on smaller Qwen3.5 GGUF models, has released the full toolset under an MIT license on GitHub. The method uses gradient descent to train per-tensor scalar gates that select optimal quant levels, targeting both KL divergence against the BF16 reference and a filesize target. It competes with Unsloth Dynamic 3.0, which remains proprietary.
359 engagement1 sourceredditTue, Sep 15, 2026, 06:59 AM
Releases·Tool Release
I squeezed Kolibri-1 78B-A3.5B to 20.9 GiB / 2.30 bpw — 59% lower KL than standard IQ2_XS
A community developer released Sakura-MicroQuality quantizations of Aleph Alpha's Kolibri-1, a ~78B MoE model with ~3.5B active parameters, compressing it to 20.94 GiB at 2.30 bpw in GGUF/llama.cpp format. The quant uses tensor/layer sensitivity to allocate bits where they matter most, achieving ~59% lower KL divergence than a standard IQ2_XS baseline and 90.5% top-token agreement against a Q8_0 reference. A second 365-expert variant prunes 950 rarely-used experts to reach 19.99 GiB at 74.4B parameters with no retraining.
A community developer released Sakura-MicroQuality quantizations of Aleph Alpha's Kolibri-1, a ~78B MoE model with ~3.5B active parameters, compressing it to 20.94 GiB at 2.30 bpw in GGUF/llama.cpp format. The quant uses tensor/layer sensitivity to allocate bits where they matter most, achieving ~59% lower KL divergence than a standard IQ2_XS baseline and 90.5% top-token agreement against a Q8_0 reference. A second 365-expert variant prunes 950 rarely-used experts to reach 19.99 GiB at 74.4B parameters with no retraining.
5 engagement1 sourceredditMon, Oct 5, 2026, 05:15 PM
Developments·Benchmark
Quant comparison of Qwen3.6 27B pits GGUF against NVFP4, AWQ, AutoRound, FP8
A developer benchmarked 16 quantizations of Qwen3.6 27B, comparing GGUF quants in llama.cpp against NVFP4, AWQ, AutoRound, and FP8 in vLLM. The comparison uses next-token probability distributions against an unquantized reference, with results and an interactive chart in a blog post.
A developer benchmarked 16 quantizations of Qwen3.6 27B, comparing GGUF quants in llama.cpp against NVFP4, AWQ, AutoRound, and FP8 in vLLM. The comparison uses next-token probability distributions against an unquantized reference, with results and an interactive chart in a blog post.
129 engagement1 sourceredditMon, Aug 10, 2026, 06:16 PM
Social·Community
Users run Qwen 3.8 27B locally on consumer GPUs: 75 tok/s on 16GB, agentic coding on 12GB, AMD slowdowns
Local LLM users are sharing configurations for running Qwen 3.8 27B on consumer GPUs. An RTX 5080 setup reaches ~75 tok/s decode using a custom hybrid GGUF quant with multi-token prediction, while a 12GB laptop GPU runs it for agentic coding at 9–11 tok/s with Unsloth Dynamic 3.0 quants. An AMD RX 9060 XT user finds the 27B too slow and switches to Ornith 1.5 9B.
Local LLM users are sharing configurations for running Qwen 3.8 27B on consumer GPUs. An RTX 5080 setup reaches ~75 tok/s decode using a custom hybrid GGUF quant with multi-token prediction, while a 12GB laptop GPU runs it for agentic coding at 9–11 tok/s with Unsloth Dynamic 3.0 quants. An AMD RX 9060 XT user finds the 27B too slow and switches to Ornith 1.5 9B.
Updated Mon, Aug 31, 2026, 09:34 AM — An RTX 5080 user reports ~75 tok/s decode, sometimes 100+ tok/s with good MTP acceptance, on the jrell hybrid quant.
210952 engagement4 sourcesredditThu, Aug 20, 2026, 11:54 PM
Releases·Tool Release
DFlash 2 released for Qwen 3.8 27B and Muse Glimmer with llama.cpp PR
The original authors of DFlash have released a second version, DFlash 2, with GGUF quants for Qwen 3.8 27B and Muse Glimmer. An accompanying llama.cpp pull request (PR #27342) suggests integration into the main inference stack. This update likely brings performance or quality improvements for local inference of these models.
The original authors of DFlash have released a second version, DFlash 2, with GGUF quants for Qwen 3.8 27B and Muse Glimmer. An accompanying llama.cpp pull request (PR #27342) suggests integration into the main inference stack. This update likely brings performance or quality improvements for local inference of these models.
1.4k engagement2 sourcesreddit, xTue, Aug 18, 2026, 09:36 PM