Releases·Tool Release
Voodoo Dynamic Quant open-sourced under MIT license
The creator of Voodoo Quant, a dynamic quantization method claiming SOTA results at aggressive quant levels on smaller Qwen3.5 GGUF models, has released the full toolset under an MIT license on GitHub. The method uses gradient descent to train per-tensor scalar gates that select optimal quant levels, targeting both KL divergence against the BF16 reference and a filesize target. It competes with Unsloth Dynamic 3.0, which remains proprietary.
359 engagement1 sourceTue, Sep 15, 2026, 06:59 AM
AI Overview
- Voodoo Quant uses gradient descent to optimize per-tensor quant layout, training scalar gates per tensor per quant level annealed via tau.
- Quant selections are trained against KL divergence of mixed-quant logits versus a BF16 reference checkpoint plus a filesize target.
- Creator reports Unsloth Dynamic 3.0 better at high-mid quants but Voodoo better at aggressive levels on small Qwen3.5 GGUF models.
- GitHub repo (curvedinf/voodoo-dyn-quant) ships tools currently set up for Qwen, adaptable to other architectures; untested at larger model sizes.
Related
Releases·Tool Release
Dev releases open-source Dynamic Quantiser for making custom dynamic GGUF quants from llama.cpp files
A developer released Dynamic Quantiser, a free open-source tool that produces dynamic and custom quantizations of llama.cpp GGUF models at any target size without calibration data. It picks a quant level per tensor to minimize whole-model cosine deviation, which the author reports correlates 0.99 with KLD. Benchmarks against Unsloth show mixed results, with Unsloth stronger on pure text.
A developer released Dynamic Quantiser, a free open-source tool that produces dynamic and custom quantizations of llama.cpp GGUF models at any target size without calibration data. It picks a quant level per tensor to minimize whole-model cosine deviation, which the author reports correlates 0.99 with KLD. Benchmarks against Unsloth show mixed results, with Unsloth stronger on pure text.
6 engagement1 sourceredditTue, Sep 22, 2026, 02:59 PM
Releases·Tool Release
Transformers now runs llama.cpp quants
Hugging Face announced that transformers can now load and run llama.cpp GGUF quantized checkpoints directly via from_pretrained, reusing ggml's Metal kernels through the kernels library to approach llama.cpp performance. Initial support targets Apple Silicon Macs, starting with the Qwen3.5 architecture. This bridges llama.cpp's local inference foundation with transformers' PyTorch model-definition ecosystem, enabling experimentation, evaluation, fine-tuning, and conversion validation on GGUF checkpoints.
Hugging Face announced that transformers can now load and run llama.cpp GGUF quantized checkpoints directly via from_pretrained, reusing ggml's Metal kernels through the kernels library to approach llama.cpp performance. Initial support targets Apple Silicon Macs, starting with the Qwen3.5 architecture. This bridges llama.cpp's local inference foundation with transformers' PyTorch model-definition ecosystem, enabling experimentation, evaluation, fine-tuning, and conversion validation on GGUF checkpoints.
295 engagement2 sourcesrss, redditTue, Sep 22, 2026, 12:00 AM
Releases·Tool Release
Two new quantization releases target Qwen3.8-27B: GSQ-RCO GGUFs and QUASAR NVFP4
Two independent teams released aggressively quantized versions of Qwen3.8-27B within two days. One ships GGUF checkpoints at 2.5-3.0 bits per weight using two new methods, GSQ and RCO, with both the search and quantizer learned. The other releases an NVFP4 W4A4 checkpoint trained via quantization-aware distillation with the QUASAR QAT algorithm, deployable on vLLM with NVIDIA Blackwell GPUs. Both matter to practitioners running large models at low precision locally or on modern accelerators.
Two independent teams released aggressively quantized versions of Qwen3.8-27B within two days. One ships GGUF checkpoints at 2.5-3.0 bits per weight using two new methods, GSQ and RCO, with both the search and quantizer learned. The other releases an NVFP4 W4A4 checkpoint trained via quantization-aware distillation with the QUASAR QAT algorithm, deployable on vLLM with NVIDIA Blackwell GPUs. Both matter to practitioners running large models at low precision locally or on modern accelerators.
Updated Fri, Aug 28, 2026, 09:46 PM — GSQ-RCO GGUF release at 2.5-3.0 bpw followed the earlier QUASAR NVFP4 checkpoint on 2026-08-28.
141 engagement2 sourcesredditWed, Aug 26, 2026, 01:04 AM
Developments·Paper
ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization
ReQuant introduces a backpropagation-free, fixed-grid discrete refinement stage for post-training quantization (PTQ) of large language models. It keeps quantized weights improvable after an executable quantized model is produced, preserving the quantized format. This complements existing PTQ methods by reducing memory and computational costs further.
ReQuant introduces a backpropagation-free, fixed-grid discrete refinement stage for post-training quantization (PTQ) of large language models. It keeps quantized weights improvable after an executable quantized model is produced, preserving the quantized format. This complements existing PTQ methods by reducing memory and computational costs further.
0 engagement1 sourcearxivFri, Aug 7, 2026, 09:27 AM
Developments·Benchmark
Quant comparison of Qwen3.6 27B pits GGUF against NVFP4, AWQ, AutoRound, FP8
A developer benchmarked 16 quantizations of Qwen3.6 27B, comparing GGUF quants in llama.cpp against NVFP4, AWQ, AutoRound, and FP8 in vLLM. The comparison uses next-token probability distributions against an unquantized reference, with results and an interactive chart in a blog post.
A developer benchmarked 16 quantizations of Qwen3.6 27B, comparing GGUF quants in llama.cpp against NVFP4, AWQ, AutoRound, and FP8 in vLLM. The comparison uses next-token probability distributions against an unquantized reference, with results and an interactive chart in a blog post.
129 engagement1 sourceredditMon, Aug 10, 2026, 06:16 PM