Releases·Tool Release
Transformers now runs llama.cpp quants
Hugging Face announced that transformers can now load and run llama.cpp GGUF quantized checkpoints directly via from_pretrained, reusing ggml's Metal kernels through the kernels library to approach llama.cpp performance. Initial support targets Apple Silicon Macs, starting with the Qwen3.5 architecture. This bridges llama.cpp's local inference foundation with transformers' PyTorch model-definition ecosystem, enabling experimentation, evaluation, fine-tuning, and conversion validation on GGUF checkpoints.
295 engagement2 sourcesTue, Sep 22, 2026, 12:00 AM
AI Overview
- Hugging Face's transformers now loads GGUF checkpoints via from_pretrained, reusing ggml Metal kernels through the kernels library.
- Initial support targets Apple Silicon local inference, starting with the Qwen3.5 architecture; Q4_K_M cuts Unsloth's Qwen3.5-4B from 8.42 GB (BF16) to 2.74 GB.
- Benchmarks on a MacBook Pro M2 Max show transformers close to llama.cpp (build 5f55650a7, release b10200) in decode throughput across dense and MoE GGUF checkpoints.
- transformers serve exposes GGUF checkpoints via an OpenAI-compatible API, with --reasoning on/off/auto and clients like Jan or Pi connecting at localhost:8000/v1.
- GgufConfig(dequantize=True) enables fine-tuning from GGUF checkpoints; the ggml kernels may extend beyond GGUF to vision, audio, and multimodal models.
Related
Products·building
Bobcat — local LLM inference engine optimized for Apple Silicon Macs
Bobcat is a local LLM inference engine optimized across the stack for consumer MacBooks on Apple Silicon. It runs models locally and claims to be the fastest way to run models like LFM, Qwen3.5, and Clef Flash on Mac. It targets Mac users who want fast local inference without cloud services.
Bobcat is a local LLM inference engine optimized across the stack for consumer MacBooks on Apple Silicon. It runs models locally and claims to be the fastest way to run models like LFM, Qwen3.5, and Clef Flash on Mac. It targets Mac users who want fast local inference without cloud services.
1 engagement1 sourceredditFri, Oct 2, 2026, 06:32 AM
Releases·Tool Release
Voodoo Dynamic Quant open-sourced under MIT license
The creator of Voodoo Quant, a dynamic quantization method claiming SOTA results at aggressive quant levels on smaller Qwen3.5 GGUF models, has released the full toolset under an MIT license on GitHub. The method uses gradient descent to train per-tensor scalar gates that select optimal quant levels, targeting both KL divergence against the BF16 reference and a filesize target. It competes with Unsloth Dynamic 3.0, which remains proprietary.
The creator of Voodoo Quant, a dynamic quantization method claiming SOTA results at aggressive quant levels on smaller Qwen3.5 GGUF models, has released the full toolset under an MIT license on GitHub. The method uses gradient descent to train per-tensor scalar gates that select optimal quant levels, targeting both KL divergence against the BF16 reference and a filesize target. It competes with Unsloth Dynamic 3.0, which remains proprietary.
359 engagement1 sourceredditTue, Sep 15, 2026, 06:59 AM
Social·Community
Developer reverse-engineers Axera AX8850 NPU engine format to run GGUFs on llama.cpp, beating vendor runtime
A developer reverse-engineered the Axera AX8850 NPU's compiled engine format to run GGUF models directly on llama.cpp without vendor model conversion. Running Qwen3-0.6B on an M5Stack LLM-8850 card hosted by a Raspberry Pi 5, the custom backend reaches roughly 1.5× the throughput of Axera's closed runtime. The work removes the vendor compiler dependency for edge LLM inference on the 24 TOPS card.
A developer reverse-engineered the Axera AX8850 NPU's compiled engine format to run GGUF models directly on llama.cpp without vendor model conversion. Running Qwen3-0.6B on an M5Stack LLM-8850 card hosted by a Raspberry Pi 5, the custom backend reaches roughly 1.5× the throughput of Axera's closed runtime. The work removes the vendor compiler dependency for edge LLM inference on the 24 TOPS card.
94 engagement1 sourceredditFri, Aug 28, 2026, 05:33 AM
Social·Community
Hugging Face's Merve shares llama.cpp conference talk covering prefill vs decode and speculative decoding
Merve Noyan of Hugging Face gave a talk at a developer conference covering llama.cpp and core local inference concepts, including prefill vs decode, memory types, and speculative decoding. She shared the slides publicly on Reddit, inviting reuse with attribution.
Merve Noyan of Hugging Face gave a talk at a developer conference covering llama.cpp and core local inference concepts, including prefill vs decode, memory types, and speculative decoding. She shared the slides publicly on Reddit, inviting reuse with attribution.
44 engagement1 sourceredditTue, Oct 6, 2026, 06:02 PM
Products·launched
Outmake — free Mac app that runs open AI models locally
Outmake is a free Mac app that downloads open models from Hugging Face and runs them locally for chat, image, video, voice, and dictation. It removes the need for accounts or cloud services by checking the Mac's memory to tell users which models will run before downloading. It targets Mac users with Apple silicon who want local AI without setup friction.
Outmake is a free Mac app that downloads open models from Hugging Face and runs them locally for chat, image, video, voice, and dictation. It removes the need for accounts or cloud services by checking the Mac's memory to tell users which models will run before downloading. It targets Mac users with Apple silicon who want local AI without setup friction.
1 engagement1 sourceredditSat, Oct 10, 2026, 06:26 AM