Releases·Tool Release
llama.cpp adds Decision Models support, with models hosted on Hugging Face
llama.cpp adds Decision Models, shared via Hugging Face
561 engagement1 sourceFri, Oct 2, 2026, 02:25 PM
AI Overview
- llama.cpp introduces Decision Models support, with a Hugging Face page documenting the feature
Entities
Related
Products·launched
hearim: open-source gateway turning ordinary local LLMs into a Jev-compatible decision API
hearim is an open-source gateway that lets existing local or hosted LLMs act as a Jev-compatible decision API. Given data and a set of choices, it returns a decision with probabilities without task-specific fine-tuning or custom weights. It targets developers running models via llama.cpp, vLLM, SGLang, Ollama, or OpenAI-compatible servers.
hearim is an open-source gateway that lets existing local or hosted LLMs act as a Jev-compatible decision API. Given data and a set of choices, it returns a decision with probabilities without task-specific fine-tuning or custom weights. It targets developers running models via llama.cpp, vLLM, SGLang, Ollama, or OpenAI-compatible servers.
1 engagement1 sourceredditMon, Sep 21, 2026, 07:59 AM
Releases·Tool Release
llama.cpp PR adds adaptive MTP depth selection
A llama.cpp contributor filed PR#27210 adding an adaptive multi-token prediction (MTP) mode that dynamically sets MTP depth via a counting-style state machine, removing the need to manually tune depth. Early benchmarks show roughly 3% performance loss versus fixed MTP depth 3 on dense prose, with potential gains in some scenarios.
A llama.cpp contributor filed PR#27210 adding an adaptive multi-token prediction (MTP) mode that dynamically sets MTP depth via a counting-style state machine, removing the need to manually tune depth. Early benchmarks show roughly 3% performance loss versus fixed MTP depth 3 on dense prose, with potential gains in some scenarios.
199 engagement1 sourceredditMon, Aug 17, 2026, 06:05 PM
Releases·Tool Release
llama.cpp dev lands three Metal MoE PRs for decode, prefill, and KV caching, tested on Qwen3-30B-A3B
A LocalLLaMA developer with an M5 MacBook Pro 24GB opened three llama.cpp pull requests improving Metal MoE decode, prefill, and caching, tested on Qwen3-30B-A3B. Because testing is limited to an M5 and an older M1 Pro 32GB, the author is recruiting community testers, including for Qwen3.8 Flash Next.
A LocalLLaMA developer with an M5 MacBook Pro 24GB opened three llama.cpp pull requests improving Metal MoE decode, prefill, and caching, tested on Qwen3-30B-A3B. Because testing is limited to an M5 and an older M1 Pro 32GB, the author is recruiting community testers, including for Qwen3.8 Flash Next.
1 engagement1 sourceredditThu, Sep 3, 2026, 02:13 PM
Social·Community
Community questions why Minimax M3 branch not merged into llama.cpp main
A Reddit user asked why the Minimax M3 model requires a separate branch in llama.cpp instead of being merged into the main branch. The question highlights ongoing integration challenges for new model architectures in the popular inference framework.
A Reddit user asked why the Minimax M3 model requires a separate branch in llama.cpp instead of being merged into the main branch. The question highlights ongoing integration challenges for new model architectures in the popular inference framework.
61 engagement1 sourceredditSat, Jul 18, 2026, 01:31 PM
Releases·Tool Release
Developer releases Compact Rollback MTP for llama.cpp to cut speculative decoding memory cost on Qwen models
A developer modified llama.cpp's MTP (multi-token prediction) speculative decoding to reduce its memory footprint, targeting low-VRAM setups like running Qwen models on 16GB cards. On a tuned IQ4 quant of a Qwen 27B-class model, the modification increased available context by ~5k tokens and raised token generation speed 17.35% versus standard MTP. It matters to local-inference practitioners who want speculative decoding speedups without sacrificing context.
A developer modified llama.cpp's MTP (multi-token prediction) speculative decoding to reduce its memory footprint, targeting low-VRAM setups like running Qwen models on 16GB cards. On a tuned IQ4 quant of a Qwen 27B-class model, the modification increased available context by ~5k tokens and raised token generation speed 17.35% versus standard MTP. It matters to local-inference practitioners who want speculative decoding speedups without sacrificing context.
33 engagement1 sourceredditMon, Aug 31, 2026, 04:58 PM