Releases·Tool Release
focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737)
A developer open-sourced focus-llama, a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737, Google DeepMind and KAIST AI). The model declares which context chunks it needs via output tags, and the engine restricts subsequent tokens' attention accordingly — no training required. The paper reports decode-time reductions to 0.71× (Gemma) and 0.77× (Qwen) versus vanilla, though the fork itself is not yet benchmarked.
41 engagement1 sourceSun, Sep 20, 2026, 02:07 PM
AI Overview
- focus-llama fork lets llama-server drop KV token ranges mid-prefill or post-prefill via model-declared tags; paper reports decode at 0.71× (Gemma), 0.77× (Qwen) on vLLM
- Fork needed because llama.cpp lacks paged block tables and CUDA single-token decode doesn't skip masked chunks; no fork benchmarks yet
Related
Developments·Paper
arXiv Paper Introduces Declarative Attention, Letting Language Models Declare Their Own Attention Regions
An arxiv paper argues that language models concentrate attention on a small fraction of context yet still read the entire KV cache at every step, so long-context decoding wastes compute. The paper positions learned self-controlled attention against proxy-score token pre-selection, which still costs O(N) per step.
An arxiv paper argues that language models concentrate attention on a small fraction of context yet still read the entire KV cache at every step, so long-context decoding wastes compute. The paper positions learned self-controlled attention against proxy-score token pre-selection, which still costs O(N) per step.
0 engagement1 sourcearxivWed, Sep 2, 2026, 03:43 PM
Developments·Paper
On-Demand Attention: Language Models Know When to Recall
An arXiv paper proposes On-Demand Attention (ODA), a decoding method where a lightweight recall head predicts when global attention helps and selectively invokes it, leaving pretrained weights untouched. Implemented in vLLM with GPU-side conditional execution, ODA recovers most of local attention's quality loss while cutting global reads and speeding up decoding at long context.
An arXiv paper proposes On-Demand Attention (ODA), a decoding method where a lightweight recall head predicts when global attention helps and selectively invokes it, leaving pretrained weights untouched. Implemented in vLLM with GPU-side conditional execution, ODA recovers most of local attention's quality loss while cutting global reads and speeding up decoding at long context.
0 engagement1 sourcearxivThu, Sep 17, 2026, 05:24 PM
Releases·Tool Release
llama.cpp PR adds adaptive MTP depth selection
A llama.cpp contributor filed PR#27210 adding an adaptive multi-token prediction (MTP) mode that dynamically sets MTP depth via a counting-style state machine, removing the need to manually tune depth. Early benchmarks show roughly 3% performance loss versus fixed MTP depth 3 on dense prose, with potential gains in some scenarios.
A llama.cpp contributor filed PR#27210 adding an adaptive multi-token prediction (MTP) mode that dynamically sets MTP depth via a counting-style state machine, removing the need to manually tune depth. Early benchmarks show roughly 3% performance loss versus fixed MTP depth 3 on dense prose, with potential gains in some scenarios.
199 engagement1 sourceredditMon, Aug 17, 2026, 06:05 PM
Social·Community
Reddit user asks which llama.cpp fork best optimizes AMD MI50 GPUs, citing mx-llama.cpp and exabit-io forks
A Reddit user with an AMD Instinct MI50 asks which llama.cpp fork best squeezes performance from the card, citing two candidate forks: mxxm-t/mx-llama.cpp and exabit-io/llama.cpp. The thread reflects ongoing community interest in optimizing inference on older AMD data-center GPUs via unofficial forks.
A Reddit user with an AMD Instinct MI50 asks which llama.cpp fork best squeezes performance from the card, citing two candidate forks: mxxm-t/mx-llama.cpp and exabit-io/llama.cpp. The thread reflects ongoing community interest in optimizing inference on older AMD data-center GPUs via unofficial forks.
9 engagement1 sourceredditSat, Sep 19, 2026, 06:44 AM