Releases·Tool Release
Developer releases Compact Rollback MTP for llama.cpp to cut speculative decoding memory cost on Qwen models
A developer modified llama.cpp's MTP (multi-token prediction) speculative decoding to reduce its memory footprint, targeting low-VRAM setups like running Qwen models on 16GB cards. On a tuned IQ4 quant of a Qwen 27B-class model, the modification increased available context by ~5k tokens and raised token generation speed 17.35% versus standard MTP. It matters to local-inference practitioners who want speculative decoding speedups without sacrificing context.
33 engagement1 sourceMon, Aug 31, 2026, 04:58 PM
AI Overview
- Compact Rollback MTP reduces the memory cost of MTP speculative decoding in llama.cpp, trading less context for more speed on constrained VRAM
- On a 16GB setup running a tuned IQ4 quant, it yields roughly +5k tokens of available context versus standard MTP
- Benchmark: standard MTP with max draft 2 gives 72,192 context at 39.53 t/s; Compact Rollback with max draft 5 gives 77,312 context at 46.39 t/s
- Higher max draft (n) in MTP increases the chance of speed gains but raises memory cost, which the modification mitigates
- The author targets setups like Qwen 27B on 16GB VRAM
Related
Social·Community
PSA: llama.cpp -cram should be increased for agentic workflows (default is 8192)
A llama.cpp user reports that the default -cram setting of 8192 can cause full context reprocessing on every turn in long multiturn agentic projects. Increasing it to 20480 enabled large-context use with Qwen at 262K context, yielding major speedups at the cost of higher RAM usage.
A llama.cpp user reports that the default -cram setting of 8192 can cause full context reprocessing on every turn in long multiturn agentic projects. Increasing it to 20480 enabled large-context use with Qwen at 262K context, yielding major speedups at the cost of higher RAM usage.
90 engagement1 sourceredditThu, Sep 24, 2026, 03:44 PM
Social·Community
16GB (and in many cases 12GB) is the max vram most people will ever reasonably have
A LocalLLaMA discussion argues that 16GB VRAM is the practical maximum for most people, and 12GB in much of the world. The poster contends recent model efficiency gains — including agentic coding with quantized Qwen 27B — have made this tier genuinely usable. Practitioners weigh hardware cost realities against the sub's high-end skew.
A LocalLLaMA discussion argues that 16GB VRAM is the practical maximum for most people, and 12GB in much of the world. The poster contends recent model efficiency gains — including agentic coding with quantized Qwen 27B — have made this tier genuinely usable. Practitioners weigh hardware cost realities against the sub's high-end skew.
1.1k engagement1 sourceredditMon, Sep 21, 2026, 12:21 PM
Releases·Tool Release
llama.cpp v0.6.0 released with MTP speculative decoding for Qwen4Exp
llama.cpp released version 0.6.0, adding MTP (multi-token prediction) speculative decoding support for Qwen4Exp among other changes. The release is notable for practitioners running Qwen4Exp locally, since MTP speculative decoding can speed up inference. Per the rules for thin excerpts, no further details beyond the title are asserted.
llama.cpp released version 0.6.0, adding MTP (multi-token prediction) speculative decoding support for Qwen4Exp among other changes. The release is notable for practitioners running Qwen4Exp locally, since MTP speculative decoding can speed up inference. Per the rules for thin excerpts, no further details beyond the title are asserted.
317 engagement1 sourceredditMon, Oct 5, 2026, 06:58 PM
Developments·Benchmark
Benchmark: DFlash2 vs MTP on RTX 5090 with Qwen 3.8 27B
A Reddit benchmark compares DFlash2 and MTP on an RTX 5090 with Qwen 3.8 27B, showing DFlash2 offers up to +20% generation speed but reduces context size by 38%. The advantage diminishes with large contexts (>100k) due to compaction.
A Reddit benchmark compares DFlash2 and MTP on an RTX 5090 with Qwen 3.8 27B, showing DFlash2 offers up to +20% generation speed but reduces context size by 38%. The advantage diminishes with large contexts (>100k) due to compaction.
57 engagement2 sourcesredditFri, Aug 21, 2026, 11:48 AM
Social·Community
User benchmarks llama.cpp RPC with 5070 Ti and 1080 Ti over gigabit ethernet
A user tested llama.cpp's RPC feature with a 5070 Ti and a 1080 Ti over gigabit ethernet, finding it functional but requiring a trade-off between prefill and generation speed. Using UD-Q4_K_XL with q8 KV cache and 96k context, they achieved 350 pp/s and 36 tg/s when prioritizing generation, or 560 pp/s and 19 tg/s when prioritizing prefill. The goal was to run Qwen 3.8 27B without compromising KV cache or quant.
A user tested llama.cpp's RPC feature with a 5070 Ti and a 1080 Ti over gigabit ethernet, finding it functional but requiring a trade-off between prefill and generation speed. Using UD-Q4_K_XL with q8 KV cache and 96k context, they achieved 350 pp/s and 36 tg/s when prioritizing generation, or 560 pp/s and 19 tg/s when prioritizing prefill. The goal was to run Qwen 3.8 27B without compromising KV cache or quant.
90 engagement1 sourceredditTue, Aug 18, 2026, 10:07 AM