Releases·Tool Release
llama.cpp contributor lands Metal MoE decode speedup for Apple Silicon
A llama.cpp contributor opened a pull request adding a Metal MoE optimization that speeds up decode of quantized models on Apple Silicon. On their test workload, a 35B A3B model at IQ3_XXS went from about 65.6 to 73.9 tok/s. The author is asking the community for before/after numbers and has a follow-up for prefill in progress.
15 engagement1 sourceTue, Sep 1, 2026, 08:55 AM
AI Overview
- Pull request ggml-org/llama.cpp#28086 implements the Metal MoE optimization
- Test workload: Qwen coder 35B A3B at IQ3_XXS quantization, decode 65.6 → 73.9 tok/s
- Author requests community before/after benchmarks to validate the speedup
- A follow-up PR targeting prefill speedups is planned soon
Related
Releases·Tool Release
llama.cpp dev lands three Metal MoE PRs for decode, prefill, and KV caching, tested on Qwen3-30B-A3B
A LocalLLaMA developer with an M5 MacBook Pro 24GB opened three llama.cpp pull requests improving Metal MoE decode, prefill, and caching, tested on Qwen3-30B-A3B. Because testing is limited to an M5 and an older M1 Pro 32GB, the author is recruiting community testers, including for Qwen3.8 Flash Next.
A LocalLLaMA developer with an M5 MacBook Pro 24GB opened three llama.cpp pull requests improving Metal MoE decode, prefill, and caching, tested on Qwen3-30B-A3B. Because testing is limited to an M5 and an older M1 Pro 32GB, the author is recruiting community testers, including for Qwen3.8 Flash Next.
1 engagement1 sourceredditThu, Sep 3, 2026, 02:13 PM
Releases·Tool Release
llama.cpp Metal PR #29869 adds few-row MMA mat-mul and batched copies, boosting DFlash2 speculative decoding up to 4x
A llama.cpp pull request by pratiknarola-t adds Metal few-row MMA matmul kernels and batched copies to accelerate speculative decoding on Apple GPUs. Benchmarks with a Qwen3.8-27B DFlash2 setup on llama-server show decode throughput rising from 16.8 tok/s to 110.0 tok/s on code prompts with no batch dimension. The work targets Apple Metal users running speculative decoding with draft models.
A llama.cpp pull request by pratiknarola-t adds Metal few-row MMA matmul kernels and batched copies to accelerate speculative decoding on Apple GPUs. Benchmarks with a Qwen3.8-27B DFlash2 setup on llama-server show decode throughput rising from 16.8 tok/s to 110.0 tok/s on code prompts with no batch dimension. The work targets Apple Metal users running speculative decoding with draft models.
3 engagement1 sourceredditTue, Oct 6, 2026, 05:44 AM
Social·Community
MTP in llama.cpp now decodes competitively with ds4 using GLM 5.3 Flash
A llama.cpp user reports multi-token prediction (MTP) now delivers competitive decode speeds on Apple Silicon, with GLM 5.3 Flash becoming the first model to outperform DeepSeek 4 (ds4) in their testing. MTP previously offered no advantage in llama.cpp. Prompt processing remains slower, and an upcoming Qwen model test is viewed as the real benchmark.
A llama.cpp user reports multi-token prediction (MTP) now delivers competitive decode speeds on Apple Silicon, with GLM 5.3 Flash becoming the first model to outperform DeepSeek 4 (ds4) in their testing. MTP previously offered no advantage in llama.cpp. Prompt processing remains slower, and an upcoming Qwen model test is viewed as the real benchmark.
16 engagement1 sourceredditThu, Oct 8, 2026, 03:56 PM
Developments·Breakthrough
Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp
A GitHub project demonstrates that running LLM inference with llama.cpp inside macOS virtual machines on Apple Silicon can be 11–16× faster than on the host. The technique likely leverages VM-specific optimizations or hardware acceleration, offering significant performance gains for practitioners.
A GitHub project demonstrates that running LLM inference with llama.cpp inside macOS virtual machines on Apple Silicon can be 11–16× faster than on the host. The technique likely leverages VM-specific optimizations or hardware acceleration, offering significant performance gains for practitioners.
58 engagement1 sourcehackernewsTue, Aug 11, 2026, 02:50 PM
Releases·Tool Release
AMD's llama.cpp branch boosts prompt processing up to 2x on Strix Halo
AMD maintains its own llama.cpp branch with ROCm/Hip patches that can more than double prompt processing speed on dense models. A user reports 550 tokens/s with a 14B dense model versus 230 tokens/s on standard llama.cpp, though token generation is about 15% slower than with Vulkan.
AMD maintains its own llama.cpp branch with ROCm/Hip patches that can more than double prompt processing speed on dense models. A user reports 550 tokens/s with a 14B dense model versus 230 tokens/s on standard llama.cpp, though token generation is about 15% slower than with Vulkan.
21 engagement1 sourceredditSun, Aug 23, 2026, 08:05 AM