Releases·Tool Release
llama.cpp v0.4.0 adds Qwen3.8-Flash-Next and Nemotron-3-Puzzle-75B-A9B support, lazy tensor loading, ggml 0.23.0
llama.cpp shipped v0.4.0, adding initial support for Qwen3.8-Flash-Next and Nemotron-3-Puzzle models. The release also brings on-demand tensor reading, per-slot server context limits, video input options, and a ggml update to 0.23.0 featuring major sparse flash attention and RDMA work.
6 engagement1 sourceFri, Sep 4, 2026, 07:56 PM
AI Overview
- Adds initial Qwen3.8-Flash-Next (qwen4exp) architecture support and NVIDIA Nemotron-3-Puzzle-75B-A9B support, plus nanbeige4.2-3B
- Introduces on-demand lazy tensor reading via llama_lazy_mode and --lazy-mode, with a quantizer RAM cap and row-slab streaming to prevent load-time RAM peaks
- Bumps ggml to v0.23.0, adding sparse flash attention (ggml_flash_attn_ext_set_n_kv_max), RPC event/async APIs, and Apple RDMA transport support
- Adds sparse flash attention for DeepSeek-V4/GLM and Qwen4exp, plus a --n-cpu-ffn option and per-layer expert routing/FFN
- Server gains per-slot context limits, data: URL media support, preserve_reasoning enabled by default, and synthetic spec acceptance options
- Multimodal work adds mtmd_tokenize_from_parts(), --video-* input options, DeepSeek-V4-Flash-Vision-Exp, and fixes for Gemma-4 vision and Qwen3-TTS
Related
Releases·Tool Release
llama.cpp v0.5.0 adds HRM-Text/MiMo-V2.6 support, CUDA and Metal fusion optimizations, ggml 0.25.0
ggml-org released llama.cpp v0.5.0 on 2026-09-23, focusing on backend performance and correctness, broader model coverage, and more robust server/router operation. The release bundles ggml 0.25.0, CUDA and Metal/SYCL/Vulkan optimizations, new model support (HRM-Text/DFM Mimir 1B, MiMo-V2.6, HunyuanOCR DFlash), and API changes like LoRA loading from an open FILE handle. Practitioners running local inference gain faster attention and MoE paths plus more stable multi-model serving.
ggml-org released llama.cpp v0.5.0 on 2026-09-23, focusing on backend performance and correctness, broader model coverage, and more robust server/router operation. The release bundles ggml 0.25.0, CUDA and Metal/SYCL/Vulkan optimizations, new model support (HRM-Text/DFM Mimir 1B, MiMo-V2.6, HunyuanOCR DFlash), and API changes like LoRA loading from an open FILE handle. Practitioners running local inference gain faster attention and MoE paths plus more stable multi-model serving.
1 engagement1 sourcegithubWed, Sep 23, 2026, 08:50 PM
Releases·Tool Release
Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant
A developer released Slipstream, an open-source C++ Metal inference engine for Apple Silicon with native SSD expert streaming and speculative drafting. It runs the 95.5 GiB Qwen3.8-Flash-Next checkpoint at 41–52 tok/s on a 64GB M5 Pro Mac, a 1.76x speedup over a llama.cpp fork, with decode staying at 33–44 tok/s out to 130k context. An optional Swift KV-sparse variant of the model raises overall benchmark accuracy by 2.8% with no decode-speed cost.
A developer released Slipstream, an open-source C++ Metal inference engine for Apple Silicon with native SSD expert streaming and speculative drafting. It runs the 95.5 GiB Qwen3.8-Flash-Next checkpoint at 41–52 tok/s on a 64GB M5 Pro Mac, a 1.76x speedup over a llama.cpp fork, with decode staying at 33–44 tok/s out to 130k context. An optional Swift KV-sparse variant of the model raises overall benchmark accuracy by 2.8% with no decode-speed cost.
54 engagement1 sourceredditThu, Oct 1, 2026, 08:21 PM
Releases·Tool Release
llama.cpp PR adds TENSOR_READ_LAZY for Qwen 3.8 Next Flash engrams
A llama.cpp pull request by ngxson introduces a TENSOR_READ_LAZY mode in the model loader, allowing Qwen 3.8 Next Flash (Qwen 4) engrams to be read lazily without requiring them to reside in VRAM or RAM. This could reduce memory pressure for large models.
A llama.cpp pull request by ngxson introduces a TENSOR_READ_LAZY mode in the model loader, allowing Qwen 3.8 Next Flash (Qwen 4) engrams to be read lazily without requiring them to reside in VRAM or RAM. This could reduce memory pressure for large models.
511 engagement2 sourcesredditThu, Aug 27, 2026, 02:49 PM
Releases·Tool Release
llama.cpp v0.4.1 adds Maple 20B-A1B, Tencent Hy 4, Spark2.5 support and bumps ggml to v0.24.0
The llama.cpp project shipped v0.4.1 on GitHub, adding support for Maple 20B-A1B (ternary MoE, CPU), Tencent Hy 4, and Spark2.5, plus Kimi-K3 recurrent-state rollback. The release refactors JSON schema handling and chat parsing, fixes MTP KV cache and GDN normalization bugs, and bumps ggml to v0.24.0 with expanded backend coverage. It matters to anyone running local inference on the affected model families.
The llama.cpp project shipped v0.4.1 on GitHub, adding support for Maple 20B-A1B (ternary MoE, CPU), Tencent Hy 4, and Spark2.5, plus Kimi-K3 recurrent-state rollback. The release refactors JSON schema handling and chat parsing, fixes MTP KV cache and GDN normalization bugs, and bumps ggml to v0.24.0 with expanded backend coverage. It matters to anyone running local inference on the affected model families.
15 engagement1 sourcegithubMon, Sep 14, 2026, 06:27 PM
Releases·Tool Release
Llama.cpp fork adds adaptive speculation, boosting token generation up to 50%
A developer introduced adaptive speculation for Llama.cpp, a fork that automatically adjusts the number of suggested tokens based on content type, improving token generation by up to 50% over mainline. The feature targets dense models like Qwen3.8, where MTP and DFlash settings vary by content.
A developer introduced adaptive speculation for Llama.cpp, a fork that automatically adjusts the number of suggested tokens based on content type, improving token generation by up to 50% over mainline. The feature targets dense models like Qwen3.8, where MTP and DFlash settings vary by content.
141 engagement1 sourceredditTue, Aug 25, 2026, 11:35 AM