Releases·Tool Release
llama.cpp b10726 changes --lazy-mode default to auto, slowing inference for large-table models
llama.cpp build b10726 changed the default --lazy-mode to auto, keeping large embedding tables mmap'd on disk instead of loading them into RAM. A user reports the 51B-parameter PLE n-gram table of Qwen 3.8 Flash Next stays on disk even with --load-mode none, causing a 50% prompt-processing and 15% token-generation speed penalty. Users with sufficient RAM should pass --lazy-mode off to restore prior behavior.
65 engagement1 sourceMon, Aug 31, 2026, 08:57 PM
AI Overview
- The default change mmap's the table and reads rows on demand during inference, even with --load-mode none
- Affected model: Qwen 3.8 Flash Next, whose PLE n-gram embedding table has 51B parameters
- Reported penalties: 50% prompt processing, 15% token generation
- Fix: pass --lazy-mode off to load the table into RAM as before
Related
Social·Community
Qwen3.8 27B with embeddings
A Reddit user reports running Qwen3.8 27B's built-in embeddings in llama.cpp via --embedding and --pooling mean, getting decent results. Enabling embeddings halves token generation speed, and the user asks how to switch tasks without reloading the model.
A Reddit user reports running Qwen3.8 27B's built-in embeddings in llama.cpp via --embedding and --pooling mean, getting decent results. Enabling embeddings halves token generation speed, and the user asks how to switch tasks without reloading the model.
19 engagement1 sourceredditTue, Sep 15, 2026, 11:25 AM
Releases·Tool Release
llama.cpp PR adds TENSOR_READ_LAZY for Qwen 3.8 Next Flash engrams
A llama.cpp pull request by ngxson introduces a TENSOR_READ_LAZY mode in the model loader, allowing Qwen 3.8 Next Flash (Qwen 4) engrams to be read lazily without requiring them to reside in VRAM or RAM. This could reduce memory pressure for large models.
A llama.cpp pull request by ngxson introduces a TENSOR_READ_LAZY mode in the model loader, allowing Qwen 3.8 Next Flash (Qwen 4) engrams to be read lazily without requiring them to reside in VRAM or RAM. This could reduce memory pressure for large models.
511 engagement2 sourcesredditThu, Aug 27, 2026, 02:49 PM
Social·Community
We have implanted 100 facts into the engram table of Qwen 3.8 Flash Next, and we have now created a website to explain it.
A team describing an 'engraft-engram' method says it implanted 100 facts into the engram table of Qwen 3.8 Flash Next without altering model weights, and launched a website with a simulator to explain the approach. The team says DeepSeek v4.1 Flash support is planned. The claims are self-reported in a Reddit post with minimal engagement and no independent verification in the posts.
A team describing an 'engraft-engram' method says it implanted 100 facts into the engram table of Qwen 3.8 Flash Next without altering model weights, and launched a website with a simulator to explain the approach. The team says DeepSeek v4.1 Flash support is planned. The claims are self-reported in a Reddit post with minimal engagement and no independent verification in the posts.
1 engagement1 sourceredditSat, Sep 26, 2026, 01:19 PM
Social·Community
Users report llama.cpp memory inefficiency with Qwen context
A user reports that llama.cpp is significantly more memory-hungry for Qwen models than for Muse Glimmer, despite Qwen's smaller per-token state. This limits concurrent context slots and batched performance on the same hardware.
A user reports that llama.cpp is significantly more memory-hungry for Qwen models than for Muse Glimmer, despite Qwen's smaller per-token state. This limits concurrent context slots and batched performance on the same hardware.
22 engagement1 sourceredditFri, Aug 14, 2026, 11:52 AM
Products·launched
Ngram hot-swappable knowledge injector for Qwen models in llama.cpp
A modification to llama.cpp that patches the Ngram PLE table of Qwen-3.8-Next-Flash models in memory on every prompt, letting users inject new knowledge in real time without reloading the model. It targets local LLM users wanting to update model knowledge on the fly, though output control is limited because embeddings are injected early in the layers.
A modification to llama.cpp that patches the Ngram PLE table of Qwen-3.8-Next-Flash models in memory on every prompt, letting users inject new knowledge in real time without reloading the model. It targets local LLM users wanting to update model knowledge on the fly, though output control is limited because embeddings are injected early in the layers.
267 engagement1 sourceredditThu, Sep 3, 2026, 11:40 AM