Releases·Model Release
Qwengram-0.8B: Qwen3.8 Flash-Next's n-gram memory grafted into Qwen3.5-0.8B, cutting validation perplexity 5.05%
An independent researcher released Qwengram-0.8B, a Qwen3.5-0.8B model augmented with Qwen3.8 Flash-Next's pretrained ~51B-parameter PLE n-gram memory via a small trained reader and dynamic token-level gate, cutting validation perplexity from 18.28 to 17.35. The model, GGUFs, training code, and a modified llama.cpp runtime are public, with the large PLE kept as an external quantized sidecar. The result suggests a recipe for scaling memory-augmentation to larger Qwen backbones like the 35B-A3B MoE.
444 engagement1 sourceFri, Sep 25, 2026, 12:46 PM
AI Overview
- Qwengram-0.8B cuts Qwen3.5-0.8B validation perplexity from 18.2759 to 17.3534 (5.05%) by transferring Qwen3.8 Flash-Next's ~51B-parameter PLE memory, frozen alongside the backbone, with an R=1 reader trained 15M tokens and a dynamic token-level gate injecting memory at decoder layers 3 and 9
- Pretrained PLE beat random and permuted-memory controls; 20M-token readers regressed math, so the author kept a 15M-token balanced checkpoint with no backbone fine-tuning
- Fixed late-layer injection hurt LAMBADA, but dynamic token arbitration recovered much of the tradeoff, and learned token placement beat shuffled placement at equal memory budget
- The author released GGUFs on Hugging Face, training and control code on GitHub, and a modified llama.cpp runtime where the ~51B PLE ships as an external quantized sidecar retaining 99.1% of the BF16 reader NLL gain at Q8_0 on a separate WikiText-2 runtime test
Related
Developments·Paper
MoNe: Modular Neural Memory Enables O(1) Long-Context Inference
MoNe is a lightweight modular neural memory that attaches to any frozen pretrained Transformer, enabling long-context inference without retraining. It decouples inference cost from context length, achieving O(N) preprocessing and O(1) query cost with peak GPU memory that does not grow with N.
MoNe is a lightweight modular neural memory that attaches to any frozen pretrained Transformer, enabling long-context inference without retraining. It decouples inference cost from context length, achieving O(N) preprocessing and O(1) query cost with peak GPU memory that does not grow with N.
0 engagement1 sourcearxivTue, Aug 18, 2026, 10:28 AM
Social·Community
Arcee finetunes GQA out of Qwen 3.8 27B, open-sources DistilKit for KV-cache reduction
A LocalLLaMa post highlights Arcee's work removing the heavy grouped-query-attention KV cache overhead from Qwen 3.8 27B via finetuning, motivated by complaints about the model's API pricing versus DeepSeek V4 Flash. Arcee open-sourced both the resulting model and its DistilKit tooling. For local-model practitioners, this suggests KV cache costs may not be an architectural constant but something removable with finetuning.
A LocalLLaMa post highlights Arcee's work removing the heavy grouped-query-attention KV cache overhead from Qwen 3.8 27B via finetuning, motivated by complaints about the model's API pricing versus DeepSeek V4 Flash. Arcee open-sourced both the resulting model and its DistilKit tooling. For local-model practitioners, this suggests KV cache costs may not be an architectural constant but something removable with finetuning.
7 engagement1 sourceredditTue, Sep 1, 2026, 06:04 PM
Social·Community
Multi-hour llama.cpp optimization experiments on Qwen MoE models, patches, benchmarks, and reproduction guides
A Reddit user shared a multi-hour llama.cpp optimization investigation, run with an agent called Astra, targeting local MoE inference for Qwen3.6-35B-A3B and Qwen3.8 Flash-Next without changing weights or quantization. Patches, benchmarks, and reproduction guides are published on GitHub. Some workloads showed substantial gains, notably prompt processing and source-code editing, but regressions and quality caveats remain.
A Reddit user shared a multi-hour llama.cpp optimization investigation, run with an agent called Astra, targeting local MoE inference for Qwen3.6-35B-A3B and Qwen3.8 Flash-Next without changing weights or quantization. Patches, benchmarks, and reproduction guides are published on GitHub. Some workloads showed substantial gains, notably prompt processing and source-code editing, but regressions and quality caveats remain.
23 engagement1 sourceredditFri, Sep 18, 2026, 11:46 AM
Releases·Tool Release
llama.cpp dev lands three Metal MoE PRs for decode, prefill, and KV caching, tested on Qwen3-30B-A3B
A LocalLLaMA developer with an M5 MacBook Pro 24GB opened three llama.cpp pull requests improving Metal MoE decode, prefill, and caching, tested on Qwen3-30B-A3B. Because testing is limited to an M5 and an older M1 Pro 32GB, the author is recruiting community testers, including for Qwen3.8 Flash Next.
A LocalLLaMA developer with an M5 MacBook Pro 24GB opened three llama.cpp pull requests improving Metal MoE decode, prefill, and caching, tested on Qwen3-30B-A3B. Because testing is limited to an M5 and an older M1 Pro 32GB, the author is recruiting community testers, including for Qwen3.8 Flash Next.
1 engagement1 sourceredditThu, Sep 3, 2026, 02:13 PM