Social·Community
Indie 2.6B 'Engram' model offloads knowledge to a 4.3B table; training passes 100M tokens
A solo developer training a small 'Engram' model posted an update on r/LocalLLaMA after roughly a week. The model pairs a 2.6B-parameter transformer with a 4.3B context-aware Engram table, aiming to test whether knowledge can be offloaded from network weights to a lookup table. At just 104M training tokens, completions show the Engram table already improving factual recall, and the developer is seeking a compute sponsor to finish training.
203 engagement1 sourceWed, Sep 23, 2026, 08:08 AM
AI Overview
- The model is 2.6B parameters total with 2.2B trained; frozen ~205M embedding and LM head are SVD-down-projected from OLMo 3 (~5k to 2k).
- A 4.3B context-aware Engram table appears after the first 4-layer block, accessed via attention-based residuals (Moonshot) in all later blocks.
- Architecture uses 3:1 SWA layering (4k/4k/8k sliding windows) across 10 blocks plus a final global layer, 41 layers total.
- Training matches 32 soft teacher logits per Wikipedia token instead of one-hot cross entropy; it passed the 100M-token mark, checkpoints on Hugging Face.
- With only ~104M tokens seen, completions show Engram stores repeated data — Aristotle prompts degrade to word definitions when the table is zeroed.
- The creator is self-funding training on Massed Compute after Verda and Qubrid declined to sponsor completion.
Related
Releases·Tool Release
DeepSeek Harness
DeepSeek Harness link shared on Hacker News
DeepSeek Harness link shared on Hacker News
4.8k engagement2 sourceshackernews, xFri, Oct 2, 2026, 03:11 AM
Social·Community
Looping on frontier models
A user reports watching a DeepSeek model enter an infinite reasoning loop on 2-3 specific tokens, and asks whether this behavior still affects frontier models or if it stemmed from a quantized deployment during peak load.
A user reports watching a DeepSeek model enter an infinite reasoning loop on 2-3 specific tokens, and asks whether this behavior still affects frontier models or if it stemmed from a quantized deployment during peak load.
10 engagement1 sourceredditFri, Oct 2, 2026, 08:13 AM
Developments·Paper
Researchers propose tokenizer-agnostic hashing for DeepSeek's Engram memory module
A new paper proposes a change to DeepSeek's Engram conditional memory module, replacing token-level N-gram hashing with a tokenizer-agnostic routine. This enables Engram embeddings to be reused across models with different tokenizers, avoiding retraining from scratch.
A new paper proposes a change to DeepSeek's Engram conditional memory module, replacing token-level N-gram hashing with a tokenizer-agnostic routine. This enables Engram embeddings to be reused across models with different tokenizers, avoiding retraining from scratch.
0 engagement1 sourcearxivFri, Jul 31, 2026, 06:32 AM
Releases·Model Release
DWARF-55M-Base released, a nearly all-sparse attention model
The DWARF-55M-Base model, the first from the DWARF architecture, has been released. It uses 9 Dynamic Sparse Query-Gather (DSQG) layers and a single full causal attention layer at 25% depth, achieving reliable retrieval up to 2048 context. This sparse attention design could reduce computational costs while maintaining performance.
The DWARF-55M-Base model, the first from the DWARF architecture, has been released. It uses 9 Dynamic Sparse Query-Gather (DSQG) layers and a single full causal attention layer at 25% depth, achieving reliable retrieval up to 2048 context. This sparse attention design could reduce computational costs while maintaining performance.
79 engagement1 sourceredditMon, Jul 20, 2026, 04:21 PM
Releases·Model Release
Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
Researchers released Daedalus-150M, a 150M-parameter language model architected specifically for CPU inference. It uses convolution in 12 of 18 blocks to avoid re-reading a growing cache, achieving 47.31 on a five-task benchmark against a 42.20 baseline.
Researchers released Daedalus-150M, a 150M-parameter language model architected specifically for CPU inference. It uses convolution in 12 of 18 blocks to avoid re-reading a growing cache, achieving 47.31 on a five-task benchmark against a 42.20 baseline.
0 engagement1 sourcearxivThu, Aug 20, 2026, 04:09 PM