Releases·Model Releasefeatured
New Architecture from Percepta: Spotlight — separating intelligence from memory, allowing knowledge and skills to grow without changing the model's weights.
Percepta unveiled Spotlight, an architecture that replaces attention with a learnable, unbounded writable memory. An intelligence module of fixed size performs computation while the model itself reads from, writes to, and overwrites memory token by token, letting knowledge and skills grow without changing weights. The company claims it is the first architecture to grow memory indefinitely without increasing access cost.
259 engagement1 sourceFri, Oct 2, 2026, 05:47 PM
AI Overview
- Percepta's Spotlight architecture replaces attention with a writable, unbounded memory indexed by learned lookups, per Percepta's blog posts.
- Each token touches a fixed small number of memory cells regardless of memory size, so access cost stays constant as memory grows.
- Unlike mixture-of-experts models, which activate a fixed fraction of capacity, Spotlight is arbitrarily sparse and its used memory fraction can shrink arbitrarily.
- Spotlight separates a fixed-size intelligence module from memory holding knowledge, procedures, and working state; weights never change as memory grows.
- The model itself decides token-by-token what to load into memory and when to overwrite it, gaining new capabilities without retraining.
Related
Developments·Paper
Memory Attention: token-indexed memory replaces value projection for faster LLM inference
A new arXiv paper introduces Memory Attention (MA), which replaces a language model's dedicated value projection with layer-specific token memory combined with contextual keys. At inference, normalization folds into the memory tables so value construction becomes lookup and addition, and token-indexed retrieval enables CPU offloading with prefetching to cut GPU parameter storage. Experiments under matched training token budgets show improved language modeling and downstream performance.
A new arXiv paper introduces Memory Attention (MA), which replaces a language model's dedicated value projection with layer-specific token memory combined with contextual keys. At inference, normalization folds into the memory tables so value construction becomes lookup and addition, and token-indexed retrieval enables CPU offloading with prefetching to cut GPU parameter storage. Experiments under matched training token budgets show improved language modeling and downstream performance.
0 engagement1 sourcearxivWed, Sep 23, 2026, 05:04 PM
Developments·Paper
ARM: Attention with Routed-Memory for Learnable Sparse Control
A new arXiv paper introduces Attention with Routed Memory (ARM), a fixed-size, fully differentiable KV caching structure organized as a hierarchical router. Instead of evicting or pruning tokens, ARM uses Gumbel-Softmax slot selection and sigmoid-gated updates that softly merge new and stored information, and a learned policy adapts memory usage at inference. The authors report superior performance and efficiency versus fixed KV-caching approaches on commonsense and long-context reasoning benchmarks.
A new arXiv paper introduces Attention with Routed Memory (ARM), a fixed-size, fully differentiable KV caching structure organized as a hierarchical router. Instead of evicting or pruning tokens, ARM uses Gumbel-Softmax slot selection and sigmoid-gated updates that softly merge new and stored information, and a learned policy adapts memory usage at inference. The authors report superior performance and efficiency versus fixed KV-caching approaches on commonsense and long-context reasoning benchmarks.
0 engagement1 sourcearxivMon, Sep 21, 2026, 11:11 AM
Developments·Paper
arXiv Survey Maps LLM Attention's Evolution Into Multidimensional Contextual Memory Design
A new arXiv survey analyzes efficient-attention research through a five-dimensional contextual-memory lens covering Memory Representation, Update, Access, Readout, and Integration. It synthesizes 59 release-level records across 14 major model lineages and 11 high-performing open-weight endpoints, arguing architecture design now concerns managing contextual memory rather than the attention operator itself.
A new arXiv survey analyzes efficient-attention research through a five-dimensional contextual-memory lens covering Memory Representation, Update, Access, Readout, and Integration. It synthesizes 59 release-level records across 14 major model lineages and 11 high-performing open-weight endpoints, arguing architecture design now concerns managing contextual memory rather than the attention operator itself.
0 engagement1 sourcearxivWed, Sep 30, 2026, 01:03 PM
Developments·Paper
Metis: First memory foundation model embeds native memory into AI agents
Researchers introduce Metis, the first memory foundation model that embeds native, persistent memory directly into foundation models, moving beyond external memory modules. This could enable AI agents to maintain evolving memory states without separate retrieval systems.
Researchers introduce Metis, the first memory foundation model that embeds native, persistent memory directly into foundation models, moving beyond external memory modules. This could enable AI agents to maintain evolving memory states without separate retrieval systems.
0 engagement1 sourcearxivWed, Jul 29, 2026, 10:58 AM
Developments·Paper
WarpState: a 2B-parameter hybrid LM pairing exact local attention with dual-timescale recurrent associative memory
A researcher posted WarpState, an experimental language-model architecture replacing ever-growing token-level KV caches with fixed-size dual-timescale associative memory per attention head, combined with exact local attention in 128-token chunks. The reference model has ~2.005B parameters; a ~150M version reportedly reached ~3.40 validation cross-entropy, but those results are unreproduced historical reports. The author is soliciting technical criticism and proposes controlled ablations against local-attention-only baselines.
A researcher posted WarpState, an experimental language-model architecture replacing ever-growing token-level KV caches with fixed-size dual-timescale associative memory per attention head, combined with exact local attention in 128-token chunks. The reference model has ~2.005B parameters; a ~150M version reportedly reached ~3.40 validation cross-entropy, but those results are unreproduced historical reports. The author is soliciting technical criticism and proposes controlled ablations against local-attention-only baselines.
3 engagement1 sourceredditSun, Sep 20, 2026, 10:12 AM