Releases·Tool Release
RandKV: a pip-installable random KV-cache eviction policy for Transformers
A developer released RandKV on GitHub and PyPI, packaging the Random Attention KV-cache eviction policy as an installable library instead of the paper's full research harness. It targets Transformers users needing KV-cache eviction without heavyweight dependencies.
3 engagement1 sourceSun, Sep 6, 2026, 06:43 PM
AI Overview
- RandKV protects the full prompt, keeps a recent-token buffer, and samples older tokens per KV head, requiring Transformers 5.16 with batch size one, no beam search, no vLLM yet.
- Author's benchmark: Qwen3-0.6B on Apple M4 hit 35.48 tok/s with RandKV versus 38.53 dense (0.921x), showing adapter overhead, not the paper's vLLM gains.
- Implementation derives deterministic eviction seeds from request, round, layer, and head, keeping sequence positions separate from compacted physical cache length.
Related
Developments·Paper
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
A new arXiv paper, 'Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning,' examines KV cache compression for long chain-of-thought reasoning, where the cache becomes a severe memory bottleneck. It argues that existing methods' core assumption — ranking tokens by predicted future importance — rests on a selection signal that contributes almost nothing, implying token importance scoring is largely unnecessary for eviction decisions.
A new arXiv paper, 'Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning,' examines KV cache compression for long chain-of-thought reasoning, where the cache becomes a severe memory bottleneck. It argues that existing methods' core assumption — ranking tokens by predicted future importance — rests on a selection signal that contributes almost nothing, implying token importance scoring is largely unnecessary for eviction decisions.
0 engagement1 sourcearxivThu, Sep 3, 2026, 06:38 AM
Developments·Paper
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
A new paper proposes VarRate, a training-free method that allocates rank (not evicts tokens) for KV cache compression, addressing accuracy collapses in token-selection methods like SnapKV and Ada-KV. VarRate avoids irreversible eviction and uniform rank waste, potentially improving long-context LLM inference efficiency.
A new paper proposes VarRate, a training-free method that allocates rank (not evicts tokens) for KV cache compression, addressing accuracy collapses in token-selection methods like SnapKV and Ada-KV. VarRate avoids irreversible eviction and uniform rank waste, potentially improving long-context LLM inference efficiency.
0 engagement1 sourcearxivThu, Jul 16, 2026, 11:03 PM
Developments·Paper
vToken: Token-Level Virtualization for Reclaimable KV Caches
A new paper introduces vToken, a token-level virtualization layer that decouples logical token liveness from physical block placement in KV caches, addressing intra-block fragmentation that leaves memory unreclaimable. This improves memory efficiency for LLM serving.
A new paper introduces vToken, a token-level virtualization layer that decouples logical token liveness from physical block placement in KV caches, addressing intra-block fragmentation that leaves memory unreclaimable. This improves memory efficiency for LLM serving.
0 engagement1 sourcearxivThu, Aug 13, 2026, 02:01 PM
Developments·Paper
Arxiv paper introduces TwinKV, a training-free KV cache eviction method exploiting pairwise key redundancy
An arxiv paper (2026-08-27) presents TwinKV, a training-free, attention-free KV cache eviction method for long-context inference that targets pairwise key redundancy. It challenges attention-based eviction by showing attention magnitude is uncorrelated with a token's causal contribution. The method aims to cut KV cache memory footprint, especially for small models under tight resource budgets.
An arxiv paper (2026-08-27) presents TwinKV, a training-free, attention-free KV cache eviction method for long-context inference that targets pairwise key redundancy. It challenges attention-based eviction by showing attention magnitude is uncorrelated with a token's causal contribution. The method aims to cut KV cache memory footprint, especially for small models under tight resource budgets.
0 engagement1 sourcearxivThu, Aug 27, 2026, 01:43 PM
Developments·Paper
Paper: Largest-weight KV-cache eviction is near-optimal across five models
A new arXiv paper enumerates the exact best subset of keys for KV-cache eviction across 168,192 attention rows from five models, finding that keeping the largest weights is already near-optimal. The best subset closes only a median 2–5% of the remaining gap to full attention, suggesting published margins between eviction methods may stem from other factors.
A new arXiv paper enumerates the exact best subset of keys for KV-cache eviction across 168,192 attention rows from five models, finding that keeping the largest weights is already near-optimal. The best subset closes only a median 2–5% of the remaining gap to full attention, suggesting published margins between eviction methods may stem from other factors.
0 engagement1 sourcearxivTue, Aug 25, 2026, 11:40 PM