Releases·Tool Release
Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache
A developer working on cache management in vLLM with dual DGX Spark units and DeepSeek v4 Flash 0731 built a validation tool after finding cache management issues. The tool runs a calibration probe to distinguish cache hits from misses, letting users verify whether advertised KV cache behavior matches reality under pressure.
33 engagement1 sourceSun, Sep 6, 2026, 09:38 AM
AI Overview
- A Reddit user built the open-source cache-pressure tool to measure real KV cache retention against vLLM's advertised capacity on 2x DGX Spark with DeepSeek v4 Flash 0731
- Pre-fix vLLM image retained 1,052,025 tokens (51.98% of 2,023,924 capacity, 27/80 contexts); with dedupe and boundfix patches it retained 3,000,048 tokens (146.56% of 2,047,043 capacity, 77/80 contexts)
- The tool calibrates cache hits via a probe, fills the cache with stable contexts, then validates in reverse order until a miss, and was tested against vLLM, ninfer, llama.cpp, and SGLang
Related
Releases·Tool Release
Developer builds cache-invalidation detection tool for LLM harnesses
A developer created a tool to detect cache invalidation in LLM calls, addressing prefill cost issues in local-first harnesses. The tool helps identify when changes to messages, system prompts, tools, or reasoning_effort trigger cache misses.
A developer created a tool to detect cache invalidation in LLM calls, addressing prefill cost issues in local-first harnesses. The tool helps identify when changes to messages, system prompts, tools, or reasoning_effort trigger cache misses.
223 engagement2 sourcesreddit, githubSat, Jul 18, 2026, 11:34 AM
Developments·Paper
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
QEvict is a new KV cache eviction method that addresses attention drift during decoding by allowing evicted tokens to be recovered. It combines quantized storage with a recoverable eviction policy, reducing memory footprint while maintaining long-context performance. The paper is available on arXiv.
QEvict is a new KV cache eviction method that addresses attention drift during decoding by allowing evicted tokens to be recovered. It combines quantized storage with a recoverable eviction policy, reducing memory footprint while maintaining long-context performance. The paper is available on arXiv.
0 engagement1 sourcearxivWed, Aug 5, 2026, 06:29 PM
Social·Community
DeepSeek-V4.1-Flash's SWA Bounded Replay rebuilds local KV from last N_window tokens, raising accuracy questions
Practitioners are examining DeepSeek-V4.1-Flash's SWA Bounded Replay, a KV cache technique that discards the sliding-window KV and approximately reconstructs it from only the last N_window tokens during prefill, saving large amounts of memory. Discussion centers on whether this approximation degrades long-range recall and session restarts, and one user demonstrated a similar fast-prefill KV approximation running in a web demo.
Practitioners are examining DeepSeek-V4.1-Flash's SWA Bounded Replay, a KV cache technique that discards the sliding-window KV and approximately reconstructs it from only the last N_window tokens during prefill, saving large amounts of memory. Discussion centers on whether this approximation degrades long-range recall and session restarts, and one user demonstrated a similar fast-prefill KV approximation running in a web demo.
Updated Fri, Sep 11, 2026, 03:36 AM — A community web demo replicating the fast-prefill KV approximation on Qwen3 appeared, with a question about scaling it t
596 engagement2 sourcesredditThu, Sep 10, 2026, 11:07 PM
Social·Community
User seeks help tuning llama-server cache on Strix Halo for Qwen 3.5 122B
A user on Reddit reports performance issues with large models (e.g., Qwen 3.5 122B) on a Strix Halo system, where a full cache miss at 100k context causes 10-20 minutes of prompt processing time. They have configured --cache-ram 16384 to increase available VRAM for cache, but seek further tuning advice.
A user on Reddit reports performance issues with large models (e.g., Qwen 3.5 122B) on a Strix Halo system, where a full cache miss at 100k context causes 10-20 minutes of prompt processing time. They have configured --cache-ram 16384 to increase available VRAM for cache, but seek further tuning advice.
28 engagement1 sourceredditSun, Jul 12, 2026, 07:16 AM
Developments·Paper
Arxiv paper introduces TwinKV, a training-free KV cache eviction method exploiting pairwise key redundancy
An arxiv paper (2026-08-27) presents TwinKV, a training-free, attention-free KV cache eviction method for long-context inference that targets pairwise key redundancy. It challenges attention-based eviction by showing attention magnitude is uncorrelated with a token's causal contribution. The method aims to cut KV cache memory footprint, especially for small models under tight resource budgets.
An arxiv paper (2026-08-27) presents TwinKV, a training-free, attention-free KV cache eviction method for long-context inference that targets pairwise key redundancy. It challenges attention-based eviction by showing attention magnitude is uncorrelated with a token's causal contribution. The method aims to cut KV cache memory footprint, especially for small models under tight resource budgets.
0 engagement1 sourcearxivThu, Aug 27, 2026, 01:43 PM