A developer benchmarked sparse attention in their AGPL-3.0 inference engine on an RK3588 board running Qwen3-VL-2B and found decode-side top-k KV sparse attention speeds up long-context decoding (up to 2.92× at 16K) but hurts at short context, where prefill-side block skipping made requests 18% slower. They added per-axis length thresholds (--sparse-min-ctx 1024, --sparse-pf-min-ctx 3072) so sparse paths only fire past break-even lengths. The findings warn that sparse attention is not free at short context and buys attention speed, not memory.
Piotr Nawrot, a researcher in efficient attention and KV cache compression, published a thread on X detailing common evaluation pitfalls that make sparse attention and KV compression methods look better than they are. The posts share his critique, which highlights practices like using single-hop retrieval tasks without distractors and useless context to inflate results. This matters to practitioners as it underscores the need for rigorous benchmarking when evaluating such methods.
Piotr Nawrot, a researcher in efficient attention and KV cache compression, published a thread on X detailing common evaluation pitfalls that make sparse attention and KV compression methods look better than they are. The posts share his critique, which highlights practices like using single-hop retrieval tasks without distractors and useless context to inflate results. This matters to practitioners as it underscores the need for rigorous benchmarking when evaluating such methods.
A new arXiv paper, CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing), targets the quadratic cost of attention prefilling in long-context LLM inference. It adaptively routes attention heads to sparse patterns using a structural-mass-motivated approach, addressing limits of both fixed-pattern sparse attention and prior dynamic routing methods.
A new arXiv paper, CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing), targets the quadratic cost of attention prefilling in long-context LLM inference. It adaptively routes attention heads to sparse patterns using a structural-mass-motivated approach, addressing limits of both fixed-pattern sparse attention and prior dynamic routing methods.
An arxiv paper introduces 'opinion leader dynamics,' a theoretical framework modeling tokens as particles on the unit sphere to explain how sparse attention shapes token representation evolution. It identifies two mechanisms producing internal convergence with distinct limiting directions, and validates predictions against frontier sparse-attention LLMs. The work gives practitioners a dynamical account of why sparse attention retains performance at lower cost.
An arxiv paper introduces 'opinion leader dynamics,' a theoretical framework modeling tokens as particles on the unit sphere to explain how sparse attention shapes token representation evolution. It identifies two mechanisms producing internal convergence with distinct limiting directions, and validates predictions against frontier sparse-attention LLMs. The work gives practitioners a dynamical account of why sparse attention retains performance at lower cost.
A new arXiv paper enumerates the exact best subset of keys for KV-cache eviction across 168,192 attention rows from five models, finding that keeping the largest weights is already near-optimal. The best subset closes only a median 2–5% of the remaining gap to full attention, suggesting published margins between eviction methods may stem from other factors.
A new arXiv paper enumerates the exact best subset of keys for KV-cache eviction across 168,192 attention rows from five models, finding that keeping the largest weights is already near-optimal. The best subset closes only a median 2–5% of the remaining gap to full attention, suggesting published margins between eviction methods may stem from other factors.