Releases·Tool Release
Dev open-sources PagedServe, a Python paged KV cache LLM serving runtime, and asks contributors to stress-test it
A developer released PagedServe, an open-source Python LLM serving runtime built to expose serving internals typically hidden inside vLLM. It implements continuous batching, prefix caching, blockwise paged attention, and an experimental Apple Metal kernel. The author is soliciting contributors and issue reports.
5 engagement1 sourceWed, Sep 23, 2026, 08:13 PM
AI Overview
- PagedServe is an open-source Python LLM serving runtime with continuous batching, prefix caching, blockwise paged attention, and an experimental Apple Metal kernel
- Blockwise paged attention cuts temporary attention memory from 12 MB to 96 KB at 2048 tokens, while the Metal kernel path remains slower than baseline
- Creator Sarthak Verma seeks contributors for benchmarks, Metal profiling, model compatibility, docs, and architecture review, promising to review every real issue and PR
Entities
Related
Developments·Paper
PATTON: Enabling Commodity PIM for Production LLM Serving
A new arXiv paper introduces PATTON, a Processing-in-Memory runtime that lets production LLM serving engines use commodity PIM hardware. It manages the full KV cache lifecycle (allocation, sharing, caching, reclamation) rather than accelerating attention alone, achieving an average 1.95x speedup and 4.83x better energy efficiency over baselines without modifying PIM processing units.
A new arXiv paper introduces PATTON, a Processing-in-Memory runtime that lets production LLM serving engines use commodity PIM hardware. It manages the full KV cache lifecycle (allocation, sharing, caching, reclamation) rather than accelerating attention alone, achieving an average 1.95x speedup and 4.83x better energy efficiency over baselines without modifying PIM processing units.
0 engagement1 sourcearxivThu, Sep 10, 2026, 11:28 AM
Social·Community
Developer seeks feedback on attention mechanism without KV cache
A developer shared a custom attention mechanism that eliminates the key-value cache, claiming it works in small-scale tests. They are requesting community feedback and are open to running comparison benchmarks.
A developer shared a custom attention mechanism that eliminates the key-value cache, claiming it works in small-scale tests. They are requesting community feedback and are open to running comparison benchmarks.
4 engagement1 sourceredditMon, Jul 20, 2026, 12:29 AM
Releases·Tool Release
Proxima vLLM plugin implements Star-KV, claims 4x throughput
A developer released Proxima, a vLLM plugin implementing the Star-KV paper, claiming it serves 4x more requests without hardware changes. The plugin includes a decode kernel faster than FlashAttention-2 at higher batch sizes, potentially improving inference efficiency.
A developer released Proxima, a vLLM plugin implementing the Star-KV paper, claiming it serves 4x more requests without hardware changes. The plugin includes a decode kernel faster than FlashAttention-2 at higher batch sizes, potentially improving inference efficiency.
4 engagement1 sourcehackernewsTue, Aug 11, 2026, 03:04 PM
Social·Community
FP8 KV cache gave +40–52% throughput — but ONLY when vLLM was already preempting. Raw results + the quality gate that failed.
FP8 KV cache gave +40–52% throughput — but ONLY when vLLM was already preempting
FP8 KV cache gave +40–52% throughput — but ONLY when vLLM was already preempting
0 engagement1 sourceredditSat, Sep 19, 2026, 11:01 AM
Releases·Tool Release
runNburn: Rust GGUF runtime targets models larger than RAM
A developer announced runNburn, an open-source Rust inference runtime for quantized GGUF models that runs on CPU, CUDA, Metal, and Android, with experimental Vulkan and OpenCL paths. The runtime is designed to handle models that exceed available fast memory, addressing a gap in existing solutions.
A developer announced runNburn, an open-source Rust inference runtime for quantized GGUF models that runs on CPU, CUDA, Metal, and Android, with experimental Vulkan and OpenCL paths. The runtime is designed to handle models that exceed available fast memory, addressing a gap in existing solutions.
0 engagement1 sourceredditFri, Jul 31, 2026, 03:42 AM