Releases·Tool Release
Keeping vLLM's Prefix Cache Warm Between Agent Turns
Keeping vLLM's Prefix Cache Warm Between Agent Turns
7 engagement1 sourceThu, Sep 17, 2026, 12:53 AM
Entities
Related
Social·Community
Best practice for batch vLLM API calls with shared prefix through KV-cache-aware llm-d router?
A practitioner running a 10-call agent workflow through vLLM asks how to best process batches of prompts sharing a common prefix. Their setup routes traffic via an llm-d/vllm-router with KV-cache-aware routing into a pool of vLLM workers. They propose sending one call first to prime the cache block before dispatching the rest, and ask whether this is a sound strategy.
A practitioner running a 10-call agent workflow through vLLM asks how to best process batches of prompts sharing a common prefix. Their setup routes traffic via an llm-d/vllm-router with KV-cache-aware routing into a pool of vLLM workers. They propose sending one call first to prime the cache block before dispatching the rest, and ask whether this is a sound strategy.
8 engagement1 sourceredditWed, Sep 30, 2026, 07:12 AM
Releases·Tool Release
A simple fix for LLM tail latency
A post on Hacker News links to an engineering article proposing a simple fix for LLM tail latency. The fix aims to reduce variability in inference response times, which is critical for real-time applications. Practitioners can apply this technique to improve consistency in LLM serving.
A post on Hacker News links to an engineering article proposing a simple fix for LLM tail latency. The fix aims to reduce variability in inference response times, which is critical for real-time applications. Practitioners can apply this technique to improve consistency in LLM serving.
4 engagement1 sourcehackernewsFri, Aug 14, 2026, 05:58 AM
Social·Community
Show HN: Eliminating text looping/latent collapse in LLM activation steering
Show HN: Eliminating text looping/latent collapse in LLM activation steering
Show HN: Eliminating text looping/latent collapse in LLM activation steering
1 engagement1 sourcehackernewsTue, Sep 8, 2026, 07:57 PM
Releases·Tool Release
Developer builds cache-invalidation detection tool for LLM harnesses
A developer created a tool to detect cache invalidation in LLM calls, addressing prefill cost issues in local-first harnesses. The tool helps identify when changes to messages, system prompts, tools, or reasoning_effort trigger cache misses.
A developer created a tool to detect cache invalidation in LLM calls, addressing prefill cost issues in local-first harnesses. The tool helps identify when changes to messages, system prompts, tools, or reasoning_effort trigger cache misses.
223 engagement2 sourcesreddit, githubSat, Jul 18, 2026, 11:34 AM
Products·launched
Coalent: LLM answer cache that invalidates when source docs change
Coalent is an LLM answer cache that automatically invalidates cached responses when the underlying source documents change. It solves the problem of stale or outdated LLM-generated answers by tying cache validity to document updates, useful for developers building document-grounded LLM applications.
Coalent is an LLM answer cache that automatically invalidates cached responses when the underlying source documents change. It solves the problem of stale or outdated LLM-generated answers by tying cache validity to document updates, useful for developers building document-grounded LLM applications.
1 engagement1 sourcehackernewsTue, Aug 18, 2026, 02:16 PM