Optimizing cost and latency with Amazon Bedrock prompt caching

Social·Community

Ollama KV cache survives only on shared prefix: a leading timestamp costs 3.5x prefill (64.6ms vs 18.6ms on qwen2.5:7b)

A developer analyzing Ollama 0.33.3+ found that its KV prompt cache survives only on an exact shared prefix from the first token, so a changing element like a timestamp early in a system prompt forces full re-prefill of everything after it. On qwen2.5:7b with a 324-token prompt, moving a 21-token timestamp from the start to the end cut prefill from 64.6 ms to 18.6 ms. The post also warns that Ollama 0.33.3 changed prompt_eval_duration to time only uncached tokens, making naive tok/s formulas over-report by the cache hit ratio.

A developer analyzing Ollama 0.33.3+ found that its KV prompt cache survives only on an exact shared prefix from the first token, so a changing element like a timestamp early in a system prompt forces full re-prefill of everything after it. On qwen2.5:7b with a 324-token prompt, moving a 21-token timestamp from the start to the end cut prefill from 64.6 ms to 18.6 ms. The post also warns that Ollama 0.33.3 changed prompt_eval_duration to time only uncached tokens, making naive tok/s formulas over-report by the cache hit ratio.

A Reddit user running six agents for a publication reports that with prompt caching enabled, longer, stable prompts achieve 97-99% cache hit rates, making them cheaper than frequently changing short prompts. This challenges the common advice to minimize prompt length for cost savings.

An AWS Machine Learning blog post lays out an eight-step 'customization spectrum' on Amazon Bedrock, ranging from prompt engineering and RAG to fine-tuning, continued pre-training, and fully custom models via Amazon Nova Forge. It gives teams escalation criteria, data requirements, and cost tradeoffs so they avoid both premature fine-tuning and stalled prompt engineering. Most workloads, per AWS, never need to go past grounding with RAG.

A prompt-structuring technique can reduce multi-turn API costs by 5-10x by placing static content (system instructions, tool definitions, few-shot examples) at the front and dynamic content (latest user message, fresh context) at the end. This exploits prefix-based caching, which only works if the beginning of the prompt remains unchanged. The post argues this is a pure ordering optimization, not a model or retrieval change.

Social·Community

Developer guide explains how to reliably trigger Anthropic and OpenAI prompt caching

A developer guide circulating on Reddit explains why many teams fail to get cache hits on Anthropic and OpenAI prompt caching despite its potential to cut API costs by up to 90% and reduce time-to-first-token latency. The post attributes misses to subtle structural mistakes in API payloads and lays out strict payload-structure rules to guarantee high hit rates. For practitioners, the takeaway is that caching savings depend less on the providers' features and more on disciplined prompt construction.

A developer guide circulating on Reddit explains why many teams fail to get cache hits on Anthropic and OpenAI prompt caching despite its potential to cut API costs by up to 90% and reduce time-to-first-token latency. The post attributes misses to subtle structural mistakes in API payloads and lays out strict payload-structure rules to guarantee high hit rates. For practitioners, the takeaway is that caching savings depend less on the providers' features and more on disciplined prompt construction.