Releases·Tool Release
Google engineers on unreliable LLM benchmarks and Inference Perf, their CNCF load generator
Google engineers Ashok Chandrasekar and Jason Kramberger describe why published LLM inference benchmarks failed to reproduce: harnesses silently under-deliver load, inflate latency, and process data inconsistently. They present Inference Perf, a CNCF multiprocess load generator from the Kubernetes serving working group, plus the Prism UI from the llm-d project.
12 engagement1 sourceSat, Sep 19, 2026, 04:00 PM
AI Overview
- Google's Chandrasekar and Kramberger found harnesses silently underdelivering (200 QPS requested, 38 delivered), traced pitfalls to Python's GIL and thrashing clients, and built Inference Perf, a CNCF multiprocess load generator that sustained 5,000 QPS with a shared workload catalog
- A thrashing client harness inflated its own measured latency by 58 seconds while the server ran fine
- A shared result claiming 20 percent better throughput had temperature set to zero, faster than realistic 0.7 workloads
- Inference Perf schedules requests via Poisson, constant rate, or fixed concurrency plans across worker processes reporting actual vs intended firing times
- Kramberger demoed Prism, a llm-d UI comparing optimizations against plain Kubernetes across eight TPU replicas
Related
Social·Community
Users report benchmark token rates don't match real-world llama.cpp and exllama usage
A user testing llama.cpp and exllama reports that benchmark token rates (50+ tks) consistently drop to around 40 tks in real-world chat or harness use, a 10-token drop across both backends. The discrepancy raises questions about benchmark validity for practical inference performance.
A user testing llama.cpp and exllama reports that benchmark token rates (50+ tks) consistently drop to around 40 tks in real-world chat or harness use, a 10-token drop across both backends. The discrepancy raises questions about benchmark validity for practical inference performance.
8 engagement2 sourcesredditTue, Aug 18, 2026, 04:03 PM
Releases·Tool Release
ShardFlow hits 28 TPS on Qwen2.5-7B across two cloud regions over public WAN
A developer built ShardFlow, a distributed LLM inference framework that splits HuggingFace transformers across GPU machines and uses neural speculative decoding to mitigate WAN latency. In a benchmark, it achieved 28 tokens per second on Qwen2.5-7B across two GCP regions connected via a public internet relay, turning per-token latency into per-round cost.
A developer built ShardFlow, a distributed LLM inference framework that splits HuggingFace transformers across GPU machines and uses neural speculative decoding to mitigate WAN latency. In a benchmark, it achieved 28 tokens per second on Qwen2.5-7B across two GCP regions connected via a public internet relay, turning per-token latency into per-round cost.
5 engagement1 sourceredditSun, Aug 23, 2026, 12:30 PM
Developments·Paper
Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
A preregistered arXiv study audits the assumption that requests sent to the same model name behave consistently over time, treating LLM judges as measurement instruments. Two preregistered campaigns with pre-fixed thresholds both failed to validate their instrument before any measurement, questioning the reliability of LLM-as-judge pipelines used for training data gating, generation scoring, and leaderboards.
A preregistered arXiv study audits the assumption that requests sent to the same model name behave consistently over time, treating LLM judges as measurement instruments. Two preregistered campaigns with pre-fixed thresholds both failed to validate their instrument before any measurement, questioning the reliability of LLM-as-judge pipelines used for training data gating, generation scoring, and leaderboards.
0 engagement1 sourcearxivThu, Sep 3, 2026, 05:59 PM
Products·launched
PolyServe: open-source LLM serving autotuner benchmarking configs across vLLM, SGLang, and llama.cpp
PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.
PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.
23 engagement1 sourceredditThu, Sep 17, 2026, 06:51 AM
Releases·Tool Release
Pydantic AI structured outputs and evals · coles.codes
A developer documented how LLM inference stacks are not batch-invariant, causing different outputs even at temperature zero. They propose running repeated evaluations to compute a pass rate instead of relying on a single flaky result, combined with structured outputs and a calibrated judge for testing.
A developer documented how LLM inference stacks are not batch-invariant, causing different outputs even at temperature zero. They propose running repeated evaluations to compute a pass rate instead of relying on a single flaky result, combined with structured outputs and a calibrated judge for testing.
1 engagement1 sourceredditTue, Jul 14, 2026, 10:16 AM