What Makes Open Models Fast in Production — Sujee Maniyam, Nebius

Social·Community

FriendliAI's Yunmo Koo: inference provider choice, caching, and speculative decoding decide same-model speed

Yunmo Koo, founding engineer at FriendliAI, speaking at AI Engineer World's Fair 2026, argues open-weight models like GLM 5.2 now compete with proprietary frontier models, making the inference provider the differentiator. He demos the same model and coding agent on two providers, with FriendliAI completing a build in about two minutes versus more than three elsewhere, and attributes the gap to caching and speculative decoding infrastructure.

Yunmo Koo, founding engineer at FriendliAI, speaking at AI Engineer World's Fair 2026, argues open-weight models like GLM 5.2 now compete with proprietary frontier models, making the inference provider the differentiator. He demos the same model and coding agent on two providers, with FriendliAI completing a build in about two minutes versus more than three elsewhere, and attributes the gap to caching and speculative decoding infrastructure.

Products·launched

PolyServe: open-source LLM serving autotuner benchmarking configs across vLLM, SGLang, and llama.cpp

PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.

PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.

Releases·Tool Release

Dev's vLLM MXFP4 build runs NVFP4 Qwen3.8-27B on AMD R9700 at 5,809 tok/s prefill, 276 tok/s decode

A developer extended their MXFP4 vLLM build on the Radiance image with online NVFP4-to-MXFP4 conversion, letting NVFP4-quantized models use the MXFP4 fast path on AMD R9700 GPUs. Running Unsloth's Qwen3.8-27B NVFP4 quant, they measured 5,809 tok/s prefill and 276 tok/s decode via their BetterBench harness.

A developer extended their MXFP4 vLLM build on the Radiance image with online NVFP4-to-MXFP4 conversion, letting NVFP4-quantized models use the MXFP4 fast path on AMD R9700 GPUs. Running Unsloth's Qwen3.8-27B NVFP4 quant, they measured 5,809 tok/s prefill and 276 tok/s decode via their BetterBench harness.

Social·Community

Red Hat engineers detail KV cache-aware routing and P/D disaggregation for agentic inference on Kubernetes

Red Hat engineers Yuchen Fama and Ashish Kamra presented KV cache-aware routing and prefill/decode disaggregation on Kubernetes, arguing public inference benchmarks miss real agentic traffic. Their traces show cache hit rates above 90% and input-to-output token ratios over 100:1, so client-controlled cache lifecycle changes which levers matter. A live demo showed cache reuse cutting a turn from ~3 seconds to ~1 second.

Red Hat engineers Yuchen Fama and Ashish Kamra presented KV cache-aware routing and prefill/decode disaggregation on Kubernetes, arguing public inference benchmarks miss real agentic traffic. Their traces show cache hit rates above 90% and input-to-output token ratios over 100:1, so client-controlled cache lifecycle changes which levers matter. A live demo showed cache reuse cutting a turn from ~3 seconds to ~1 second.

AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.