Releases·Tool Release
开源 LLM 推理容量规划手册：三层模型、排队论 P95 以及从 GPU 预算中推导 QPS
开发者在GitHub发布开源的《大模型推理案头手册》，填补中文圈LLM推理容量规划的系统化内容空白。手册针对'给定N张GPU卡，一套推理服务能承载多少流量'这一落地核心问题，提供从公式、算例到可照抄SOP的闭环体系。它指出行业常见错误——用闭环压测吞吐当容量（隐含ρ=100%时排队等待趋于无穷），并提供排队论、开环压测到500万DAU采购决策的完整链路，对推理服务部署和成本决策的从业者具有直接实用价值。
43 engagement1 sourceWed, Sep 2, 2026, 02:41 AM
AI Overview
- GitHub 手册（2026-09-02）通过 Roofline 物理限制、'三堵墙'单节点 λ_sat 和用于 P50/P95 延迟的排队论模型来估算 LLM 服务容量。
- 书中指出了常见的容量错误：将闭环基准测试吞吐量视为容量，以及忽略 Prefill 成本 —— 这种方法在 2048/512 工作负载下会高估约 4 倍的 QPS。
- 书中包含了两份已解答的计算示例（70B×H100 和 32B×H20）、一个 ρ-to-P95 查找表，以及涵盖 Vidur 和 LLMServingSim 在内的五条仿真路线的综述。
- 书中针对 CTO 和售前工程师提供了为期七步的采购工作流，涵盖了成本转换和自建与云端的选择，而非介绍 kernel 级调优。
Related
Releases·Tool Release
Free, open-source book 'How to Make Your Model Fast' covers ML performance from silicon to agents
An ML performance engineer released a free, open-source book, 'How to Make Your Model Fast: A Systems View of Efficient Machine Learning, from Silicon to Agents', on GitHub. It teaches roofline analysis and bottleneck reasoning before optimization, covering kernels through serving and agents. The author is soliciting feedback and contributions from ML systems, inference, and edge AI practitioners.
An ML performance engineer released a free, open-source book, 'How to Make Your Model Fast: A Systems View of Efficient Machine Learning, from Silicon to Agents', on GitHub. It teaches roofline analysis and bottleneck reasoning before optimization, covering kernels through serving and agents. The author is soliciting feedback and contributions from ML systems, inference, and edge AI practitioners.
2 engagement1 sourceredditTue, Sep 29, 2026, 10:35 AM
Products·launched
Browser-based LLM deployment calculator: VRAM feasibility, TTFT, and throughput for any model × GPU × parallelism
A static performance model that estimates VRAM feasibility, time-to-first-token, and throughput for serving LLMs across model, GPU, and parallelism configurations. It helps ML engineers plan deployments by answering 'how many GPUs do I need?' analytically, without running benchmarks. Runs entirely in the browser with no backend.
A static performance model that estimates VRAM feasibility, time-to-first-token, and throughput for serving LLMs across model, GPU, and parallelism configurations. It helps ML engineers plan deployments by answering 'how many GPUs do I need?' analytically, without running benchmarks. Runs entirely in the browser with no backend.
5 engagement1 sourceredditFri, Aug 28, 2026, 01:01 PM
Social·Community
Medium explainer walks through local LLM inference with GPUs, bandwidth, and the roofline model
A Medium article explains how local LLM inference performance is governed by GPU compute, memory bandwidth, and the roofline model. It offers practitioners a framework for understanding why inference speed depends more on bandwidth than raw compute in many workloads.
A Medium article explains how local LLM inference performance is governed by GPU compute, memory bandwidth, and the roofline model. It offers practitioners a framework for understanding why inference speed depends more on bandwidth than raw compute in many workloads.
0 engagement1 sourcerssSun, Aug 30, 2026, 08:59 AM
Releases·Tool Release
LLM Inference Handbook released on GitHub, covering inference from first principles to production serving
A GitHub repository, 'LLM Inference Handbook', presents a guide to LLM inference spanning fundamentals to production serving, with the economics of Arabic as a first-class concern. It targets the gap between introductory token-level explainers and expert material that assumes GPU memory hierarchy, distributed systems, and CUDA kernel knowledge. Practitioners gain a single resource intended to prepare them to read vLLM changelogs and OSDI serving papers.
A GitHub repository, 'LLM Inference Handbook', presents a guide to LLM inference spanning fundamentals to production serving, with the economics of Arabic as a first-class concern. It targets the gap between introductory token-level explainers and expert material that assumes GPU memory hierarchy, distributed systems, and CUDA kernel knowledge. Practitioners gain a single resource intended to prepare them to read vLLM changelogs and OSDI serving papers.
27 engagement1 sourcegithubThu, Aug 27, 2026, 11:39 AM
Releases·Tool Release
AWS introduces tiered KV cache for LLMs on SageMaker HyperPod with Curvine
AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.
AWS announced a tiered KV cache feature for large language model inference on SageMaker HyperPod, developed with Curvine. It reduces infrastructure costs and improves time-to-first-token by avoiding recomputation of identical prompts, benefiting teams running models like Qwen, Llama, and DeepSeek.
0 engagement1 sourcerssWed, Aug 12, 2026, 01:42 PM