Releases·Tool Release
Free, open-source book 'How to Make Your Model Fast' covers ML performance from silicon to agents
An ML performance engineer released a free, open-source book, 'How to Make Your Model Fast: A Systems View of Efficient Machine Learning, from Silicon to Agents', on GitHub. It teaches roofline analysis and bottleneck reasoning before optimization, covering kernels through serving and agents. The author is soliciting feedback and contributions from ML systems, inference, and edge AI practitioners.
2 engagement1 sourceTue, Sep 29, 2026, 10:35 AM
AI Overview
- Book 'How to Make Your Model Fast: A Systems View of Efficient Machine Learning' is free and open source on GitHub (usamahz/make-your-model-fast)
- Author argues reducing FLOPs doesn't necessarily speed up a model; engineers must first determine whether the system is compute, bandwidth, memory, or system bound
- Chapters progress from roofline analysis and hardware through kernels, compilers, quantisation, pruning, vision, on-device LLMs, robotics, profiling, serving, and agents
- Teaches five reasoning questions, including how fast a model can possibly run on given hardware and whether quantisation, pruning, or kernel optimisation is worth doing
- Author solicits feedback and contributions from ML systems, inference, compilers, edge AI, and performance engineering practitioners
Related
Releases·Tool Release
Quantprobe open-source framework runs 30B models at 22 tok/s on 6GB GPU
A developer released Quantprobe, an open-source memory allocation optimization framework that lets users run large local LLMs on old hardware. It achieves 30B parameter models at 22 tokens per second on a 6GB GPU with 16GB RAM. The tool aims to maximize performance for specific use cases.
A developer released Quantprobe, an open-source memory allocation optimization framework that lets users run large local LLMs on old hardware. It achieves 30B parameter models at 22 tokens per second on a 6GB GPU with 16GB RAM. The tool aims to maximize performance for specific use cases.
26 engagement1 sourceredditMon, Aug 3, 2026, 08:11 PM
Developments·Paper
Researchers propose layer importance metric balancing speed and quality in small LLM quantization
An arXiv paper introduces a composite metric for selecting per-layer quantization precision in small language models, combining information retention (normalized SQNR) with throughput gains from a roofline-based latency analysis. It targets autoregressive inference, which is memory-bandwidth bound, where uniform quantization often hurts models with limited redundancy. The authors profile Gemma 3 1B to validate the approach.
An arXiv paper introduces a composite metric for selecting per-layer quantization precision in small language models, combining information retention (normalized SQNR) with throughput gains from a roofline-based latency analysis. It targets autoregressive inference, which is memory-bandwidth bound, where uniform quantization often hurts models with limited redundancy. The authors profile Gemma 3 1B to validate the approach.
0 engagement1 sourcearxivThu, Aug 27, 2026, 10:23 AM
Releases·Tool Release
开源 LLM 推理容量规划手册：三层模型、排队论 P95 以及从 GPU 预算中推导 QPS
开发者在GitHub发布开源的《大模型推理案头手册》，填补中文圈LLM推理容量规划的系统化内容空白。手册针对'给定N张GPU卡，一套推理服务能承载多少流量'这一落地核心问题，提供从公式、算例到可照抄SOP的闭环体系。它指出行业常见错误——用闭环压测吞吐当容量（隐含ρ=100%时排队等待趋于无穷），并提供排队论、开环压测到500万DAU采购决策的完整链路，对推理服务部署和成本决策的从业者具有直接实用价值。
开发者在GitHub发布开源的《大模型推理案头手册》，填补中文圈LLM推理容量规划的系统化内容空白。手册针对'给定N张GPU卡，一套推理服务能承载多少流量'这一落地核心问题，提供从公式、算例到可照抄SOP的闭环体系。它指出行业常见错误——用闭环压测吞吐当容量（隐含ρ=100%时排队等待趋于无穷），并提供排队论、开环压测到500万DAU采购决策的完整链路，对推理服务部署和成本决策的从业者具有直接实用价值。
43 engagement1 sourcegithubWed, Sep 2, 2026, 02:41 AM
Social·Community
Community questions legitimacy of eLLM CPU inference project
A Reddit user questions whether eLLM, a project claiming faster CPU inference for LLMs, is legitimate or "vibecoded slop." The user notes the premise of optimizing for CPUs is sound but hasn't read the paper and observes the project has gone quiet.
A Reddit user questions whether eLLM, a project claiming faster CPU inference for LLMs, is legitimate or "vibecoded slop." The user notes the premise of optimizing for CPUs is sound but hasn't read the paper and observes the project has gone quiet.
7 engagement1 sourceredditThu, Aug 13, 2026, 10:31 PM
Products·launched
LLM inference simulator for hardware bottleneck analysis
A free, self-hosted web simulator that visualizes how hardware runs local LLMs, allowing users to adjust parameters and observe bottlenecks. It helps developers and enthusiasts understand concepts like prompt processing and tokens per second, and how hardware choices affect performance.
A free, self-hosted web simulator that visualizes how hardware runs local LLMs, allowing users to adjust parameters and observe bottlenecks. It helps developers and enthusiasts understand concepts like prompt processing and tokens per second, and how hardware choices affect performance.
2 engagement1 sourceredditThu, Jul 23, 2026, 04:36 PM