Magnitude (YC S25) launches self-optimizing open-source inference engine for local agents, claiming up to 2x faster than llama.cpp

A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.

Products·concept

Spite: a modular local LLM inference engine where every layer is replaceable

Spite is a customizable inference engine for local LLMs that compiles kernels and optimizations only for your specific GPUs and chosen models. Instead of one engine supporting all hardware, users assemble per-model kernel modules and design optimizations for their own setup. It targets people running local models who want setup-specific performance without touching unrelated layers.

Spite is a customizable inference engine for local LLMs that compiles kernels and optimizations only for your specific GPUs and chosen models. Instead of one engine supporting all hardware, users assemble per-model kernel modules and design optimizations for their own setup. It targets people running local models who want setup-specific performance without touching unrelated layers.

AutoTuneLLM is a tool that automatically optimizes local LLM inference for a user's specific hardware. It reduces time-to-first-token by 39% and agent wall times by 46%, and prevents memory swaps by dynamically managing KV cache, prefix caching, RAM pressure, context trimming, and KV quantization. It solves the problem of local LLMs being slow or unreliable on consumer devices, targeting developers and power users running models locally.

Social·Community

Qwen3.8-Flash-Next in llama.cpp vs SGLang vs FreeToken: 35s vs 258s to first token at full context. My findings on new PRs coming to engines.

A Reddit user benchmarked Qwen3.8-Flash-Next on one workstation across llama.cpp, SGLang, and FreeToken, varying engine, weight format, and memory placement while holding hardware and model family fixed. The tests found large time-to-first-token gaps (35s vs 258s at full context) and evaluated upcoming engine PRs including speculative decoding patches. For practitioners, the results suggest engine choice and PR-level patches substantially affect inference latency on local setups.

A Reddit user benchmarked Qwen3.8-Flash-Next on one workstation across llama.cpp, SGLang, and FreeToken, varying engine, weight format, and memory placement while holding hardware and model family fixed. The tests found large time-to-first-token gaps (35s vs 258s at full context) and evaluated upcoming engine PRs including speculative decoding patches. For practitioners, the results suggest engine choice and PR-level patches substantially affect inference latency on local setups.

Products·launched

PolyServe: open-source LLM serving autotuner benchmarking configs across vLLM, SGLang, and llama.cpp

PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.

PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.