Releases·Tool Release
Artificial Analysis releases agentperf-local, an open-source benchmark replaying real agent trajectories against local LLM servers
Artificial Analysis released agentperf-local, an open-source tool that benchmarks how fast a machine serves an AI agent by replaying recorded agent trajectories against an OpenAI-compatible model server. It reports throughput and latency while carrying full conversation context per request, measuring speed without quality. It matters to practitioners running local inference who want agent-representative performance numbers rather than generic throughput benchmarks.
58 engagement1 sourceSat, Sep 26, 2026, 03:11 AM
AI Overview
- Artificial Analysis's open-source agentperf-local replays recorded agent conversations against OpenAI-compatible servers (llama.cpp, vLLM, SGLang, LM Studio, Ollama), reporting throughput and latency
- Each replayed request carries the full conversation so far, matching real agent context growth; the tool measures speed only, not output quality
- agentperf-local can benchmark an existing server or download a pinned model and start it; bundled recipes use llama.cpp, vLLM/SGLang on NVIDIA GPUs, or Splash on Apple Silicon
- The tool supports macOS, Linux, and Windows via uv (Python 3.12); managed Windows runs require an NVIDIA GPU and llama.cpp, while vLLM and SGLang run only on Linux
Related
Developments·Paper
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
A research paper argues that local inference serving stacks (Ollama, llama.cpp, vLLM, SGLang) materially change measured tool-call fidelity, independent of model behavior. Ollama's default tools= handling gates requests per model via a static template flag, and harnesses that discard rejection metadata can misclassify failures as model non-calls. The authors propose treating serving behavior as part of the evaluation protocol, with a checklist.
A research paper argues that local inference serving stacks (Ollama, llama.cpp, vLLM, SGLang) materially change measured tool-call fidelity, independent of model behavior. Ollama's default tools= handling gates requests per model via a static template flag, and harnesses that discard rejection metadata can misclassify failures as model non-calls. The authors propose treating serving behavior as part of the evaluation protocol, with a checklist.
0 engagement1 sourcearxivTue, Sep 22, 2026, 04:48 PM
Developments·Benchmark
AA-AgentPerf: Artificial Analysis ranks NVIDIA Vera Rubin top on inference energy efficiency
Artificial Analysis's AA-AgentPerf benchmark replays real coding-agent trajectories — long contexts, tool-call pauses, heavy KV-cache reuse — to evaluate inference hardware. It plots total server throughput per megawatt of GPU-only power against p25 per-request output speed. Across the four tested models, NVIDIA's Vera Rubin sits at the frontier on energy efficiency at nearly every speed.
Artificial Analysis's AA-AgentPerf benchmark replays real coding-agent trajectories — long contexts, tool-call pauses, heavy KV-cache reuse — to evaluate inference hardware. It plots total server throughput per megawatt of GPU-only power against p25 per-request output speed. Across the four tested models, NVIDIA's Vera Rubin sits at the frontier on energy efficiency at nearly every speed.
3 engagement1 sourceredditFri, Oct 16, 2026, 08:40 AM
Products·launched
PolyServe: open-source LLM serving autotuner benchmarking configs across vLLM, SGLang, and llama.cpp
PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.
PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.
23 engagement1 sourceredditThu, Sep 17, 2026, 06:51 AM
Releases·Tool Release
We built an open-source GPU profiler you point an AI agent at, instead of reading traces yourself
Graphsignal released an open-source sidecar GPU profiler designed to be read by AI agents rather than humans. It wraps inference servers, outputs one JSON endpoint of kernel, memory, power, and engine metrics, and lets an agent run the profile-and-tune loop autonomously.
Graphsignal released an open-source sidecar GPU profiler designed to be read by AI agents rather than humans. It wraps inference servers, outputs one JSON endpoint of kernel, memory, power, and engine metrics, and lets an agent run the profile-and-tune loop autonomously.
40 engagement1 sourceredditThu, Sep 17, 2026, 12:27 PM
Releases·Tool Release
Magnitude (YC S25) launches self-optimizing open-source inference engine for local agents, claiming up to 2x faster than llama.cpp
Anders and Tom, two YC S25 founders, launched Magnitude, an Apache 2.0 inference engine written in Rust that tunes kernels on the user's device to run local agents faster. It benchmarks up to 92% faster decode than llama.cpp while using less memory, and ships as a desktop app that serves agents like Codex and OpenCode on demand. It targets the gap between batch-optimized datacenter engines and compatibility-first local engines.
Anders and Tom, two YC S25 founders, launched Magnitude, an Apache 2.0 inference engine written in Rust that tunes kernels on the user's device to run local agents faster. It benchmarks up to 92% faster decode than llama.cpp while using less memory, and ships as a desktop app that serves agents like Codex and OpenCode on demand. It targets the gap between batch-optimized datacenter engines and compatibility-first local engines.
275 engagement1 sourcehackernewsWed, Sep 30, 2026, 05:37 PM