Releases·Tool Release
Distributed Local Agents Benchmark
A developer launched airbench.ai, a website where anyone can run and share benchmark tests comparing agent harnesses against local models across different hardware. The goal is a fully distributed agent benchmark, since exhaustively testing every harness-model-hardware combination exceeds any single person's capacity. It includes a leaderboard the creator hopes others will populate with their own tests.
1 engagement1 sourceTue, Sep 29, 2026, 11:48 AM
AI Overview
- A Redditor launched airbench.ai, a community site for testing AI coding harnesses with local models across hardware and sharing results
- The site includes a leaderboard seeded with the creator's own tests, which they hope others will populate
Related
Releases·Tool Release
Show HN: Compute:Arena – Community submitted local AI benchmarks
The Compute:Arena team open-sourced their internal testing harness, letting anyone benchmark open source models on their own hardware and submit results to a public leaderboard. The site is live with a few hundred submissions already, covering combinations of open models, quantizations, and chips from AMD, NVIDIA, Apple Silicon, Intel, and Qualcomm. It gives practitioners a practical way to check real-world model-on-hardware performance.
The Compute:Arena team open-sourced their internal testing harness, letting anyone benchmark open source models on their own hardware and submit results to a public leaderboard. The site is live with a few hundred submissions already, covering combinations of open models, quantizations, and chips from AMD, NVIDIA, Apple Silicon, Intel, and Qualcomm. It gives practitioners a practical way to check real-world model-on-hardware performance.
7 engagement1 sourcehackernewsThu, Sep 17, 2026, 06:54 AM
Social·Community
Harness x model combination: more data! (airbench.ai crowdsources best configs)
A Reddit user launched airbench.ai, a site where people test and share their best local LLM harness-model-hardware combinations. Crowdsourced data already surfaced a winning config: a stranger beat the creator's 5090 baseline using a 3090. The site added a contributor section and reveals which harness performs best per model.
A Reddit user launched airbench.ai, a site where people test and share their best local LLM harness-model-hardware combinations. Crowdsourced data already surfaced a winning config: a stranger beat the creator's 5090 baseline using a 3090. The site added a contributor section and reveals which harness performs best per model.
1 engagement1 sourceredditThu, Oct 8, 2026, 11:52 AM
Social·Community
Are we missing a benchmark for agent runtimes, not just models?
A practitioner argues current benchmarks like SWE-bench and OSWorld evaluate models while deployments increasingly depend on the surrounding agent runtime (OpenAI Agents, Anthropic's stack, AWS AgentCore, LangGraph). They propose a benchmark measuring task success rate, cost, wall-clock time, tool-call retries, and reliability, with controls isolating harness versus model contributions.
A practitioner argues current benchmarks like SWE-bench and OSWorld evaluate models while deployments increasingly depend on the surrounding agent runtime (OpenAI Agents, Anthropic's stack, AWS AgentCore, LangGraph). They propose a benchmark measuring task success rate, cost, wall-clock time, tool-call retries, and reliability, with controls isolating harness versus model contributions.
24 engagement1 sourceredditFri, Sep 11, 2026, 07:28 AM
Products·building
Harness Arena - blind benchmark for AI agent harnesses
An open-source benchmark that compares agent harnesses like Claude Code and Codex fairly. Multiple harnesses run the same task in isolated workspaces, outputs are anonymized, and humans score them blind. It solves the problem of existing comparisons changing model, prompt, tools, and environment simultaneously.
An open-source benchmark that compares agent harnesses like Claude Code and Codex fairly. Multiple harnesses run the same task in isolated workspaces, outputs are anonymized, and humans score them blind. It solves the problem of existing comparisons changing model, prompt, tools, and environment simultaneously.
2 engagement1 sourceredditWed, Sep 2, 2026, 10:18 AM
Social·Community
Community call to build a harness-only benchmark
A Reddit user proposed a community-driven benchmark focused solely on harness performance, aiming to create a leaderboard across diverse real-world tasks grouped by underlying models and reasoning efforts. The project would be open to contributions and decisions made by a group rather than a single person.
A Reddit user proposed a community-driven benchmark focused solely on harness performance, aiming to create a leaderboard across diverse real-world tasks grouped by underlying models and reasoning efforts. The project would be open to contributions and decisions made by a group rather than a single person.
77 engagement2 sourcesredditWed, Aug 5, 2026, 10:54 AM