Releases·Tool Release
Benchmark your custom Pi tools
A developer shared RoastMyHarness, a small open-source engine that benchmarks custom Pi harness setups against bare Pi on DeepSWE coding tasks. It runs the same tasks with a control and a modified variant (extension, skill, AGENTS.md), showing what improved, broke, or cost extra tokens. The tool helps the Pi community validate whether their harness modifications actually help.
3 engagement1 sourceSun, Sep 13, 2026, 01:40 AM
AI Overview
- Reddit user released RoastMyHarness, an open-source engine that runs DeepSWE benchmark tasks comparing bare Pi against custom harnesses, extensions, skills, or AGENTS.md files.
- The tool includes a Pi extension that a wizard uses for setup, tested only on Linux, and its creator notes base Pi is hard to beat on task quality and token efficiency.
Entities
Related
Social·Community
I forced an early fresh-worker replacement in Terminal-Bench. 10/12 still passed.
An OSS agent harness called Threshold was tested with forced fresh-worker replacement at roughly 60k cumulative tokens on 12 Terminal-Bench tasks. Three configurations — native Pi, Threshold continuation, and Threshold plus replacement — each passed 10/12. Traces, graders, failed runs, and an audit are public.
An OSS agent harness called Threshold was tested with forced fresh-worker replacement at roughly 60k cumulative tokens on 12 Terminal-Bench tasks. Three configurations — native Pi, Threshold continuation, and Threshold plus replacement — each passed 10/12. Traces, graders, failed runs, and an audit are public.
1 engagement1 sourceredditWed, Sep 23, 2026, 07:22 PM
Products·launched
Open-source testbed for comparing MCP servers and agent skills
An open-source interactive environment to run the same prompt under different configurations (with/without MCP servers, agent skills, or other tools) and compare results side by side. It helps developers evaluate whether an MCP server or skill actually improves agent output, rather than relying on isolated demos.
An open-source interactive environment to run the same prompt under different configurations (with/without MCP servers, agent skills, or other tools) and compare results side by side. It helps developers evaluate whether an MCP server or skill actually improves agent output, rather than relying on isolated demos.
16 engagement1 sourceredditTue, Jul 28, 2026, 04:57 PM
Developments·Benchmark
We found defects in 37 of DeepSWE's 113 tasks
An analysis on scrimdata.com reports defects in 37 of DeepSWE's 113 tasks. The findings question the reliability of DeepSWE as an evaluation benchmark for software-engineering agents.
An analysis on scrimdata.com reports defects in 37 of DeepSWE's 113 tasks. The findings question the reliability of DeepSWE as an evaluation benchmark for software-engineering agents.
7 engagement1 sourcehackernewsSat, Sep 19, 2026, 12:56 PM
Social·Community
Users share pi-agent setup tips for local models
A developer building their own harness is setting up pi.dev with local models qwen 3.6 and gpt-oss, and asks the community for extensions, skills, and configuration advice to improve multi-step task performance. The thread reflects growing community use of pi-agent with local models.
A developer building their own harness is setting up pi.dev with local models qwen 3.6 and gpt-oss, and asks the community for extensions, skills, and configuration advice to improve multi-step task performance. The thread reflects growing community use of pi-agent with local models.
12 engagement1 sourceredditFri, Aug 14, 2026, 08:58 AM
Products·launched
Oqoqo: Build realistic evals and custom benchmarks for agent tasks
Oqoqo is a platform for creating realistic evals and custom benchmarks for real-world tasks, measuring how agent-friendly product surfaces are against various AI coding agents. It helps developers regression test MCP, CLI, skills, SDK, and other agent-facing interfaces, and allows creating and sharing custom benchmarks.
Oqoqo is a platform for creating realistic evals and custom benchmarks for real-world tasks, measuring how agent-friendly product surfaces are against various AI coding agents. It helps developers regression test MCP, CLI, skills, SDK, and other agent-facing interfaces, and allows creating and sharing custom benchmarks.
1 engagement1 sourcehackernewsMon, Aug 10, 2026, 09:27 PM