Researchers introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. It decouples evaluation into three independent components—Benchmark, Harness, and Environment—to improve reproducibility and reduce redundant engineering.
The paper, posted on arXiv on 2026-07-15, addresses fragmentation in current agent evaluation pipelines. AgentCompass allows flexible configurations without requiring extensive code changes, aiming to standardize agent evaluation as LLMs evolve into autonomous agents.
Researchers introduced OmniaBench, a benchmark for evaluating large language models as general agents across diverse application scenarios with explicit state spaces. It addresses the limitation of existing benchmarks that focus on narrow tool ecosystems or interaction formats.
Researchers introduced OmniaBench, a benchmark for evaluating large language models as general agents across diverse application scenarios with explicit state spaces. It addresses the limitation of existing benchmarks that focus on narrow tool ecosystems or interaction formats.
A new arXiv paper presents AuditKIT, a source-available evaluation library that records benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation. It targets the practical problem that differing interfaces, metric semantics, and trace formats complicate comparisons and can hide failures in agentic workflows.
A new arXiv paper presents AuditKIT, a source-available evaluation library that records benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation. It targets the practical problem that differing interfaces, metric semantics, and trace formats complicate comparisons and can hide failures in agentic workflows.
A new paper proposes A2E (Agent Auditing Engine), an end-to-end evaluation engine for agent harnesses, built on a new Agent Task Protocol (ATP). It aims to streamline systematic capability evaluation of LLM-based agents.
A new paper proposes A2E (Agent Auditing Engine), an end-to-end evaluation engine for agent harnesses, built on a new Agent Task Protocol (ATP). It aims to streamline systematic capability evaluation of LLM-based agents.
Researchers propose Copy-on-Write (CoW) Scoring, a framework that evaluates LLM-based agents directly within application environments using a PostgreSQL-level copy-on-write mechanism. This approach aims to overcome limitations of existing benchmarks and replica environments by providing granular, application-specific evaluation without expensive duplication.
Researchers propose Copy-on-Write (CoW) Scoring, a framework that evaluates LLM-based agents directly within application environments using a PostgreSQL-level copy-on-write mechanism. This approach aims to overcome limitations of existing benchmarks and replica environments by providing granular, application-specific evaluation without expensive duplication.
Researchers introduce HarnessDev, a benchmark that shifts agent evaluation from task outputs to runnable infrastructure. It tests whether models can create and improve their own agent harness — the model-external execution infrastructure that increasingly determines agent capability. The work addresses a gap: current evaluations report performance under a fixed harness without probing models' ability to develop one themselves.
Researchers introduce HarnessDev, a benchmark that shifts agent evaluation from task outputs to runnable infrastructure. It tests whether models can create and improve their own agent harness — the model-external execution infrastructure that increasingly determines agent capability. The work addresses a gap: current evaluations report performance under a fixed harness without probing models' ability to develop one themselves.