A new arXiv paper presents AuditKIT, a source-available evaluation library that records benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality in a single result representation. It targets the practical problem that differing interfaces, metric semantics, and trace formats complicate comparisons and can hide failures in agentic workflows.
An arXiv paper argues agentic benchmark results reflect the joint behavior of models and harnesses, so harnesses should be evaluated through interactions with models. It introduces five complementary measures including Valid Completion Rate, Model-Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load. The authors evaluate four harnesses and four models across six benchmarks and find harness rankings shift with model and benchmark, supporting interaction-level evaluation.
An arXiv paper argues agentic benchmark results reflect the joint behavior of models and harnesses, so harnesses should be evaluated through interactions with models. It introduces five complementary measures including Valid Completion Rate, Model-Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load. The authors evaluate four harnesses and four models across six benchmarks and find harness rankings shift with model and benchmark, supporting interaction-level evaluation.
An arXiv paper introduces behavioral correctness assumptions, a framework for evaluating reference-based automatic evaluation methods in natural language generation. It argues existing meta-evaluation measures agreement with human judgments or benchmark labels and offers limited insight into evaluator behavior under controlled conditions.
An arXiv paper introduces behavioral correctness assumptions, a framework for evaluating reference-based automatic evaluation methods in natural language generation. It argues existing meta-evaluation measures agreement with human judgments or benchmark labels and offers limited insight into evaluator behavior under controlled conditions.
An arXiv paper presents BekchiAI, combining a benchmark for agentic skill with a platform for observing and controlling live LLM agents. It targets skills accuracy-only leaderboards miss: tool sequencing, planning under dependencies, untrusted-input judgment, and argument grounding. For practitioners it offers a way to both measure and supervise tool-using agents in one system.
An arXiv paper presents BekchiAI, combining a benchmark for agentic skill with a platform for observing and controlling live LLM agents. It targets skills accuracy-only leaderboards miss: tool sequencing, planning under dependencies, untrusted-input judgment, and argument grounding. For practitioners it offers a way to both measure and supervise tool-using agents in one system.
Researchers introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. It decouples evaluation into three independent components—Benchmark, Harness, and Environment—to improve reproducibility and reduce redundant engineering.
Researchers introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. It decouples evaluation into three independent components—Benchmark, Harness, and Environment—to improve reproducibility and reduce redundant engineering.
A new paper proposes A2E (Agent Auditing Engine), an end-to-end evaluation engine for agent harnesses, built on a new Agent Task Protocol (ATP). It aims to streamline systematic capability evaluation of LLM-based agents.
A new paper proposes A2E (Agent Auditing Engine), an end-to-end evaluation engine for agent harnesses, built on a new Agent Task Protocol (ATP). It aims to streamline systematic capability evaluation of LLM-based agents.