Releases·Tool Release
Agent Evaluation Metric (AEM): decomposable turn-level correctness scoring for multi-turn agents
A new blog post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level metric for evaluating multi-turn agents. It computes correctness from truthfulness and completeness sub-metrics per turn, attributes failures to root causes versus cascading effects, and composes scores into a single indicator. The approach aims to fix the opacity of holistic task-level scoring, where a single wrong tool call silently corrupts all downstream turns.
0 engagement1 sourceThu, Sep 10, 2026, 03:55 PM
AI Overview
- AEM decomposes correctness into per-turn completeness and truthfulness sub-metrics, with a six-reason failure taxonomy separating root causes from cascades via prior_action_failed.
- AEM score is the unweighted mean of passing turns, with pluggable composition (weighted means, gating rules) and semantic similarity scoring (embedding or LLM-as-judge, 0.5 default threshold).
- AEM supports order-invariant tool-call evaluation and tracks action chain length, since longer chains concentrate degradation.
- A five-turn sales-report example shows one wrong parameter ("profit" for "revenue") in turn 2 cascading into turns 3-5, with only turn 2 flagged as root cause.
Related
Social·Community
Community discusses why agent evals miss bad tool calls in the middle
A Reddit post argues that current agent evals grade only the final answer, so incorrect tool calls in the middle of a run go unpunished. This can let broken agent paths pass evaluation and fail when inputs change. The discussion highlights a gap in agent evaluation methodology.
A Reddit post argues that current agent evals grade only the final answer, so incorrect tool calls in the middle of a run go unpunished. This can let broken agent paths pass evaluation and fail when inputs change. The discussion highlights a gap in agent evaluation methodology.
12 engagement1 sourceredditTue, Aug 18, 2026, 01:44 PM
News·Incident
Engineer finds per-answer context adherence catches LLM prod failures that aggregate metrics miss
A retrieval config change slipped through CI, causing fabricated sources in LLM outputs that aggregate task-success metrics failed to flag. The engineer reports that per-answer context adherence scores caught the issue, while aggregate metrics remained flat. This highlights the need for granular evaluation metrics in production LLM systems.
A retrieval config change slipped through CI, causing fabricated sources in LLM outputs that aggregate task-success metrics failed to flag. The engineer reports that per-answer context adherence scores caught the issue, while aggregate metrics remained flat. This highlights the need for granular evaluation metrics in production LLM systems.
3 engagement1 sourceredditThu, Jul 23, 2026, 01:41 PM
Social·Community
Developer builds open-source analytics separating agent execution success from task success
Developer shares open-source analytics separating agent execution success from task success
Developer shares open-source analytics separating agent execution success from task success
3 engagement1 sourceredditMon, Sep 21, 2026, 11:00 AM
Social·Community
New guide breaks LLM benchmarking into 7 production dimensions beyond raw benchmark scores
Author publishes practical framework for benchmarking LLMs before production
Author publishes practical framework for benchmarking LLMs before production
3 engagement1 sourceredditTue, Sep 22, 2026, 04:26 PM
Developments·Paper
When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
An arXiv paper evaluates whether improvements under gold-history next-turn evaluation predict autonomous workflow execution in agent models. Across Qwen3 (4B, 14B) and Gemma 3 (4B, 12B) models on multi-turn customer-support workflows, SFT improves next-turn success but does not transfer to holistic workflow completion. The authors argue next-turn evaluation is not a reliable proxy for workflow success.
An arXiv paper evaluates whether improvements under gold-history next-turn evaluation predict autonomous workflow execution in agent models. Across Qwen3 (4B, 14B) and Gemma 3 (4B, 12B) models on multi-turn customer-support workflows, SFT improves next-turn success but does not transfer to holistic workflow completion. The authors argue next-turn evaluation is not a reliable proxy for workflow success.
0 engagement1 sourcearxivFri, Sep 18, 2026, 01:11 AM