Chronicle: Cut-Point Replay Turns Recorded LLM Agent Failures into CI Regression Tests
Researchers present Chronicle, a record-and-replay tool for regression-testing LLM agents. It records agent runs at non-deterministic boundaries as immutable envelopes, and its cut-point replay operation replays a chosen subset of boundaries from the record while executing the rest live with new code. This turns a recorded incident into a regression test that runs in continuous integration, addressing the fact that agent failures are rarely reproducible due to non-deterministic inference, stateful tools, and multi-step trajectories.
- Chronicle (arxiv, 2026-09-17) records agent runs as immutable envelopes at non-deterministic boundaries and replays them from the record.
- Cut-point replay serves chosen boundaries from the record and executes the rest live with new code, making incidents regression tests in CI.
- Benchmark: recording adds 23 μs per boundary crossing, full replay is bit-stable across 20 repetitions with zero model calls.
- Cut-point tests fail on faulty code and pass on guarded/benign changes for all 6 recorded incidents; a stub-all-boundaries baseline catches none.
- Mutation study: cut-point tests catch every mutant allowing the recorded unsafe action through; code and benchmark are on GitHub.
Related
How should CI decide if a stochastic agent actually regressed?
A developer released AgentSeism, an open-source tool that treats agent-in-CI evaluation like an experiment rather than a deterministic test, with repeated runs, effect thresholds, uncertainty, and an explicit 'insufficient evidence' state. It targets the problem that agents produce meaningfully different outcomes with no code change, so naive score-threshold rules both raise false alarms and miss concentrated regressions.
A developer released AgentSeism, an open-source tool that treats agent-in-CI evaluation like an experiment rather than a deterministic test, with repeated runs, effect thresholds, uncertainty, and an explicit 'insufficient evidence' state. It targets the problem that agents produce meaningfully different outcomes with no code change, so naive score-threshold rules both raise false alarms and miss concentrated regressions.
ZenML open-sources Kitaru to replay production agent traces against different models
ZenML maintainers open-sourced Kitaru, an Apache 2.0 tool that replays production agent traces against different models or prompts. It addresses the ad hoc eval workflow teams face when handling thousands of traces, enabling systematic regression testing and model comparison.
ZenML maintainers open-sourced Kitaru, an Apache 2.0 tool that replays production agent traces against different models or prompts. It addresses the ad hoc eval workflow teams face when handling thousands of traces, enabling systematic regression testing and model comparison.
I open-sourced Stepfork, a Python tool that turns AI agent failures into reproducible pytest tests
A developer open-sourced Stepfork, an early-alpha Python tool (Apache-2.0, pip installable) that records instrumented LLM and tool calls from agent runs, replays them locally for deterministic debugging, and exports failures as pytest regression tests. It addresses the common pain that agent behavior is hard to reproduce because LLM responses and tool outputs vary between runs.
A developer open-sourced Stepfork, an early-alpha Python tool (Apache-2.0, pip installable) that records instrumented LLM and tool calls from agent runs, replays them locally for deterministic debugging, and exports failures as pytest regression tests. It addresses the common pain that agent behavior is hard to reproduce because LLM responses and tool outputs vary between runs.
AgentSnap snapshot-testing library for LLM agents released as open source
A developer released AgentSnap, an MIT-licensed snapshot-testing library that records LLM agent calls as golden snapshots and diffs future runs to catch silent regressions. The tool compares tool-call sequences, arguments, model-requested tool calls, and semantic content of responses.
A developer released AgentSnap, an MIT-licensed snapshot-testing library that records LLM agent calls as golden snapshots and diffs future runs to catch silent regressions. The tool compares tool-call sequences, arguments, model-requested tool calls, and semantic content of responses.
AgentCheck: Open-Source Workbench for Testing LLM Agents Under Real-World Tool Failures
Researchers introduced AgentCheck, an open-source web workbench that allows developers to reproduce, intervene, and mitigate failures in tool-using LLM agents by injecting 12 types of faults into MCP server responses. It records real tool interactions, replays them with perturbations, and caches matching calls for efficient debugging before deployment.
Researchers introduced AgentCheck, an open-source web workbench that allows developers to reproduce, intervene, and mitigate failures in tool-using LLM agents by injecting 12 types of faults into MCP server responses. It records real tool interactions, replays them with perturbations, and caches matching calls for efficient debugging before deployment.