Releases·Tool Release
digline — open-source regression gate for LLM apps, with the baseline in your repo (LangChain example inside)
The author of digline announced an open-source Python tool that gates CI on LLM app regressions. Users commit approved baseline runs (scores, prompt, model config, commit) into the repo, and digline flags changes that drop beyond measured LLM judge noise.
1 engagement1 sourceThu, Sep 10, 2026, 08:24 AM
AI Overview
- Author launches digline, an Apache-2.0 Python tool that stores approved score baselines in-repo and gates CI via exit code on regressions
- digline measures LLM-judge noise by sampling each case on the approved version, flagging only drops beyond the min/max band over K samples
- New `digline diff run1 run2` compares prompts or models as a report with per-class aggregates; an MCP server lets agents run suites but never promote baselines
- digline targets in-process LangChain functions with a FakeListChatModel CI example and offers codeless TOML suites against HTTP endpoints; pre-1.0 via `pip install digline`
Related
Products·building
Assay Server — evaluation and observability for AI pipelines
Assay Server is an open-source project for evaluation and observability of AI pipelines, motivated by the author's work on a document intelligence system for a company. It provides regression testing, canary evaluation, and behavioral change detection to catch cases where an AI system passes basic tests but its behavior shifts after prompt, model, or workflow changes. It targets developers building LLM-powered systems like document processing.
Assay Server is an open-source project for evaluation and observability of AI pipelines, motivated by the author's work on a document intelligence system for a company. It provides regression testing, canary evaluation, and behavioral change detection to catch cases where an AI system passes basic tests but its behavior shifts after prompt, model, or workflow changes. It targets developers building LLM-powered systems like document processing.
1 engagement1 sourceredditMon, Oct 12, 2026, 03:06 PM
Products·launched
Open-source eval and monitoring workbench for LLM apps
An open-source workbench combining LLM test suites, production monitoring, and version comparison. LLMs are used as automated judges for evaluation checks via the user's own OpenAI key. It solves the problem of testing and observing LLM app behavior across development and production for developers.
An open-source workbench combining LLM test suites, production monitoring, and version comparison. LLMs are used as automated judges for evaluation checks via the user's own OpenAI key. It solves the problem of testing and observing LLM app behavior across development and production for developers.
1 engagement1 sourceredditMon, Sep 28, 2026, 02:54 AM
Products·launched
Noise measurement for LLM-as-judge regression gates in CI
An open-source (Apache-2.0) approach to making LLM-judged CI quality gates trustworthy by measuring the judge's own variance. Born from a case where an identical input scored 5/5 then 2/5, wasting an hour debugging a phantom regression. Aimed at developers running LLM-judged regression gates whose flaky verdicts erode trust.
An open-source (Apache-2.0) approach to making LLM-judged CI quality gates trustworthy by measuring the judge's own variance. Born from a case where an identical input scored 5/5 then 2/5, wasting an hour debugging a phantom regression. Aimed at developers running LLM-judged regression gates whose flaky verdicts erode trust.
18 engagement1 sourceredditWed, Sep 9, 2026, 11:55 AM
Releases·Tool Release
Dev open-sources dependency-free LLM eval harness (MIT) with JSON gold sets, CI exit codes, and regression-diffing notes
A developer open-sourced llm-eval-harness (MIT license), a standard-library-only harness for evaluating LLM output against a JSON gold set. It reports accuracy and per-item failures, and exits non-zero so it can gate regressions in CI pipelines. It targets the common problem of repeatedly rewriting ad-hoc scoring scripts.
A developer open-sourced llm-eval-harness (MIT license), a standard-library-only harness for evaluating LLM output against a JSON gold set. It reports accuracy and per-item failures, and exits non-zero so it can gate regressions in CI pipelines. It targets the common problem of repeatedly rewriting ad-hoc scoring scripts.
8 engagement1 sourceredditMon, Sep 7, 2026, 12:37 PM
Releases·Tool Release
AgentSnap snapshot-testing library for LLM agents released as open source
A developer released AgentSnap, an MIT-licensed snapshot-testing library that records LLM agent calls as golden snapshots and diffs future runs to catch silent regressions. The tool compares tool-call sequences, arguments, model-requested tool calls, and semantic content of responses.
A developer released AgentSnap, an MIT-licensed snapshot-testing library that records LLM agent calls as golden snapshots and diffs future runs to catch silent regressions. The tool compares tool-call sequences, arguments, model-requested tool calls, and semantic content of responses.
31 engagement2 sourcesreddit, githubWed, Jul 29, 2026, 08:19 PM