Releases·Tool Release
Dev open-sources dependency-free LLM eval harness (MIT) with JSON gold sets, CI exit codes, and regression-diffing notes
A developer open-sourced llm-eval-harness (MIT license), a standard-library-only harness for evaluating LLM output against a JSON gold set. It reports accuracy and per-item failures, and exits non-zero so it can gate regressions in CI pipelines. It targets the common problem of repeatedly rewriting ad-hoc scoring scripts.
8 engagement1 sourceMon, Sep 7, 2026, 12:37 PM
AI Overview
- zahid23saim released llm-eval-harness (MIT, stdlib-only) that runs JSON gold sets with exact/contains/numeric matching, prints failing items, and exits non-zero for CI
- The author advises diffing runs for items that flipped from pass to fail rather than trusting average score, since averages can hide regressions
- For open-ended answers, the author recommends model-judged rubrics at temperature 0, spot-checked against human labels
Entities
Related
Social·Community
Community discussion: most practitioners skip rigorous LLM evaluation discipline
A Reddit discussion argues that most people changing LLM prompts or models rely on eyeballing a few outputs rather than disciplined evaluation. The poster advocates bootstrap confidence intervals and paired significance testing to know whether changes actually help. The thread highlights a gap between available eval methods and common practice.
A Reddit discussion argues that most people changing LLM prompts or models rely on eyeballing a few outputs rather than disciplined evaluation. The poster advocates bootstrap confidence intervals and paired significance testing to know whether changes actually help. The thread highlights a gap between available eval methods and common practice.
53 engagement1 sourceredditTue, Sep 1, 2026, 06:30 AM
Products·launched
EvalSeal: reproducibility receipts for LLM evals
EvalSeal is an open-source tool that runs LLM eval cases multiple times, measures per-case instability, and seals results into a tamper-evident ledger. It addresses the problem of not knowing whether an eval score change came from model, judge, prompt, or noise. It targets developers building and CI-testing LLM and agent systems.
EvalSeal is an open-source tool that runs LLM eval cases multiple times, measures per-case instability, and seals results into a tamper-evident ledger. It addresses the problem of not knowing whether an eval score change came from model, judge, prompt, or noise. It targets developers building and CI-testing LLM and agent systems.
21032 engagement3 sourcesredditFri, Sep 18, 2026, 08:08 AM
Products·launched
Noise measurement for LLM-as-judge regression gates in CI
An open-source (Apache-2.0) approach to making LLM-judged CI quality gates trustworthy by measuring the judge's own variance. Born from a case where an identical input scored 5/5 then 2/5, wasting an hour debugging a phantom regression. Aimed at developers running LLM-judged regression gates whose flaky verdicts erode trust.
An open-source (Apache-2.0) approach to making LLM-judged CI quality gates trustworthy by measuring the judge's own variance. Born from a case where an identical input scored 5/5 then 2/5, wasting an hour debugging a phantom regression. Aimed at developers running LLM-judged regression gates whose flaky verdicts erode trust.
18 engagement1 sourceredditWed, Sep 9, 2026, 11:55 AM
Social·Community
Practitioners swap canary-prompt techniques to catch silent model regressions after provider updates
A series of Reddit posts describes a shared operational pain: models from OpenAI, Anthropic, OpenRouter, and Moonshot quietly change under production workflows, with no changelog covering behavioral shifts. Practitioners converge on the same mitigation — fixed canary prompts or small golden eval sets re-run on a schedule, checking format, required fields, refusal behavior, latency, and cost. The thread reflects teams currently catching regressions only by accident or customer complaints.
A series of Reddit posts describes a shared operational pain: models from OpenAI, Anthropic, OpenRouter, and Moonshot quietly change under production workflows, with no changelog covering behavioral shifts. Practitioners converge on the same mitigation — fixed canary prompts or small golden eval sets re-run on a schedule, checking format, required fields, refusal behavior, latency, and cost. The thread reflects teams currently catching regressions only by accident or customer complaints.
24261 engagement8 sourcesredditTue, Aug 18, 2026, 10:19 AM
Social·Community
Practitioners ask how to detect drift in hosted LLM judges
A practitioner using a hosted LLM judge for summary faithfulness asks how to detect when the judge itself drifts, since the model updates without warning. The question highlights a growing concern about the reliability of LLM-as-a-judge evaluation as the measuring instrument changes on the same schedule as what it measures.
A practitioner using a hosted LLM judge for summary faithfulness asks how to detect when the judge itself drifts, since the model updates without warning. The question highlights a growing concern about the reliability of LLM-as-a-judge evaluation as the measuring instrument changes on the same schedule as what it measures.
15 engagement1 sourceredditThu, Aug 27, 2026, 01:15 PM