A developer shared Wayfinder on GitHub as a reference implementation for evaluating AI applications. It targets the challenge that AI apps are non-deterministic, so traditional software testing approaches don't transfer directly. The project aims to help AI engineers ship changes without breaking existing behavior.
Medium essay: why evaluation became a core AI engineering problem
Medium essay: why evaluation became a core AI engineering problem
A Medium post argues unit tests are insufficient for AI systems and outlines what an engineering-grade evaluation stack looks like. It reflects a growing practitioner shift toward treating evals as core engineering infrastructure rather than an afterthought.
A Medium post argues unit tests are insufficient for AI systems and outlines what an engineering-grade evaluation stack looks like. It reflects a growing practitioner shift toward treating evals as core engineering infrastructure rather than an afterthought.
A new article argues that AI evals should be treated like unit tests in software development, providing a systematic way to build reliable AI products. The piece likely offers practical guidance for integrating evals into development workflows.
A new article argues that AI evals should be treated like unit tests in software development, providing a systematic way to build reliable AI products. The piece likely offers practical guidance for integrating evals into development workflows.
A new arXiv paper presents an end-to-end eval pipeline combining eval checklist creation with learned aggregation of checklist responses to improve LLM judge agreement and accuracy against human judgments. It targets the full set of practical eval requirements rather than isolated aspects, offering a quality-gate tool for LLM-based software development.
A new arXiv paper presents an end-to-end eval pipeline combining eval checklist creation with learned aggregation of checklist responses to improve LLM judge agreement and accuracy against human judgments. It targets the full set of practical eval requirements rather than isolated aspects, offering a quality-gate tool for LLM-based software development.
A tool for builders to apply statistical methods like confidence intervals and Fischer exact test to evaluate prompt and agent changes, providing reliable signals of application performance in realtime for potential buyers. It aims to make statistical rigor accessible to those unfamiliar with these concepts.
A tool for builders to apply statistical methods like confidence intervals and Fischer exact test to evaluate prompt and agent changes, providing reliable signals of application performance in realtime for potential buyers. It aims to make statistical rigor accessible to those unfamiliar with these concepts.