Releases·Tool Release
Driftproof, a Claude skill drift checker, launches v0.10.0; launch-day auditors forge a pass, trigger spend, and expose 6.2% noise
A developer shipped Driftproof v0.10.0, a tool that tests Claude skills and CLAUDE.md files across models to detect silent regressions, and published the results of a same-day public audit. Two independent auditors confirmed the core functionality works but broke three things: forged passes via edited grader notes, slash commands spending user subscriptions without permission flags, and a 6.2% false-positive rate on unchanged inputs. The developer committed to fixing the slash command issue immediately and treating the forged-pass and noise problems as next priorities.
1 engagement1 sourceSun, Sep 13, 2026, 05:00 PM
AI Overview
- Driftproof v0.10.0 runs skills with and without on chosen models, scores runs, and writes dated hashed receipts to flag regressions
- An auditor forged a pass by editing only the grader's notes, which the tool claims unsaved but actually stores
- Three slash commands lacked the flag blocking autonomous Claude execution; one makes many model calls
- A synthetic test of 20,000 no-change comparisons reported change 1,240 times (6.2%), half regressions, half improvements
- Author confirms Windows untested and macOS failing 19 checks; slash command fixes land launch day, repo github.com/driftproofhq/driftproof
Entities
Related
Products·launched
Free 60-second drill that scores your ability to catch LLM hallucination and prompt drift
A free web-based test that measures how well users notice when an LLM silently reinterprets their question — softening framing, inventing details, or smoothing tension. It is built around 12 documented, reproducible drift patterns with a scoring rig. It targets people who use LLMs and want to spot when answers drift from what they actually asked.
A free web-based test that measures how well users notice when an LLM silently reinterprets their question — softening framing, inventing details, or smoothing tension. It is built around 12 documented, reproducible drift patterns with a scoring rig. It targets people who use LLMs and want to spot when answers drift from what they actually asked.
0 engagement1 sourceredditWed, Sep 30, 2026, 04:55 PM
Releases·Tool Release
Your CLAUDE.md can reference code that no longer exists. I made a small pre-commit check for it
A developer released check-docs, a 39-line sh pre-commit script that catches when CLAUDE.md or AGENTS.md instruction files reference files or components that no longer exist. Stale agent instructions make Claude appear to hallucinate when the real problem is outdated project context. The tool runs with no model calls, packages, or services.
A developer released check-docs, a 39-line sh pre-commit script that catches when CLAUDE.md or AGENTS.md instruction files reference files or components that no longer exist. Stale agent instructions make Claude appear to hallucinate when the real problem is outdated project context. The tool runs with no model calls, packages, or services.
1 engagement1 sourceredditFri, Sep 18, 2026, 07:59 PM
Products·launched
DriftproofHQ — tests whether Claude skills actually improve results across tasks and model updates
DriftproofHQ evaluates Claude skills (saved instruction sets) by running the same tasks with and without those skills repeatedly, so users can compare outcomes rather than trust a single example. The author also built 'Train Your Replacement', a game made with Claude to explain the concept. It targets Claude users who want to know if their skills still help after model updates.
DriftproofHQ evaluates Claude skills (saved instruction sets) by running the same tasks with and without those skills repeatedly, so users can compare outcomes rather than trust a single example. The author also built 'Train Your Replacement', a game made with Claude to explain the concept. It targets Claude users who want to know if their skills still help after model updates.
6 engagement2 sourcesredditMon, Oct 12, 2026, 07:50 AM
Social·Community
I read the scoring code of 7 tools people use to grade Claude Code skills and LLM apps (NVIDIA, MLflow, LangSmith, DSPy, DeepEval...). Found 13 cases where the result wasn't backed by what the tool ac
A developer read the actual scoring code of 7 eval tools used to grade Claude Code skills and LLM apps, including NVIDIA SkillEvaluator, MLflow, LangSmith, DSPy, DeepEval, and Harbor, and found 13 cases where reported scores weren't backed by what the tool checked. They filed all 13 upstream, sent fixes for 12, and 6 were merged, including an MLflow fix merged the same day. The writeup highlights that eval infrastructure can confidently report wrong answers, undermining the core question of whether skills still help after model updates.
A developer read the actual scoring code of 7 eval tools used to grade Claude Code skills and LLM apps, including NVIDIA SkillEvaluator, MLflow, LangSmith, DSPy, DeepEval, and Harbor, and found 13 cases where reported scores weren't backed by what the tool checked. They filed all 13 upstream, sent fixes for 12, and 6 were merged, including an MLflow fix merged the same day. The writeup highlights that eval infrastructure can confidently report wrong answers, undermining the core question of whether skills still help after model updates.
4 engagement1 sourceredditTue, Oct 6, 2026, 08:01 AM
Social·Community
Driftproof skill-testing tool refuses to publish results after baseline control fails to reproduce
The maintainer of Driftproof, an open-source tool that re-tests agent skills (SKILL.md files) when the underlying model changes, found that all three skill revisions under test came back 'refused' because the no-skill baseline control failed to reproduce. Rather than suppress the finding, the maintainer published the refusal itself as the report. The episode highlights variance in agent evals: the same prompt and model produced scores from 0.30 to 0.81 across ten runs.
The maintainer of Driftproof, an open-source tool that re-tests agent skills (SKILL.md files) when the underlying model changes, found that all three skill revisions under test came back 'refused' because the no-skill baseline control failed to reproduce. Rather than suppress the finding, the maintainer published the refusal itself as the report. The episode highlights variance in agent evals: the same prompt and model produced scores from 0.30 to 0.81 across ten runs.
10 engagement2 sourcesredditSat, Aug 29, 2026, 06:44 PM