Releases·Tool Release
Developer's daily probes of 34 LLM APIs since Aug 20 catch DeepSeek reasoner using 10-12x thinking tokens on Sept 10
A developer has run identical probes against 34 models from 15 labs daily since Aug 20, comparing each model only to its own hashed baseline in a public transparency log. On Sept 10 the monitor caught DeepSeek's reasoner using 10-12x more thinking tokens than in prior weeks, with no announcement. The project, Seismograph, gives API users a way to prove quiet behavioral changes they previously could only describe as a model feeling 'off'.
25 engagement1 sourceWed, Sep 30, 2026, 12:48 PM
AI Overview
- Developer runs identical code-graded probes daily against 34 models from 15 labs since Aug 20, hashing each reading into a public transparency log
- On Sept 10, DeepSeek's reasoner began using roughly 10 to 12x the thinking tokens of the prior three weeks on the same probes, with no announcement
- No other model among the 34 shows a quiet capability drop; Opus 5.5 has been monitored since launch
- Methodology and live readings published on GitHub (piperoll/seismograph) and seismo.piperoll.org
Related
Developments·Benchmark
Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D]
A Reddit writeup presents findings from 31,352 repeated LLM benchmark measurements, arguing that static benchmarks are snapshots while API-served models can change behavior over time without public version transitions. The author frames benchmarking as a longitudinal measurement problem, continuously evaluating models on coding, multi-turn reasoning, and tool use. For practitioners, published scores may not describe how a model behaves days or weeks later as serving infrastructure and configurations shift.
A Reddit writeup presents findings from 31,352 repeated LLM benchmark measurements, arguing that static benchmarks are snapshots while API-served models can change behavior over time without public version transitions. The author frames benchmarking as a longitudinal measurement problem, continuously evaluating models on coding, multi-turn reasoning, and tool use. For practitioners, published scores may not describe how a model behaves days or weeks later as serving infrastructure and configurations shift.
21010 engagement3 sourcesredditMon, Sep 7, 2026, 07:44 AM
Social·Community
Anthropic traces 'Claude got dumber' reports to infrastructure bugs, not silent model changes
A therapist-researcher's Reddit post argues that perceived Claude 'nerfs' are mostly explained by documented infrastructure bugs, changing workloads, and novelty effects rather than secret weight swaps. Anthropic's own postmortems confirm several serving-side degradations in 2025-2026, and independent daily benchmark tracking caught one statistically significant dip. Daily benchmark noise of ~15 points makes short-term judgments unreliable.
A therapist-researcher's Reddit post argues that perceived Claude 'nerfs' are mostly explained by documented infrastructure bugs, changing workloads, and novelty effects rather than secret weight swaps. Anthropic's own postmortems confirm several serving-side degradations in 2025-2026, and independent daily benchmark tracking caught one statistically significant dip. Daily benchmark noise of ~15 points makes short-term judgments unreliable.
50 engagement2 sourcesredditWed, Sep 23, 2026, 12:44 PM
Developments·Paper
Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases
A paper on arxiv introduces a cheap, replicable methodology for studying language model behavior: frozen public stimulus batteries run identically across cross-vendor model panels for a few dollars per model. Applied to four years of releases from frontier and open-source labs, the assays find convergence, sycophancy resistance, and lab-specific behavioral signatures.
A paper on arxiv introduces a cheap, replicable methodology for studying language model behavior: frozen public stimulus batteries run identically across cross-vendor model panels for a few dollars per model. Applied to four years of releases from frontier and open-source labs, the assays find convergence, sycophancy resistance, and lab-specific behavioral signatures.
0 engagement1 sourcearxivThu, Sep 24, 2026, 03:51 PM
Social·Community
Student seeks advice on building LLM black-box behavior drift detection system
A student is building an automated system to detect statistically significant behavioral drift in deployed LLM/API services by repeatedly evaluating fixed probe prompts and comparing current responses against historical baselines. The approach scores raw responses on semantic and behavioral features, then applies change point and anomaly detection to flag unusual drift. The student is seeking advice on the project.
A student is building an automated system to detect statistically significant behavioral drift in deployed LLM/API services by repeatedly evaluating fixed probe prompts and comparing current responses against historical baselines. The approach scores raw responses on semantic and behavioral features, then applies change point and anomaly detection to flag unusual drift. The student is seeking advice on the project.
1 engagement1 sourceredditThu, Aug 20, 2026, 01:30 PM
Social·Community
Are They Nerfing Opus 5.5? Livenerf Day 20 update
The LiveNerf project, a community effort to detect post-release capability degradation ('nerfs') in Anthropic's Opus 5.5, published its day-20 update. After three weeks of daily measurement, variance decreased after the baseline period, suggesting regression to the mean rather than a nerf. The author plans a final verdict in 10 days and asks the community to contribute anecdotal reports of perceived capability drops.
The LiveNerf project, a community effort to detect post-release capability degradation ('nerfs') in Anthropic's Opus 5.5, published its day-20 update. After three weeks of daily measurement, variance decreased after the baseline period, suggesting regression to the mean rather than a nerf. The author plans a final verdict in 10 days and asks the community to contribute anecdotal reports of perceived capability drops.
5 engagement1 sourceredditWed, Oct 14, 2026, 07:10 PM