Developments·Benchmark
I gave Claude models a web_search tool and 40 questions. Opus decided right 79 of 80 times. Sonnet 5 with no system prompt told me Harald V is still king.
A Reddit user ran a 2,000-call harness testing when 16 models via OpenRouter decide to invoke a web_search tool, using 20 post-cutoff and 20 evergreen questions across four system-prompt conditions. Claude Opus decided correctly 79 of 80 times with zero false searches, while Sonnet 5 without a system prompt confidently answered stale facts — including calling Harald V, who died 28 August, Norway's reigning king. A one-line instruction to search fixed Sonnet 5 and Haiku, but not GPT-5.6 Luna, whose issue was certainty rather than attention.
38 engagement1 sourceFri, Oct 2, 2026, 12:17 PM
AI Overview
- A Reddit user's harness ran 16 models through OpenRouter, 2000 calls, temperature 0, $6.53 self-funded, recording search decisions on 20 post-cutoff and 20 evergreen prompts.
- Opus made 79 of 80 correct search decisions with zero false searches; Fable scored 78 of 80, confabulating a WTA title winner when it skipped the search.
- Sonnet 5 with no system prompt gave three confident stale answers and zero searches; one instruction line cut its misses from 3 to 0, and cut Haiku's false searches from 5 to 0.
- GPT-5.6 Luna kept the dead king through all four system prompts, including the instructed one; the author attributes the failure to certainty rather than attention.
Entities
Related
Developments·Benchmark
User-built EchoNet benchmark tests 9 open models on resisting misinformation during agentic search
A Reddit user introduced EchoNet, a benchmark measuring how well open models handle misinformation encountered during agentic web search. Agents answer factual questions using a synthetic web seeded with fake pages, contradicting sources, and outdated information. The results matter to practitioners building search agents, where trusting a wrong source can corrupt answers.
A Reddit user introduced EchoNet, a benchmark measuring how well open models handle misinformation encountered during agentic web search. Agents answer factual questions using a synthetic web seeded with fake pages, contradicting sources, and outdated information. The results matter to practitioners building search agents, where trusting a wrong source can corrupt answers.
102 engagement1 sourceredditFri, Aug 28, 2026, 07:03 PM
Products·launched
Chrome extension that freezes websites at their 2019 design
A Chrome extension that restores 170+ popular websites to how they looked around 2019, for users who dislike frequent redesigns. After failed attempts with Wayback Machine archives and an algorithmic restyling approach, the builder used Claude Opus to generate the tailored designs.
A Chrome extension that restores 170+ popular websites to how they looked around 2019, for users who dislike frequent redesigns. After failed attempts with Wayback Machine archives and an algorithmic restyling approach, the builder used Claude Opus to generate the tailored designs.
14 engagement2 sourcesredditMon, Oct 5, 2026, 04:10 PM
Social·Community
Users report Claude Opus 5 WebFetch research returns fabricated details
A user reports that Claude Opus 5's WebFetch tool for research returned fabricated stats and quotes, because a smaller, cheaper model summarizes web pages before Opus sees them. The user discovered the summaries were not faithful to the source papers, raising concerns about the reliability of agentic research workflows.
A user reports that Claude Opus 5's WebFetch tool for research returned fabricated stats and quotes, because a smaller, cheaper model summarizes web pages before Opus sees them. The user discovered the summaries were not faithful to the source papers, raising concerns about the reliability of agentic research workflows.
864 engagement1 sourceredditSat, Aug 8, 2026, 04:22 AM
Social·Community
Litigation attorney criticizes Anthropic's model opacity, says unexplained Claude results undermine legal work
An attorney using Claude for legal document analysis posted on Reddit asking Anthropic to explain how its models differ, noting unpredictable results like Opus returning 12 court opinions where Fable returned 14 on the same research query. The user argues high-stakes legal work needs predictable outputs and proposes a high-priced, NDA-backed subscription tier with enhanced security audits as a compromise.
An attorney using Claude for legal document analysis posted on Reddit asking Anthropic to explain how its models differ, noting unpredictable results like Opus returning 12 court opinions where Fable returned 14 on the same research query. The user argues high-stakes legal work needs predictable outputs and proposes a high-priced, NDA-backed subscription tier with enhanced security audits as a compromise.
27 engagement1 sourceredditFri, Sep 18, 2026, 02:44 PM
Releases·Tool Release
Humanbound adds OpenRouter support to its AI red-teamer CLI; tests show thinking models can drop 81% of scorer replies
Humanbound released hb 2.13 of its open-source AI red-teaming CLI, adding OpenAI-compatible endpoint support so any OpenRouter model can drive scans. Testing five setups showed reasoning models often exhaust the scorer's 50-token budget and return nothing, while DeepSeek V4.1 Flash with thinking off completed full scans for $0.14.
Humanbound released hb 2.13 of its open-source AI red-teaming CLI, adding OpenAI-compatible endpoint support so any OpenRouter model can drive scans. Testing five setups showed reasoning models often exhaust the scorer's 50-token budget and return nothing, while DeepSeek V4.1 Flash with thinking off completed full scans for $0.14.
4 engagement2 sourcesredditWed, Oct 7, 2026, 07:10 AM