Developments·Benchmark
I made my own cybersecurity benchmark and ran Qwen3.8 27B, here's how a local model actually does at hacking
A Reddit user released a 19-task cybersecurity benchmark where models attack isolated Docker boxes to find flags, with 544 scored attempts across 6 models. Local Qwen3.8 27B scored 28.1%, while cheap API models like GPT-6 Luna reached 90.9%. The post was prompted by a John Hammond video showing criminals use the same model on RunPod.
88 engagement1 sourceFri, Oct 2, 2026, 12:15 PM
AI Overview
- A Redditor built a 19-task CTF-style cyber benchmark (544 scored attempts, 6 models) where models hunt flags in isolated Docker boxes with shell access.
- Stock Qwen3.8 27B on a two-GPU llama.cpp RPC pool solved 28.1% of tasks first-try and 0% of pwn tasks.
- MiMo 2.6 Flash solved 73.7% first-try; GPT-6 Luna solved 90.9% within 3 tries; top models hit 92-96% on multi-stage ranges.
- Pwn remains hardest (best model: 56%), and 3 tasks went unsolved in 82 attempts; the MIT-licensed harness and results are public.
- The poster cites John Hammond's video showing a criminal-forum guide for running abliterated models like Qwen3.8 27B on RunPod.
- GLM 5.3, not among benchmarked models, helped author several tasks and barely refuses anything, per the benchmark's creator.
Related
Developments·Paper
Anthropic red team: GLM-5.3 achieves full control flow hijacks on 4% of binary exploitation tasks
Anthropic's Frontier Red Team published an evaluation of GLM-5.3 and Claude Mythos Preview on 100 randomly selected tasks from an internal Binary Exploitation benchmark. GLM-5.3 produced full control flow hijacks in 4% of trials versus 6% for Claude Mythos Preview, while earlier models like Claude Opus 4.6 and GLM-5.2 succeeded in none. The report frames this as a meaningful threshold being crossed in AI cyber capabilities.
Anthropic's Frontier Red Team published an evaluation of GLM-5.3 and Claude Mythos Preview on 100 randomly selected tasks from an internal Binary Exploitation benchmark. GLM-5.3 produced full control flow hijacks in 4% of trials versus 6% for Claude Mythos Preview, while earlier models like Claude Opus 4.6 and GLM-5.2 succeeded in none. The report frames this as a meaningful threshold being crossed in AI cyber capabilities.
0 engagement1 sourcerssTue, Sep 29, 2026, 10:20 PM
Developments·Benchmark
Community launches HunterBench to compare LLMs on pentesting tasks
A Reddit post introduces HunterBench, a benchmark for comparing which LLMs perform best at penetration testing. The author argues existing options like CyberGym focus on promoting agents and tooling rather than comparing models, and cover mainly exploit/PoC generation for known vulnerabilities in isolated environments. The benchmark aims to fill the gap for evaluating the latest models on real pentesting work.
A Reddit post introduces HunterBench, a benchmark for comparing which LLMs perform best at penetration testing. The author argues existing options like CyberGym focus on promoting agents and tooling rather than comparing models, and cover mainly exploit/PoC generation for known vulnerabilities in isolated environments. The benchmark aims to fill the gap for evaluating the latest models on real pentesting work.
5 engagement1 sourceredditSun, Aug 30, 2026, 11:08 PM
Releases·Model Release
CyberKimi, an unrestricted Kimi K3 fine-tune, solves ExploitBench's hard V8 bug
AI researcher Taha (lordx64) released CyberKimi, a privacy-first, unrestricted cybersecurity model fine-tuned from Moonshot's Kimi K3, and demonstrated strong results on ExploitBench's v8-cve-2024-6100 challenge. The model is designed for both red and blue team operations.
AI researcher Taha (lordx64) released CyberKimi, a privacy-first, unrestricted cybersecurity model fine-tuned from Moonshot's Kimi K3, and demonstrated strong results on ExploitBench's v8-cve-2024-6100 challenge. The model is designed for both red and blue team operations.
2 engagement2 sourcesredditSun, Aug 9, 2026, 04:14 PM
Products·building
Runs local and cloud LLMs against GitHub repos to find real security vulnerabilities
A benchmark-style project comparing local open-source and frontier cloud models on real security auditing of public GitHub codebases. Over two weeks it logged 1,665 model runs across 27 repos, producing 1,067 findings with per-model accuracy results. Results are published in a verifiable audit database, aimed at developers evaluating which models are reliable for security work.
A benchmark-style project comparing local open-source and frontier cloud models on real security auditing of public GitHub codebases. Over two weeks it logged 1,665 model runs across 27 repos, producing 1,067 findings with per-model accuracy results. Results are published in a verifiable audit database, aimed at developers evaluating which models are reliable for security work.
81 engagement1 sourceredditMon, Sep 7, 2026, 06:51 PM
Developments·Benchmark
Adverserial launches CyberPVP: CyberKimi vs Aikido's ALTAR-1 head-to-head on 100 CyberGym tasks with live traces
A small team at Adverserial introduced CyberPVP, a live head-to-head benchmark where security-focused models compete on the same cybersecurity tasks. The first public run matches CyberKimi against Aikido Security's ALTAR-1 on 100 randomly selected CyberGym tasks, with traces streamed live and published for independent verification.
A small team at Adverserial introduced CyberPVP, a live head-to-head benchmark where security-focused models compete on the same cybersecurity tasks. The first public run matches CyberKimi against Aikido Security's ALTAR-1 on 100 randomly selected CyberGym tasks, with traces streamed live and published for independent verification.
1 engagement1 sourceredditMon, Sep 28, 2026, 02:49 PM