Developments·Paper
KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards
A new arXiv paper introduces KaliBench, a benchmark and dataset for natural-language-to-CLI translation on Kali Linux, with 8,504 query-command pairs spanning 1,642 tools. Across 24 configurations of open-weight models, none exceeded 42% exact-command accuracy without tool hints. Fine-tuning an 8B model on its verifiable rewards matched a 685B MoE model.
0 engagement1 sourceThu, Oct 1, 2026, 05:59 PM
AI Overview
- KaliBench introduces 8,504 query-command pairs spanning 1,642 Kali Linux tools across 23 capability dimensions and 5 security phases.
- No open-weight model exceeds 42% exact-command accuracy in unrestricted settings across 24 configurations and three evaluation modes.
- Fine-tuning and reinforcement learning with verifiable rewards lift an 8B model to performance comparable to a 685B MoE model.
- A multi-stage verification pipeline combines LLM-based validation, sandboxed terminal execution, and human-in-the-loop refinement for correctness and executability.
Entities
Related
Developments·Paper
arXiv Audit: Cybersecurity LLM Benchmark Scores Shift 80+ Points on Pipeline Choices
An arXiv paper audits eight cybersecurity LLM benchmarks across 10 models, treating benchmarks as configurable measurement pipelines rather than fixed datasets. The authors identify 15 systematic failure modes and show a single pipeline choice can swing a model's score by more than 80 percentage points and reorder model rankings. The findings caution practitioners against treating benchmark scores as stable properties of a model.
An arXiv paper audits eight cybersecurity LLM benchmarks across 10 models, treating benchmarks as configurable measurement pipelines rather than fixed datasets. The authors identify 15 systematic failure modes and show a single pipeline choice can swing a model's score by more than 80 percentage points and reorder model rankings. The findings caution practitioners against treating benchmark scores as stable properties of a model.
0 engagement1 sourcearxivTue, Sep 8, 2026, 02:01 PM
Developments·Benchmark
Community launches HunterBench to compare LLMs on pentesting tasks
A Reddit post introduces HunterBench, a benchmark for comparing which LLMs perform best at penetration testing. The author argues existing options like CyberGym focus on promoting agents and tooling rather than comparing models, and cover mainly exploit/PoC generation for known vulnerabilities in isolated environments. The benchmark aims to fill the gap for evaluating the latest models on real pentesting work.
A Reddit post introduces HunterBench, a benchmark for comparing which LLMs perform best at penetration testing. The author argues existing options like CyberGym focus on promoting agents and tooling rather than comparing models, and cover mainly exploit/PoC generation for known vulnerabilities in isolated environments. The benchmark aims to fill the gap for evaluating the latest models on real pentesting work.
5 engagement1 sourceredditSun, Aug 30, 2026, 11:08 PM
Developments·Paper
SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction
A new arxiv paper introduces SWE-Test, a benchmark measuring LLM agents' ability to discover vulnerabilities via input prediction. It targets flaws in existing benchmarks: data contamination, unknowable vulnerability sets, synthetic bugs, and single end-to-end verdicts that cannot localize agent failures.
A new arxiv paper introduces SWE-Test, a benchmark measuring LLM agents' ability to discover vulnerabilities via input prediction. It targets flaws in existing benchmarks: data contamination, unknowable vulnerability sets, synthetic bugs, and single end-to-end verdicts that cannot localize agent failures.
0 engagement1 sourcearxivSat, Sep 5, 2026, 07:04 PM
Developments·Benchmark
SecRespond benchmark evaluates LLM agents on post-compromise incident response
Researchers introduced SecRespond, the first benchmark for evaluating LLM agents on post-compromise incident response workflows. It addresses the gap left by existing cybersecurity benchmarks that focus only on pre-compromise settings. The benchmark tests agents on real-world tasks like forensic analysis and remediation using host artifacts and CLI access.
Researchers introduced SecRespond, the first benchmark for evaluating LLM agents on post-compromise incident response workflows. It addresses the gap left by existing cybersecurity benchmarks that focus only on pre-compromise settings. The benchmark tests agents on real-world tasks like forensic analysis and remediation using host artifacts and CLI access.
0 engagement1 sourcearxivWed, Jul 29, 2026, 11:32 AM
Developments·Benchmark
CS-Guard: First Benchmark of LLM Guardrails for Code Generation Security
A new arXiv paper introduces CS-Guard, presented as the first benchmark to systematically evaluate LLM guardrails for code generation security. It matters because LLMs are being exploited to generate malware, and the effectiveness of guardrails against such misuse had not been systematically measured. Code-generation teams and safety researchers get a standardized way to assess guardrail coverage.
A new arXiv paper introduces CS-Guard, presented as the first benchmark to systematically evaluate LLM guardrails for code generation security. It matters because LLMs are being exploited to generate malware, and the effectiveness of guardrails against such misuse had not been systematically measured. Code-generation teams and safety researchers get a standardized way to assess guardrail coverage.
0 engagement1 sourcearxivWed, Sep 9, 2026, 06:50 AM