Deterministic SAST vs. AI Security Agents: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
The paper presents VulnBench 2.0, a benchmark of 108 hand-curated, attacker-reachable vulnerabilities across 20 open-source web apps (Go, JavaScript/TypeScript, Python, Java), credited only when a finding locates both the attacker-controlled source and dangerous sink within two lines. Nine productized configurations were evaluated blind, including Snyk Code, Claude Code's /security-review, Codex Security with GPT-5.6 variants, and DeepSec. Snyk Code reached 82.4% macro recall (89/108) and Claude Code with Opus 5 at xhigh effort 79.3% (85/108), disagreeing on 28 vulnerabilities (16–12, too few to distinguish them). Neither sufficed alone: combined they reached 101/108, the ceiling of all seven comparable configurations, and AI tools also found at least seven real vulnerabilities missing from the reference set. What AI engines miss depends heavily on the harness as well as the model, and coverage was only loosely tied to cost. All harnesses, fixtures, reference sets, and run data are public.
Comments
Post a Comment