ExploitBench – Real exploitation is a ladder

ExploitBench is a benchmark from Carnegie Mellon University that measures how far AI agents climb the exploitation ladder, from reaching vulnerable code (T5 coverage) to triggering the bug (T4 reproduction) to building target-specific primitives (T3) to generic arbitrary read/write primitives (T2) to full arbitrary code execution (T1). The first benchmark, v8-bench, targets V8 (the JavaScript and WebAssembly engine inside Chrome, Edge, Node.js, and Cloudflare Workers) with the V8 security sandbox enabled, testing against 41 CVEs. Grading is deterministic with no LLM-as-judge. As of May 18, 2026, the leaderboard shows Claude Mythos Preview (with and without AutoNudge) achieving mean capability scores of 9.90/16 and 9.55/16, and GPT-5.5 (Codex) at 5.51. Mythos Preview reached Tier 1 (full arbitrary code execution) on 21 of 41 CVEs (51%), while GPT-5.5 cracked Tier 1 on 2 CVEs. Claude Opus 4.7 with AutoNudge escaped the V8 sandbox into Tier 2 on one CVE. The cheapest full ACE run cost $14 (GPT-5.5/Codex on one CVE), while Mythos Preview's typical full ACE cost is around $220 (range $72-$360). The benchmark includes an MCP server integration for Claude Code. AutoNudge is a component that automatically reminds stalled models to continue working with no human in the loop. The project is by Seunghyun Lee and Prof. David Brumley. 

https://exploitbench.ai/

Comments

Post a Comment

Popular posts from this blog

Prompt Engineering Demands Rigorous Evaluation

In the blog post “Prompt Engineering Requires Evaluation” on the Shostack + Associates website, the author argues that treating prompts for large language models (LLMs) merely as creative artefacts is insufficient. Engineering prompts properly demands structured evaluation frameworks — what the AI community calls “evals” — to test which prompt versions work better, with which models, and under which conditions. The post highlights that simply assuming a prompt is “good enough” creates risks when LLMs are integrated into production systems (e.g., for threat modeling). It advocates for measuring prompt performance, variation effects, and tool-chain dependencies (model, context, ancillary materials).  Ultimately the message is: prompt engineering should borrow disciplined practices from software engineering (versioning, testing, benchmarking) rather than relying on informal experimentation.  https://shostack.org/blog/prompt-enignieering-requires-evaluation/
Read more

SecObserve: Simplified Vulnerability and License Management for CI/CD Pipelines

 SecObserve is an open-source tool for managing vulnerabilities and licenses in software development and cloud environments. It integrates various vulnerability scanners into CI/CD pipelines using GitLab CI templates and GitHub Actions for streamlined setup. It offers a centralized dashboard for assessing and reporting vulnerabilities, with tools for filtering, sorting, and evaluating results. SecObserve supports automation and manual assessments to focus on resolving critical issues.  https://github.com/MaibornWolff/SecObserve/tree/dev
Read more

Secure Vibe Coding Guide: Best Practices for Writing Secure Code

Ken Huang's "Secure Vibe Coding Guide" emphasizes the importance of integrating security into the software development lifecycle. The guide provides best practices for writing secure code, including input validation, proper authentication mechanisms, and secure data storage techniques. It also highlights the necessity of regular code reviews and staying updated with the latest security vulnerabilities and patches. By following these guidelines, developers can create applications that are resilient against common security threats and contribute to a safer digital environment.  https://kenhuangus.substack.com/p/secure-vibe-coding-guide
Read more