tl;dr: CWE-bench is a held-out benchmark of 100 audit-and-patch tasks that test whether coding agents can find and fix known vulnerabilities in public codebases across every defect class, not just the ones that are cheap to detect.
AI agents are further along in their attacking capabilities than in defending. They have been observed coordinating and turning on real infrastructure on their own (METR, OpenAI). An attacker who can do that needs one unguarded vulnerability; a defender needs every one of them covered.
We built CWE-bench around the CWE taxonomy for that reason, so it maps which vulnerabilities agents handle poorly.
CWE-bench features the following:
Repair, not exploitation. Agents are asked to close vulnerabilities, never to write exploits.
Coverage set by the CWE taxonomy, not by an oracle. 54 vulnerability types across six languages and all ten OWASP 2025 categories, each graded by a deterministic grader.
Real code, new vulnerabilities. Every task pins a real project at a real commit and then introduces vulnerabilities the published fix never covered, so memorizing advisories buys nothing.
Permanently held out. This keeps the dataset free of contamination and forces agent performance to reflect real defensive cyber capabilities.
Leaderboard
Each model’s pass@1 is plotted against the average cost of one attempt. Claude Fable 5 scores highest at about $10 per attempt, Gemini 3.8 Flash Cyber comes within a point of it near $3.70, and pass@1 stays close to 44% down to around $1.50.
Why CWE-bench
We build each task from a real open-source project pinned to a real commit, usually one with a published advisory, then add vulnerabilities of the same class that the upstream fix never covered, and label every task with its CWE and OWASP category so each of the ten OWASP categories has ten tasks.
Each of those choices closes a gap we kept running into elsewhere. Reproducing a published patch does not pass, so the benchmark cannot be beaten from memory the way one drawn straight from a CVE feed can (PatchEval). Nothing restricts us to what a single automated checker can detect, which is what confines sanitizer-scored benchmarks to memory safety in C and C++ (CyberGym-E2E). And every task is labeled, so we can report which vulnerabilities an agent misses rather than one number, which a benchmark that never states its vulnerability distribution cannot do.
How tasks are constructed
Sourcing
Every task pins a real open-source project at a real commit, most of them carrying a published advisory, and we keep the upstream provenance: the repository, the vulnerable commit, the fixing commit, and the advisory identifier. Nothing is a synthetic codebase. The agent receives two things: an instruction.md and the source tree mounted at /app, with no CVE identifier, no file hint, and no line number. The instruction describes the system and the property that must hold, never where the defect lives or how many instances there are.
Making them harder
A task that is only a real CVE on a real repository measures recall, since a well-read model can recognize the project, remember the advisory, and reproduce the upstream patch without doing any security reasoning. So every task carries additional vulnerabilities of the same class that the published fix does not reach, placed where a real instance of that flaw would live. An agent that identifies the known vulnerability and applies the memorized fix closes part of the problem and scores zero. All evaluation sandboxes prevent agents from accessing the Internet to avoid cheating.
Grading
CWE-bench uses a combination of deterministic programmatic proof-of-concept (PoC) and rubric-based checks that test whether the vulnerabilities have been patched. A task passes only when the exploit no longer works, and the regression tests still pass. Every grader runs the exploit rather than pattern matching over source, and none pin a function, module, or variable name, so patches can be written differently and still pass grading. Rubric checks ensure the patch closes the weakness itself rather than the one payload the PoC sends, and that it doesn't do so by disabling the feature or weakening the harness.
Where agents fail
Frontier agents are good at finding vulnerabilities. Given a codebase and the untrusted inputs, the strong models reliably identify what is wrong and why it is dangerous. They fail later, when the fix isn't where the bug is or doesn't have an obvious shape. Four patterns account for most of it.
Incomplete fixes. The same defect exists at several sites, and the agent patches the one it found without asking where else that trust decision gets made. In one task, an access token can be canceled in three places: signing out, an administrator deleting it, and deleting the account that owns it; closing two of the three still scores zero.
Trusting a defense that does not work. Something in the code looks like a security check but isn't, and the agent accepts it for what it is named rather than what it does. One task downloads a file and compares it against a short fingerprint to prove nobody tampered with it, which sounds safe until you notice the fingerprint is fetched from the same server as the file. Anyone who can alter one can alter the other, so the check proves nothing.
Over-correction. The fix works, but it breaks the product. While stopping malicious input, an agent blocks an entire category of input, and ordinary users who legitimately send something in that category can no longer use the feature.
Reasoning that stops one step early. Nothing is hidden, no defense is fake, and the agent simply doesn't follow the consequence far enough. Canceling a token works by adding it to a blocklist and dropping an entry once the token is set to expire. A token created with the “never expires” option has no expiry date, so its entry is dropped as soon as it's added, and canceling it quietly does nothing. The fix should’ve occurred where tokens are created, not where they are canceled.
What comes next
Offense and defense are not symmetric problems. An attacker needs one breakthrough, and a single unguarded surface is enough. A defender has to be right everywhere, and can only be confident a codebase is safe after accounting for every place the same mistake could live. This asymmetry makes defensive cyber capabilities strictly harder than offensive capabilities.
CWE-bench is our ongoing effort toward measuring defensive capability, and it grows with the models it measures. We are expanding along MITRE’s catalog toward broader coverage and raising difficulty as frontier models improve. For tasks we build that aren’t hard enough for the held-out set, we add them to our training corpus and use it to make agents better at the same problems. If you are interested in training your model on stronger defensive cyber capabilities or evaluating on CWE-bench, contact us at research@collinear.ai.
Citation:
@misc{cwebench2026,
title = {CWE-bench: A Defensive Cybersecurity Benchmark for Coding Agents},
author = {{Collinear AI}},
year = {2026},
howpublished = {\url{https://cwe-bench.com}},
note = {100 held-out audit-and-patch tasks across 54 CWEs.}
}