Collinear Zero: a Pareto-optimal defensive cyber agent

TL;DR

• We built Collinear Zero, an agent for defensive cybersecurity agents.

  The key idea: agents act on whatever they’re shown, so Collinear Zero’s independent tester “attacks” the generated patch, filters its findings before the agent sees them, then reverts the fix if it breaks legitimate use.

• With DeepSeek V4.1 Flash 1017, Collinear Zero lifts PatchEval success rate from 0.80 to 0.83 at only $0.15 per attempt, and CWE-bench v1 from 0.66 to 0.69 at $0.21 per attempt.

• We are open-sourcing the agent and its tools as Collinear Zero, installable into OpenCode, Claude Code and Codex (evaluated in OpenCode):

  Repo link: https://github.com/collinear-ai/collinear-zero/tree/main

  One-click install: curl -fsSL https://raw.githubusercontent.com/collinear-ai/collinear-zero/main/install.sh | bash

What makes a defensive cybersecurity task?

A defensive audit-then-patch task gives the agent a repository and a description of a weakness. The agent must find it and patch it. A hidden verifier then runs two kinds of input against the final code:

• exploit inputs, including variants of the attack (other encodings, other entry points, other input forms), which must now be refused;

• legitimate-use inputs, which must keep working exactly as before.

A patch passes only if both hold, so “block everything” fails as surely as “block one case”.

What data do we develop on? We developed the agent on 100 tasks sampled randomly from our training distribution, near-disjoint from the evaluation suites (see the overlap analysis in the Appendix).

What do we evaluate on? We report results on CWE-bench v1 (120 tasks with 73 distinct CWEs) and PatchEval (230 real CVEs in Go, JavaScript and Python, graded by each CVE’s own exploit and unit tests).

Motivations

Key observations from our 100-task development set shaped the design.

1. The agent’s own verification is blind to its failures: 98% of failing attempts end with the agent reporting a successful self-check, the same rate as passing ones. Its tests encode its own reading of the task, so they miss the same variants its fix misses.

2. A tester the agent can call is not enough: When the agent could call a review subagent and read its whole report, its changes fixed 7 attempts and broke 8: no net gain. The report mixed real problems with wrong ones, and the agent acted on both. So Collinear Zero runs the tester independently and after the agent finishes patching, and forwards only actionable findings.

3. Even correct findings can be over-fixed: With that filter in place, the loop still broke as many attempts as it fixed (6 and 6). The agent closed real bypasses so aggressively that legitimate inputs were refused, such as a legitimate redirect. Filtering can’t catch this, because the finding was right. So after the fix round, Collinear Zero re-checks legitimate inputs and reverts the fix if any now fail.

How Collinear Zero works?

Figure 1: Workflow illustration of how Collinear Zero agent operates.

1. The agent patches, with rules and a skill.

   Skills adapted from codex-security and the 15 weakness-class guides from deepsec (the repository’s NOTICE file lists every change)

   1. Rules: fail closed, the task outranks existing tests, handle every input form the task names, never weaken a check after it ran.

   2. Security-patch skill: map the description to code, find every path to it, list the attack variants, fix, prove it, review the diff.

   3. 15 short guides, one per weakness class (injection, SSRF, path handling, crypto, deserialization, …).

2. An independent tester attacks the patch.

   When the agent finishes, the agent saves a checkpoint and starts a fresh session. The tester sees only the task instructions and a copy of the original repository. It then finds the patch by diffing, checks each requirement on both the original and the patched code, then tries variants of the attack. Each finding is one line: a bypass, a regression (with a backing: file showing the behaviour is expected), or a requirement it could not test.

3. A simple filter decides what the agent sees.

   Because models, even strong ones, can hallucinate or decide inconsistently, this step is a simple parsing script, not a model. Bypasses and untested requirements always go through. A regression goes through only if its backing: file really exists in the repository (a test, doc or example), so the tester cannot invent requirements. Everything else is dropped.

4. The agent fixes once, and a recheck can undo it.

   The agent fixes the forwarded findings, minimally and without changing function signatures. The tester then re-runs its legitimate-use checks; if any now fail, the agent reverts to the checkpoint. The number of fix rounds is configurable; every result in this post uses one.

By default the agent and the tester are the same model. With Opus 5.5 as the agent, we use DeepSeek V4.1 Flash 1017 as the tester: it is cheap, effective, and never refused to write attack inputs (see the Appendix).

Results of Collinear Zero

We evaluate our agent with a cheap open-weight model (DeepSeek V4.1 Flash 1017) and a frontier model (Opus 5.5), alongside other agents. Results are shown in Figure 2.

Figure 2: CWE-bench v1 pass@1 (average of 4 attempts) vs cost per attempt.

Looking closer at the results for DeepSeek V4.1 Flash 1017 and Opus 5.5 (Table 1):

• With DeepSeek V4.1 Flash 1017, Collinear Zero makes a single attempt more likely to succeed (pass@1), but rarely solves a task that plain OpenCode never solves in four attempts (pass@4, solved at least once in 4 attempts: +0.009 on PatchEval, −0.017 on CWE-bench v1).

• With Opus 5.5, it also raises pass@4 (+0.017 on PatchEval, +0.058 on CWE-bench v1), so it solves tasks where plain Opus never solved. One likely reason is that the two models know different things: on CWE-bench v1, plain DeepSeek solved 11 tasks that plain Opus never did, and Opus solved 10 that DeepSeek never did. We hypothesize that the DeepSeek tester brings findings Opus would not reach on its own.

Table 1: The main results: each row compares plain OpenCode with Collinear Zero on the same model (plain → Collinear Zero). Δ is the paired difference over the same tasks, with its 95% CI. With Opus 5.5, the attack-tester is DeepSeek V4.1 Flash 1017.

Cost vs accuracy (the Pareto frontier)

Long-horizon tasks like CWE-bench v1 and PatchEval take many agent steps, so inference cost adds up quickly. Figure 2 plots effectiveness vs. cost per attempt to show the trade-off.

• A cheap model with Collinear Zero matches a frontier model. On CWE-bench v1, DeepSeek V4.1 Flash 1017 + Collinear Zero matches plain Opus 5.5 at about 5× lower cost per attempt.

• Collinear Zero also lifts the frontier model. Opus 5.5 + Collinear Zero has the highest scores we know of on both benchmarks (0.729 on CWE-bench v1, 0.901 on PatchEval), at 1.5–3.6× the cost of plain Opus 5.5.

Comparison with open-source cybersecurity harnesses

Figure 3: Harness performance comparison on CWE-bench v1 with pass@1. Collinear Zero scores highest (0.694; its +3.6-point lift is not significant), ahead of plain OpenCode (0.658) and the patching procedures of codex-security (0.654) and Anthropic’s defending-code-reference-harness (0.600).

We ported two open-source defensive harnesses, OpenAI’s codex-security and Anthropic’s defending-code-reference-harness, into OpenCode and ran them on CWE-bench v1 with DeepSeek V4.1 Flash 1017. Neither improved pass@1 over plain OpenCode: codex-security was level (−0.4 points) and defending-code-reference-harness was 5.8 points lower. These results reflect our ports, not the original products; the Appendix describes each port. Collinear Zero adapts its patch skill from codex-security, so the review loop is the main difference between the two.

Ablation and Analysis

A weaker model: Nemotron 3.5 Super VL as agent and tester

To check that Collinear Zero does not depend on a strong model, we ran it on CWE-bench v1 with the open-source model Nemotron 3.5 Super VL as both the main agent and the attack-tester, so no stronger model is anywhere in the loop. The overall gain (+0.033) is not significant: a significant gain from the review loop is partly offset by a slightly weaker first patch (analysed below).

Table 2: Results of Nemotron 3.5 Super VL on CWE-bench v1 with and without Collinear Zero.

How did the “attacker” help?

We count the patches the review loop flipped (fix ↔ break) on CWE-bench v1, with the same three agents as above.

To see what the review loop itself did, we replay each attempt’s patch (from its transcript) in a fresh container, cut before the review, grade that tree, and compare it with the final grade. Because the attack-tester runs only after the main agent finishes its patch, the patch before review is the harness with its rules and skill alone; comparing it with the final patch isolates what the review loop adds.

Table 3: break down comparison analysis of what attacker-flagged patches flip into.

• The gain comes from the review loop, not the first patch: with DeepSeek V4.1 Flash 1017 as agent and tester, Collinear Zero’s patches before review (rules and skill alone) score 0.667, level with plain OpenCode’s 0.658 (+0.008). The review lifts them to 0.694: 22 patches fixed, 9 broken, a loop effect of +0.027.

• A stronger main agent leaves the loop less to do: with Opus 5.5 as the main agent, the patch before review already scores 0.719, against 0.671 for plain OpenCode Opus 5.5: for Opus, the rules and skill alone add about 5 points. The loop fixes 15 and breaks 10, for +0.010, a smaller gain than with DeepSeek and not significant.

• With Nemotron, the gain comes entirely from the review loop: Before review, Nemotron’s patches score 0.412, level with plain OpenCode (−0.016), so the rules and skill alone do not help this model. The loop then fixes 24 patches and breaks 1, a loop effect of +0.054 over the 451 attempts that replay faithfully. That is significant, and the largest of the three main agents (DeepSeek V4.1 Flash 1017 +0.027, Opus 5.5 +0.010): the weaker the agent, the more the tester’s filtered findings have to fix.

• A weak tester’s findings stay safe to act on: With Nemotron also writing the attacks, the filtered findings almost never broke a correct patch (1 of 25 flips), and the recheck reverted 16 fix rounds, saving 2 patches and losing none.

What the loop fixed and what it broke

We read every attempt the loop flipped across the three agents (61 fixed, 20 broken) and sorted each by what happened. For Nemotron 3.5 Super VL we use the 451 of 480 attempts that replay faithfully; for the other two agents, all 480 do. Below is our analysis on the successful replays.

Table 4: detailed class breakdown of fixed/broken patches.

• Fixes: the tester finds holes the first patch left open: 54 of 61 fixes closed an attack that already worked on the original code, either at the code the agent had changed or on a path or input form it missed. 4 more undid a regression the first patch introduced, and in 3 Opus attempts the tester’s report got an agent that had changed nothing to write the fix. Weaker agents leave more holes: 19 of Nemotron’s 24 fixes covered a path its first patch missed entirely.

• Breaks: mostly the agent trusting a wrong finding: 9 of the 20 breaks came from findings the agent should have questioned, such as a “bypass” that asked to restore weaker security behaviour, or a value the task lists as public being treated as secret. Nemotron’s tester produced none of these. 2 more were correct findings fixed too broadly (a server that hung instead of refusing early), and 1 was an existing check the agent dropped while editing.

• 8 breaks are verifier quirks, not bad patches. The fix was reasonable but changed something the CWE-bench v1 verifier checks literally, such as a function signature or an exact output format. About 10 fixes may likewise owe something to a lenient check. With these suspect cases removed from both sides, the loop still fixes more than it breaks: 19 vs 5 with DeepSeek V4.1 Flash 1017, 11 vs 6 with Opus 5.5, and 21 vs 1 with Nemotron 3.5 Super VL. We are tightening these verifiers for the next CWE-bench release.

What didn’t work

Unless noted, we ran these on our 100-task development set, mostly with DeepSeek V4.1 Flash (not the 1017 version).

• An enforced checklist gate (a requirement ledgerCyber agent harness blog review plus a check script that must pass before the agent can finish): Δ pass@1 −0.026 (not significant) on the 35 development tasks DeepSeek solves only sometimes, about −0.9 points over the whole set.

  • The gate almost never fired. DeepSeek wrote a complete ledger and passing checks on its own in 98% of attempts, and only 3 attempts were sent back.

  • The checks encoded the agent’s own reading of the task, so they passed even where the agent had missed the vulnerable code.

  • It also added 14% to cost and 34% to agent time.

• Static-analysis hints: we added findings from open-source static analyzers (Semgrep, CodeQL) to the agent’s context.

  • The locations were barely better than chance. The top-10 Semgrep findings fell in a file the fix changes 15.5% of the time, against 12.6% for a random source file. About 3% of hint lines were within 5 lines of the real fix. CodeQL raised no alerts on 49% of tasks. So adding their findings to the agent’s context adds more noise than signal.

  • On CWE-bench v1, the same setup with DeepSeek V4.1 Flash gave Δ pass@1 −0.023 (not significant).

Takeaway: neither change gave the agent anything it could use to choose a better fix. The gate re-checked the agent’s own reading of the task, and the hints pointed at the fix little better than chance (noisy context from the tool).

Limitations

• Collinear Zero costs ~1.5× to ~3.8× as much as plain OpenCode per attempt, depending on the model and benchmark.

• We sample at each model’s default temperature for optimal performance, so individual attempts vary; we report the average of 4 attempts per task with task-level 95% CIs.

• We evaluated Collinear Zero only in OpenCode.

• The recheck catches few breaks in practice: on CWE-bench v1 it passed 19 of the 20 patches the loop broke, because it re-runs only the tester’s own legitimate-use inputs. We keep it as a cheap guard; with Nemotron 3.5 Super VL it reverted 16 fix rounds, saving 2 patches and losing none.

• On CWE-bench v1, we only use programmatic verifiers for evaluation — no LLM judge was used, to reduce variance in the results.

Disclaimer

• Collinear Zero is a defensive tool. It audits and patches a weakness that a report already describes, in code you own or are authorized to change. It is not designed or evaluated for finding new vulnerabilities in third-party systems, or for any offensive use.

• The attack-tester writes and runs inputs only against the agent’s own patch. In our evaluations, every agent and tester ran inside the task’s network-isolated container. Because the tester executes code it writes, run Collinear Zero in a sandbox without network access.

• Patches are proposals. Review every change before merging it: passing these benchmarks does not guarantee that a patch is complete or free of regressions.

• Our results reflect the models, providers, versions and dates listed in the Appendix, and may not transfer to other setups.

• The released GitHub repository’s NOTICE file lists the projects Collinear Zero adapts and every change. Collinear Zero is not affiliated with OpenAI, Vercel or Anthropic.

Citation

@misc{collinear2026zero,
  title        = {Collinear Zero: A Pareto-Optimal Defensive Cyber Agent},
  author       = {{Collinear AI}},
  year         = {2026},
  month        = oct,
  howpublished = {\url{<https://github.com/collinear-ai/collinear-zero>}},
  note         = {Blog post and open-source release}
}

Appendix

Technical details on how we evaluate codex-security and defending-code-reference-harness

We ported each tool’s prompts into OpenCode instead of running it as shipped. Neither could run as-is in our offline evaluation containers, and even if they could, running them natively would compare different agents and models rather than different methods.

Why codex-security can’t run natively here

• It drives the bundled Codex CLI, which only speaks OpenAI’s Responses API. Our usage-logging relay only speaks chat completions, so it couldn’t reach DeepSeek through our pinned, cost-logged path without new relay work.

• Every task image would need Node 22, the npm packages and Codex preinstalled, because the containers have no network.

• Its full flow is built around a scan workbench, an MCP server and artifact storage. Only the fix-finding skill is the patch method itself.

Why defending-code-reference-harness can’t run natively here

• It orchestrates the Claude CLI from the host, starting a gVisor-sandboxed Docker container per agent and committing images between stages. It can’t run inside a task container, and it only talks to Anthropic endpoints, not an OpenAI-compatible relay.

• Its patch pipeline is specific to C/C++ and AddressSanitizer, and it expects a crash proof-of-concept as input. Our tasks give only a text description of the weakness.

• Its static /patch path never applies the fix; it only emits a diff for a human. Our verifiers grade the final tree, so something had to apply the patch.

Why porting is the fairer comparison

• Running each tool natively would change the agent (Codex or Claude Code instead of OpenCode) and the model along with the method, so a difference in results couldn’t be attributed to the method.

• Porting keeps everything else identical to the plain-OpenCode and Collinear Zero runs: same model (DeepSeek V4.1 Flash 1017), task prompt, offline network, time cap and concurrency. Only the harness differs.

• To stay faithful, we copied their prompts verbatim and changed only what couldn’t work here. Each change is listed in the profile’s NOTICE.md.

What we ported

• codex-security: we ported the relevant skill, prompts and subagents from the codex-security repository. This port is not OpenAI’s Codex Security product and is not affiliated with OpenAI. Specifically, we ported:

  • The fix-finding skill: config/skills/fix-finding/SKILL.md is plugins/codex-security/skills/fix-finding/SKILL.md.

  • The per-request instructions: config/AGENTS.md holds the instructions the codex-security SDK sends with every patch request (sdk/typescript/src/api.ts, around line 1044, at commit f9b5c2e3).

  • Two subagents: config/agents/boundary-investigator.md and config/agents/bypass-reviewer.md, whose assignments are copied from the skill.

  • We made minor prompt updates to the original SKILL.md files to adapt them to our evaluation settings, such as removing the section “Workbench Remediation Stages” in the SKILL.md file since the Generate/Apply/Verify stages are driven by the workbench (scan IDs, action tokens) in the codex-security framework.

• defending-code-reference-harness (at commit d3bea6b5): similar to how we evaluate codex-security:

  • config/AGENTS.md: the pipeline’s patch-agent prompt (harness/prompts/patch_prompt.py: ENV_HEADER, FULL_TASK, RETRY_SECTION), merged with the static-mode patch procedure in .claude/skills/patch/SKILL.md (read the code, do not trust the description alone, match the surrounding style, add one regression test).

  • config/agents/patch-reviewer.md: the static-mode independent reviewer from .claude/skills/patch/SKILL.md. It sees only the location, the category and the diff, answers four questions, and ends with REVIEW: ACCEPT|REJECT. On REVIEW: REJECT, the review counts as failed and the patch already applied is left in place.

  • Instructions in the prompt were updated to align with the requirements of audit-then-patch task evaluation.

Overlap between the development set and the test sets

We compared the 100 development tasks with CWE-bench v1’s 120 test tasks by task, source repository, weakness class, language and wording. No task is shared and no task description is reused. Of the roughly 100 source repositories behind the test set, only 2 repositories also appear in the development set: fast-jwt (1 test task) and xmldom (3 test tasks), each with a single development task. No pair of repository and CWE is shared. Removing those 4 test tasks changes Collinear Zero’s paired gain over plain OpenCode little: +0.037 instead of +0.036 with DeepSeek V4.1 Flash 1017, +0.060 instead of +0.058 with Opus 5.5, and +0.041 instead of +0.033 with Nemotron 3.5 Super VL. The separation is not a side effect of language mix: every test-set language also occurs in the development set, and the shared repositories are all JavaScript/TypeScript.

Against PatchEval, the development set shares 1 of 230 CVEs and 6 repositories. Dropping the 9 PatchEval CVEs on those repositories leaves Collinear Zero’s PatchEval gain at +0.029 [+0.002, +0.059], against +0.032 on all 230.

Evaluation setup

• Agent: OpenCode 1.18.31 in every run, fully offline: each task runs in a container on an internal network whose only reachable host is our usage-logging relay. OpenCode’s web tools are disabled.

• Inference: every call goes through OpenRouter, pinned to a single provider with fallbacks disabled. DeepSeek V4.1 Flash 1017 runs on DigitalOcean with the provider’s default sampling and reasoning settings. Opus 5.5 runs on Anthropic with reasoning effort medium. Nemotron 3.5 Super VL runs on Together with the provider’s default settings.

• Limits: a context window of 1,048,576 tokens for DeepSeek V4.1 Flash 1017, 1,000,000 for Opus 5.5 and 240,000 for Nemotron 3.5 Super VL (Together counts the input plus the full output budget against its 262,144-token limit); 32,000 output tokens per call; 16 attempts run concurrently per configuration.

• Time caps: 3,600 s per attempt on CWE-bench v1 and 2,400 s on PatchEval. With Collinear Zero, the last 900 s of the cap are reserved for the review, so the agent’s own phase ends 900 s early.

• Cost: $ per attempt is the cost OpenRouter billed for each call, logged by our relay and summed per attempt, for runs on 23-28 October 2026. List prices then were $0.30 / $1.20 per million input / output tokens ($0.006 cached input) for DeepSeek V4.1 Flash 1017, $4 / $20 ($0.20 cached input) for Opus 5.5, and $0.15 / $0.50 ($0.02 cached input) for Nemotron 3.5 Super VL.

Tester refusals

We use DeepSeek V4.1 Flash 1017 as the attack-tester because it does the job without refusing:

• Provider content filter: none of its calls was stopped by the content filter: about 219,000 calls across our Collinear Zero runs, as tester and as main agent, on CWE-bench v1, PatchEval and the development set (20,547 of them as the tester for Opus 5.5 on CWE-bench v1 and 36,608 on PatchEval).

• Refusals in its reports: none of its 1,160 tester reports (480 each on CWE-bench v1 with DeepSeek V4.1 Flash 1017 and Opus 5.5 as main agents, and 200 on the development set) contains an explicit refusal. One CWE-bench v1 review ended without a report, which was not a refusal.

• Opus 5.5 for comparison: in an earlier version of the harness on the development set, with Opus 5.5 as the tester, the content filter blocked 27 of 100 tester calls. As the main agent in Collinear Zero, Opus 5.5 hit the content filter on 13 calls in 10 of 480 CWE-bench v1 attempts and on 6 calls on PatchEval; none of these ended an attempt.