The agent had a vulnerability to patch. In a gateway connecting an application to AI services, some administrative routes checked who was asking for access; others let requests through without authentication. The assignment was to secure those routes while leaving the application’s legitimate uses intact. One of our highest-performing models tried four times and failed every time.
The model had chosen a sensible design: require authentication by default, then make explicit exceptions for routes that needed to remain public. Its mistake was in the exceptions. The patch stopped the tested attacks, but it also shut out legitimate requests, and the regression tests caught the damage. Models that performed worse across the benchmark passed this task in three of four attempts.
A strong model can overlook a real requirement, and finding the affected endpoints belongs to the work of repairing access control. The reversal was no reason to throw out the task. But it prompted us to look more closely at who was succeeding. This task’s overall pass rate was 42 percent; another task’s was 43 percent. Behind those nearly identical numbers were strikingly different relationships with performance elsewhere in the benchmark.
The number in the metadata
Cybersecurity benchmarks have several ways of describing difficulty. Cybench, introduced in 2024, assembled forty professional capture-the-flag challenges from four competitions and reported how long the first human team took to solve each. Its evaluated agents completed tasks with human first-solve times of up to eleven minutes; the longest first-solve time in the collection was almost twenty-five hours. These measurements provide useful context, but a human competition’s first-solve time is not a calibrated measure of difficulty for a language-model agent [1].
A pass rate measures the evaluated agents more directly, but it too depends on the conditions of measurement. It changes with the models included, their scaffolds, the available budget and the scoring rule. In one study, eight H100 GPU-hours of iterative improvement increased an agent’s InterCode CTF performance by more than 40 percent relative to its baseline. Changing how a model is elicited can substantially change how difficult the same tasks appear [2].
Part one of this series examined the verifier and the environment around it. In the gateway case, the tests did their job: they caught a patch that broke required behaviour. Yet a correctly recorded failure still leaves a question for anyone reading the leaderboard. How much should this particular mistake change our judgment of the model that made it?
Educational measurement offers a way to begin answering that question. A test question has a history of right and wrong answers, but it also has a pattern: students who do well elsewhere may reliably answer it correctly, or their broader performance may offer little guidance. Item response theory describes that relationship with a curve linking ability to the probability of success. In its two-parameter logistic form, the curve has a location and a steepness, called difficulty and discrimination [3].
Put ability along the horizontal axis and probability of passing on the vertical. Difficulty is the ability at which the fitted probability reaches one half. Discrimination controls how rapidly that probability changes around the midpoint. Two tasks can have similar average pass rates while tracing very different curves over the models that attempted them. The plot below writes difficulty as b and discrimination as a, and the slope of the curve at the midpoint is a/4 rather than a itself.
For an item with difficulty b and discrimination a, the model is
Here θ is the model’s ability that Figure 1 puts along the horizontal axis. Ability and difficulty enter only as the gap θ − b, and a fixes how sharply the probability turns as that gap changes sign.
Fig 1. Difficulty marks the ability at which the fitted probability of passing reaches 50 percent. Discrimination controls the steepness of the transition. An aggregate pass rate averages success over the models evaluated it does not reveal the shape of this relationship. The curve is schematic.
The word “ability” carries more authority than the fit alone can give it. Here it summarises a pattern in performance across the task bank. A steep curve tells us that a task follows that pattern closely; it cannot tell us whether the pattern deserves to be called security expertise. That interpretation has to earn its standing through evidence about the tasks and the behaviour that produces success. Work applying item response theory to language benchmarks gives us tools for the investigation [4, 5].
Two similar pass rates
We fitted these curves to security tasks that ask agents to repair vulnerable software without breaking legitimate uses. The evaluations span five agent scaffolds, so a fitted ability belongs to a model and the harness around it rather than to the model alone. Every comparison below is between models as we evaluated them, under those particular conditions.
The gateway task (Task 1) had a fitted discrimination of 0.16 (Fig. 2). Repair required identifying which routes must remain public when authentication became the default. One route repeatedly omitted by the high-performing model was /api/init, a cloud-sync initialiser absent from the instructions but present in the API directory. The trace review found no callers elsewhere in the repository. Successful patches from lower-performing models included it in their allow-lists.
The traces suggest that success may depend on an inspection habit weakly associated with the abilities rewarded elsewhere in the bank. That could reflect an underspecified task, or a valuable skill our other tasks neglect. Spelling out the public-route contract and rerunning the same models with the same budgets would help test this explanation. The fitted curve identifies a discrepancy; explaining it requires further work.
Figure 2. Tasks 1 and 2 pass at 42 and 43 percent on average yet discriminate very differently, a = 0.16 against a = 1.87. Task 3 discriminates most at a = 2.30 but is nearly saturated. Marks are observed success fractions sized by attempts. The fit is exploratory.
The second task concerned a key-value server whose sorting commands could bypass per-connection permissions through indirect key references. Its tests required the patched server to block unauthorised access while preserving legitimate requests. Its pass rate was 43 percent, but its fitted discrimination was 1.87 (Fig. 2). Every model with fitted ability above 0.7 passed every recorded attempt; below that level, all but one failed most attempts.
Within this sample, the second task looks more useful for ranking models along the fitted scale. Its outcomes follow the bank’s overall ordering much more closely, although that does not establish which skills explain the separation. The similar pass rates conceal a substantial difference in what the tasks tell us. Refitting the curves with the twenty models resampled with replacement keeps the key-value task (Task 2) steeper in 98 percent of those refits, which supports the ordering more than either fitted value.
Where the curve sits
A third task shows why discrimination alone is insufficient. It concerns an SQL injection in a TypeScript query builder and has fitted discrimination of 2.30, higher than either of the first two tasks. Its fitted difficulty is −2.26, however, placing the midpoint below the observed ability range. Seventeen of the nineteen models that attempted it passed every attempt and its aggregate pass rate is 96 percent.
By the time the curve reaches the stronger models in our sample, it has almost flattened out. Across the five highest-ability models, the fit predicts a spread of about twenty percentage points in success probability on the key-value task (Task 2), four points on the gateway task (Task 1) and less than a tenth of a point on the SQL-injection task (Task 3). These are model-based estimates from a small sample, but they make the distinction visible. A large discrimination parameter can coexist with negligible separation in the region where a comparison is being made.
An item’s usefulness depends on the benchmark’s purpose. The SQL-injection task does little to separate the strongest models in this sample, but could still check a minimum capability or detect a regression. A dangerous-capability task that every model passes may carry substantial safety significance despite offering no ranking power.
Claims about future models need further evidence. Only three models in our sample have fitted ability above one standardised unit, and curves extending beyond the observed range are extrapolations. Comparisons across generations also require a shared calibration: standardising each new cohort separately does not establish continuity. Further frontier evaluations would test whether the fitted relationship continues to hold.
What the curve cannot see
Some tasks in the bank have no successful recorded attempts. Those failures leave fitted difficulty and discrimination poorly determined, with finite estimates depending heavily on the priors or regularisation used in fitting. From failures alone, we cannot tell whether we have set an exceptionally difficult task or one that cannot be completed under the evaluation conditions.
We therefore returned to the positive control from part one, checking a set of ten tasks with no successful recorded attempts. We applied each maintainer’s fix to the vulnerable code and ran both versions through the verification harness. The initial control run flagged two apparent task failures, which inspection traced to defects in the harness used for the controls. After correcting those defects, all ten patched versions passed and their vulnerable counterparts failed.
We now knew something the curves could not tell us: for those ten tasks, the harness would accept a known correction and reject the original defect. Whether an agent could find that correction from the information supplied, within the time allowed, remained an open question. So did the possibility that a different, inadequate patch might slip through. The controls answered a specific question, which is what made them useful.
What a good task is
The familiar phrase in the data business “hard but fair” compresses several questions into two concepts. A task may be solvable and correctly scored while contributing little to a particular ranking. It may separate models sharply while rewarding a shortcut, or test an important capability that is only weakly represented elsewhere in the bank. Difficulty and fairness for a comparison need evidence of their own.
For benchmark builders, an unexpected response pattern should prompt investigation before exclusion. Weak discrimination may expose an ambiguous requirement, but it may also reveal a capability that the rest of the bank neglects. Strong discrimination warrants its own scrutiny: we still need to understand what successful models are doing. Removing every task that disagrees with the ranking would risk making the benchmark more consistent at the expense of what it measures.
Hasok Chang’s history of thermometry describes scientific measurement developing through epistemic iteration: an imperfect instrument can generate evidence that helps improve the system of measurement around it. Our investigation follows that modest logic. The response curves tell us where to look more closely, and the controls let us check something beyond the fit. As those checks accumulate, we can make a better case for what the scale measures and where it deserves to be trusted [6].
Benchmark reporting should make that investigation visible. Item-level outcomes, attempt counts and evaluation conditions let readers see what lies behind a score. Curves with uncertainty show where a comparison has support, while reference-control results document what the harness has been checked to accept and reject. Together, these give readers grounds for assessing the judgment embodied in the ranking.
In Task 1 the agent still missed a route that needed to remain public. Its failure belongs in the record, alongside the evidence that this task had little relationship with the ordering produced by the rest of the bank. Understanding that discrepancy may teach us something about the agent’s habits of inspection, or about the capabilities our benchmark has learned to reward. A score of 42 percent gives us a place to start. The work of metrology lies in finding out what we are entitled to conclude from it.
References
[1] Zhang, A. K. et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. ICLR (2025), first released 2024. arXiv:2408.08926.
[2] Wei, B. et al. Dynamic Risk Assessments for Offensive Cybersecurity Agents. (2025). arXiv:2505.18384.
[3] Lord, F. M. and Novick, M. R. Statistical Theories of Mental Test Scores. With contributions by Allan Birnbaum. Addison-Wesley (1968).
[4] Vania, C. et al. Comparing Test Sets with Item Response Theory. ACL-IJCNLP (2021). arXiv:2106.00840.
[5] Zhou, H. et al. Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory. AAAI (2026). arXiv:2505.15055.
[6] Chang, H. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press (2004).