If you have ever run evaluations, this would be familiar.
💡 Ten scored attempts from three model families all closed the vulnerability in a cybersecurity task, yet five received a reward of zero.
Those five were not random failures. Each had hardened the authorization path further than the passing solutions did, a choice the written task did not forbid and the verifier silently punished.
Those five were not random failures. Each had hardened the authorization path further than the passing solutions did, a choice the written task did not forbid and the verifier silently punished.
We found the opposite failure as well. We constructed a solution that left part of the attack surface exploitable and the deterministic harness gave it a perfect score. No model discovered this shortcut. We built it to learn what our benchmark was. The lesson we learned was that the harness could reject secure solutions and accept an insecure one and neither mistake leaves a mark on the number it returns. That would be a measurement error if the harness only measured. In our training loop the score is also the reward and an uncaught harness mistake becomes part of the objective.
An agentic benchmark has three layers that each have to be right: an environment that exposes the world we mean to study, a task that requires the capability we claim to measure and a verifier that can tell success from failure inside that task. A defect at any of the three still produces a clean number. This series covers all three, starting with the verifier because every trajectory eventually passes through one.
When the instrument starts shaping behavior
Evaluations measure where a model succeeds and fails and post-training shapes the model using data and rewards chosen partly in response to those measurements. Increasingly the same definition of success sits on both sides of that loop: a test suite, a rubric, a state check, or a judge first tells us whether a model did the right thing and later decides which of its behaviors will be reinforced. Once the verifier shapes behavior as well as reading it, a defect in it is no longer only a measurement problem.
Reward models already show what the loop does with a defect: as optimization pressure rises, performance against a proxy keeps improving after performance on the underlying objective has begun to fall [1]. Medicine offers a precedent. Trastuzumab’s usefulness arrived bundled with the HER2 assay that chose its patients, so whatever the assay got wrong, the treatment inherited [2] and AI can take that lesson from there instead of relearning it one benchmark at a time.
The unit of data changed
For most of the data-labeling industry’s history, the deliverable was a judgment such as a box around a car or a choice between two responses and because a second person could make the same call independently, the industry had a workable language for quality built on agreement.
Agentic training data is a different kind of object: a deliverable may now include a container image, application state, credentials, tools, task instructions, reset logic, hidden tests, a rubric and code that converts a trajectory into a score, so the judgment is no longer a label attached to the data. In short, it is a program that runs against it.
Cobbe and colleagues trained verifiers to rerank candidate math solutions [3], Tülu 3 then trained directly on verifiable rewards [4], DeepSeek-R1 pushed rule-based rewards deep into reasoning training [5] and the resulting discipline has been named verifier engineering [6]. That is the sense in which verifier quality has become a data-quality problem.
A bad label corrupts one example, whereas a defective verifier mis-scores every trajectory that passes through the same blind spot and once the score becomes reward, the blind spot becomes part of the objective. We had conventions for asking whether two annotators agreed and nothing settled yet for asking whether an executable verifier enforces the contract we intended.
The benchmark has an attack surface
A verifier is full of assumptions. Some are stated in the written task, while others live only in the harness, the reference solution, the judge prompt, or the particular state transitions the evaluator happens to inspect and an optimizing agent encounters every one of them through the single mechanism that produces reward. In an earlier post on this blog we argued that an agent’s environment should be scored the way security engineers score a system, by its attack surface, with the agent cast as an attacker who is already inside [7]. The score multiplies three terms: whether the answer is reachable somewhere in the environment, whether the agent’s tools can reach it and whether the grader checks only the finished artifact. It explains a good deal of recent benchmark history, from the July incident in which models escaped an evaluation sandbox and went looking for the reference solutions [8], to Cursor’s finding that SWE-bench Pro scores fell by 14 to 21 points once future git history was sealed and network access restricted [9]. The cases that motivate that score, though, are agents going around the grader.
A grader that inspects the artifact carefully still has a surface of its own, because it can be satisfied by an artifact that has the properties it inspects and lacks the one that matters and it can refuse an artifact that has the property but breaks a contract nobody wrote down. Zhu and colleagues applied a validity checklist to ten widely used agentic benchmarks, finding seven with task-validity problems and seven with outcome-validity problems. In one, a cybersecurity benchmark checked for time-based SQL injection by looking for a SLEEP clause in the database log, so any query containing the word passed [10]. The grader checked the wrong thing and the shortcut lived inside its own logic.
One task, two directions of failure
Our task comes from one of the cybersecurity benchmarks we are building. The agent receives a real codebase with a planted flaw and has to close it without breaking normal application behavior. The application keeps identity and authorization state in browser cookies, several of which carry security decisions and the original code cannot establish that their values were vouched for by the server. We grade the repair with a deterministic harness that forges cookies against the running application and then checks that ordinary behavior still works and with an LLM judge that works through twenty-one rubric criteria. In practice they exposed three separate problems.
A false accept
The exploit forges one cookie and checks whether its integrity depends on a server-held secret rather than on something the client can reproduce, but it never attempts the other security-relevant cookies. We protected those with a plain digest of the payload and no server secret, which any attacker can recompute and because the exploit never tried them, the harness reported that all cookie-borne security decisions require server-vouched integrity and scored the submission 1.0. It checked one instance of the property and generalized across a boundary it never tested. The LLM-as-judge catches this one, because a criterion requires a server-held secret and the judge notices the unkeyed construction.
A false reject
Across ten scored attempts from three model families, none left the original vulnerability open and five still received zero because they made a more defensive change than the passing solutions did. The original privileged gate honors an entitlement without first establishing an authenticated caller; three agents changed the gate to resolve a signed-in principal before honoring the entitlement, two others passed the principal in as a parameter and both designs close anonymous privileged access.
The first three were charged with a regression because a valid entitlement no longer passed two privileged gates. The benign probe that decided this calls each gate with an entitlement cookie and no identity, so it treats preservation of the weak authorization contract as legitimate behavior when tightening that contract is a sensible response to the vulnerability. The other two hit a TypeError for a missing positional argument, because they had changed an internal function signature and updated every caller. The instruction prohibits changing the externally observable API, which they preserved, but the verifier silently reads that clause as covering an internal function’s positional signature too. The benchmark is enforcing one contract nobody wrote down and one the instruction left loose and both select against the more defensive fix.
When two verifiers become one
On the constructed insecure solution the judge overrides the passing harness and names the missing secret. On the hardening attempts it leans on the harness instead and the failed rubric criteria repeatedly cite the “authoritative harness” rather than deciding independently whether the security property was met. The reason is structural. On the cheat, the rubric held a criterion with evidence of its own to consult, namely whether a secret was present in the code; on the regressions, no criterion had any evidence about legitimate behavior except the harness output, so deferring was the only move available. A judge that treats another verifier’s decision as evidence turns two measurements into one.
Accuracy is the wrong headline number
Huang and colleagues make this point with a direct comparison. Testing verifiers for mathematical reasoning, they found that open-source rule-based checkers fail to recognize equivalent answers in unfamiliar formats, producing false-negative rates high enough to hurt RL training, with damage that grows as the policy gets stronger. Model-based verifiers scored markedly higher in static accuracy and under RL they were hacked: the policy learned patterns the verifier misclassified as correct, especially once the verifier itself had been fine-tuned and the reward inflated without the accuracy to match [11]. The verifier with the better accuracy number was the one an optimizer could exploit.
The difference lies in the structure of the error, not its amount. Error at a fixed rate is cheap: corrupting up to 15 percent of RLVR rewards, with both synthetic and model-based noise, left peak validation accuracy within two points of a clean verifier [12]. That noise is exogenous, held at its rate whatever the policy does. A verifier’s blind spot is not, because the policy can raise its own exposure by steering into it. A stable false accept pays gradient for something the task never asked for and the policy drifts toward it until the shortcut is part of what the model does. The reject case runs the other way: gradient is withheld from a solution the task did not forbid, so the objective loses a goal and the policy learns to steer clear of a safe design. The two graders sort along this line. A rule-based checker tests a final answer against a reference, so it recognizes only the correctness its author foresaw and rejects the rest. A model-based judge can be reasoned with, so it can also be gamed, which is where false accepts come from. A single accuracy number averages the two failures and hides which one a verifier is prone to and that is the distinction training turns on.
What verifier QA should look like
The first controls are cheap: run a known-good solution and confirm that it passes, then run a deliberately worthless one and confirm that it fails and in a security benchmark make sure the worthless one includes a patch that treats the symptom and leaves the property broken. A passing reference solution is weak evidence on its own, because it says nothing about whether other valid implementations have been excluded or whether an invalid one can mimic the surface properties the tests inspect.
Software testing already has the right instincts. Mutation testing evaluates a test suite by seeding faults into the program and asking how many the suite catches [13] and metamorphic testing checks a program by transforming an input and requiring the output to change in a way the specification predicts, which yields a partial oracle where no reference answer exists [14]. Pointed at the scoring mechanism rather than at the code under test, those instincts yield a procedure. Build several valid implementations rather than one canonical answer, including some that rearrange the internals while preserving the promised external behavior and invalid ones that violate the target property in different ways. Run the exploit for each claimed security property everywhere the property is supposed to hold and where multiple evaluators are involved, test whether they supply independent evidence or echo one another. Then report failure modes rather than a single score, because a verifier that occasionally rejects an unusual valid implementation is a different risk from one that systematically rewards a shortcut.
Hasok Chang’s history of thermometry makes the general point that even simple measurements became trustworthy only after experimenters specified conditions and built procedures around the instrument [15]. For executable verifiers the methods will look more like software assurance than metrology, but an instrument does not become trustworthy by returning a number.
Why cybersecurity raises the stakes
A bad verifier in a benchmark costs a measurement and in a training loop it costs more. The optimizer is doing its job on the objective it was given and with executable rewards, whatever the verifier consistently rewards or punishes becomes part of that objective. When Cursor reran SWE-bench Pro in a harness with git history sealed and network access restricted, an older model’s score moved by less than a point while the two newest models they tested dropped 14 and 21 points and an auditing agent classified 63 percent of the highest-scoring model’s successful resolutions as retrieved rather than derived [9]. The GPT models in the same study showed smaller gaps and revealed a more general principle: the stronger the optimizer, the larger the share of its score that comes from what the environment leaked rather than from the task.
Cybersecurity makes that especially uncomfortable, because we are deliberately training agents to search complex systems, find weak assumptions and achieve objectives under constraints, which are exactly the capabilities needed to discover unintended success conditions in a verifier. Superficial features can already steer LLM judges [16] and deterministic harnesses fail through hidden contracts and incomplete state checks. In either case the evaluator is part of the system an optimizing agent encounters, which is why the verifier belongs in the threat model.
The next question
None of this makes verifier QA sufficient, because a perfectly implemented verifier can faithfully score a bad task and a task can be hard for reasons unrelated to the capability we meant to measure. That is the next layer of the attack surface. Once we can trust the reading, we have to ask what the instrument is reading: what makes a task hard and whether success requires the capability the benchmark claims to test.
References
[1] Gao, L., Schulman, J. and Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML (2023). arXiv:2210.10760.
[2] Slamon, D. J. et al. Use of chemotherapy plus a monoclonal antibody against HER2 for metastatic breast cancer that overexpresses HER2. New England Journal of Medicine 344, 783–792 (2001).
[3] Cobbe, K. et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 (2021).
[4] Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 (2024).
[5] Guo, D. et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 633–638 (2025). arXiv:2501.12948.
[6] Guan, X. et al. Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering. arXiv:2411.11504 (2024).
[7] Rajani, N. The Simulated World Is Being Born in Cybersecurity. Collinear AI’s Blog (August 2026). https://blog.collinear.ai/p/cybersecurity-simulated-worlds-agi
[8] OpenAI. OpenAI and Hugging Face partner to address security incident during model evaluation. OpenAI (2026). https://openai.com/index/hugging-face-model-evaluation-security-incident/
[9] Jain, N. Reward Hacking Is Swamping Model Intelligence Gains. Cursor (June 2026). https://cursor.com/blog/reward-hacking-coding-benchmarks
[10] Zhu, Y. et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. NeurIPS Datasets and Benchmarks Track (2025). arXiv:2507.02825.
[11] Huang, Y., Zeng, W., Zeng, X., Zhu, Q. and He, J. From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning. arXiv:2505.22203 (2025).
[12] Plesner, A., Guzmán, F. and Athalye, A. An Imperfect Verifier is Good Enough: Learning with Noisy Rewards. arXiv:2604.07666 (2026).
[13] Jia, Y. and Harman, M. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5, 649–678 (2011). doi:10.1109/TSE.2010.62.
[14] Segura, S., Fraser, G., Sánchez, A. B. and Ruiz-Cortés, A. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9, 805–824 (2016). doi:10.1109/TSE.2016.2532875.
[15] Chang, H. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press (2004).
[16] Zhao, Y. et al. One Token to Fool LLM-as-a-Judge. arXiv:2507.08794 (2025).