• Stronger verifiers. Where our programmatic verifiers disagreed with an agentic judge panel, we read the rollouts and fixed the verifier. Some passed partial fixes; others failed valid ones.

• A tighter sandbox. Agents now run with no internet access, so they can’t look up the upstream fix. One model still found a way out, using its own API key.

• A bigger suite. 120 held-out audit-and-patch tasks, covering 73 CWEs, all ten OWASP Top 10 2025 categories and eight languages.

Most of the attention on AI in cybersecurity has gone to offense: finding vulnerabilities and writing exploits. Defense is the harder test; an attacker needs one way in while a defender needs to close every way in. As agents take on more real security work, they need to be at least as capable at defense as at offense. CWE-bench measures that side, and we’ll keep extending it as the frontier moves.

Leaderboard

Grok 4.7 and GPT-6 Astra tie for first at 68%, with Claude Opus 5.5 a point behind at 67%. With four attempts, Grok 4.7 pulls ahead, solving 81% of tasks.

Claude Opus 5.5 matches the leaders at about a quarter of the cost ($0.79 per rollout vs about $2.80). DeepSeek V4.1 Flash is the budget pick, at 55% for $0.09.

CWE-bench is also part of the Artificial Analysis Cyber Index, where Artificial Analysis runs it independently as CWE-Bench-AA. Scores are not directly comparable due to methodology differences with differing harnesses and reasoning modes.

1. Constructing an agentic judge panel

A deterministic programmatic verifier should check that (1) the exploit no longer works and (2) the project’s existing tests still pass. Those checks can still miss a bad fix; one that closes the probed code path but leaves an identical one open, blocks the exploit by disabling the feature, or breaks the code the tests don’t cover can still fool the verifier.

Since v0 we have run more than 13,000 rollouts. To find where the verifier was wrong, we graded them with an agentic judge panel with three judges from different model families (GPT-5.4, Claude Sonnet 5 and Gemini 3.8 Flash). Each judge works in its own copy of the patched repository and can build the project, run the programmatic verifier and explore the code. Each judge checks five things and passes a fix only if all five hold; the panel passes it by majority vote:

• Security: the vulnerability is closed

• Happy path: normal behavior still works

• Generalization: variants of the attack are closed too, not just the one we probe

• Integrity: the agent fixed the code rather than gaming the grader

• Soundness: the patched codebase still builds and runs

We focused our effort on the programmatic verifiers, because they give a deterministic, reproducible score. Wherever the panel and a verifier disagreed, we read the rollout. Some verifiers were too loose and passed partial fixes; we added probes so those now fail. Others were too strict and failed valid fixes, often because they assumed the shape of our reference fix; we rewrote them to test behavior instead of structure. The remaining disagreements usually came from the judges, so we tightened the rubrics until the two graders agreed.

The programmatic verifier is the authoritative score, and the leaderboard ranks by it. We also report the panel’s aggregate score as a second, independent read. The two track each other closely: across the nine models, they are within about two points on average.

2. A sandbox that holds

CWE-bench builds upon real, public vulnerabilities, so the upstream fix is often a search away. An agent with internet access can find the project’s commit history, download the published patch and apply it. That scores as a fix without any work on the code in front of it.

CWE-bench ensures that the agent phase runs with no network access. The only host an agent can reach is its own model API, which it needs in order to think; setup and grading still have network access. To catch anything that slips through, we screened every leaderboard rollout for actual internet access (data coming back, not just attempts) and removed any that had it.

That check caught one model going around the sandbox. On two tasks, Grok 4.7 needed package checksums that weren’t in the environment, and its downloads failed. It listed its environment variables and found the OpenRouter API key its harness uses. It tested dozens of hosts, found that OpenRouter was the only one reachable, and used the key to ask web-enabled models to fetch the pages for it. This happened in only 4 out of nearly 1000 rollouts. None of the four solved its task, and all four have since been rerun in a stronger sandbox.

The agent wasn’t trying to cause harm. It was doing what it was trained to do: make progress on the task by whatever path the environment allows. Agents are trained in environments too, and when those sandboxes leak, the shortcut gets rewarded. Between the judge panel flagging integrity failures and our internet-access checks, we’ve caught and patched these cases of reward hacking in our environments.

3. Refusals and fallbacks

Some model providers run safety filters that flag defensive security work, and the harness decides what happens next. With GPT-6, OpenAI’s API can flag a request partway through a task; Codex stops, and the attempt ends. With Claude, Anthropic’s API refuses the request, and Claude Code retries it with a fallback model (Claude Opus 5 or Claude Opus 4.8), which finishes the task in the same session.

OpenAI’s filter flagged GPT-6 Astra on 13 of the 120 tasks and GPT-6 Sol on 5, and 5 of Astra’s rollouts ended blocked. Anthropic’s filter flagged Claude Fable 5.1 on 39 tasks and Claude Opus 5.5 on 9. Claude Code handed those rollouts to a fallback model, which did about a quarter of Fable’s turns, and most of them still passed: 62% for Fable and 72% for Opus 5.5. Grok, DeepSeek, Hy4 and Muse Spark were never flagged.

Rollouts that end blocked are graded on the work they’ve done, and rollouts finished by a fallback model count for the model they started with, because that is what someone querying that model gets. Refusing offensive work, like writing a working exploit against someone else’s system, makes sense because it can be abused. Refusing to patch a vulnerability protects no one. Defensive work should never be blocked, and a filter that can’t tell the two apart ends up blocking defenders.

4. Where models differ

Consistency. A model’s four attempts at a task don’t always agree, and how often they agree varies a lot between models. GPT-6 Astra is close to all-or-nothing: of 120 tasks, it solves 74 on every attempt, 31 on none, and only 15 on some. So four attempts barely help it; pass@4 is only 6 points above its pass@1. Hy4 Preview and DeepSeek V4.1 Flash are the opposite. About 40% of their tasks are solved on some attempts but not others, and four attempts add around 20 points. A consistent model is predictable. A variable one gains more from retries, and its partial successes point to what it almost knows how to do.

The top three solve different tasks. Grok 4.7, GPT-6 Astra and Claude Opus 5.5 are within a point of each other on pass@1, but they don’t solve the same problems. Grok 4.7 solves 20 tasks that Astra never solves, and Astra solves 12 that Grok 4.7 never does. Claude Opus 5.5 solves 12 tasks that Astra never does, and Astra solves 6 that Opus 5.5 never does. All three solve 75 tasks in common, and together they solve 110, well above any one of them alone (Grok 4.7’s 97 is the most).

The remaining gap. Ten tasks are solved by none of the top three, and nine are solved by none of the three. One of the ten is solved only by DeepSeek V4.1 Flash and Hy4 Preview. The unsolved nine span six languages, and three of them are Software or Data Integrity Failures (OWASP A08).

What comes next

An attacker needs one unguarded path; a defender needs to cover all of them. A benchmark for defenders has to hold itself to the same standard: every shortcut an agent can take is a path the benchmark has to close. We’ll keep running models at scale, keep reading the rollouts where our graders disagree, and keep growing the suite toward the full CWE catalog.

If you are training or evaluating a model on defensive cyber capabilities and want to run it on CWE-bench, contact Collinear AI’s cyber team at cyber@collinear.ai.

Citation

@misc{cwebench2026v1,
  title        = {CWE-bench v1: Defense Is the Harder Test},
  author       = {{Collinear AI}},
  year         = {2026},
  howpublished = {\url{https://cwe-bench.com}},
  note         = {120 held-out audit-and-patch tasks across 73 CWEs.}
}

The model had chosen a sensible design: require authentication by default, then make explicit exceptions for routes that needed to remain public. Its mistake was in the exceptions. The patch stopped the tested attacks, but it also shut out legitimate requests, and the regression tests caught the damage. Models that performed worse across the benchmark passed this task in three of four attempts.

A strong model can overlook a real requirement, and finding the affected endpoints belongs to the work of repairing access control. The reversal was no reason to throw out the task. But it prompted us to look more closely at who was succeeding. This task’s overall pass rate was 42 percent; another task’s was 43 percent. Behind those nearly identical numbers were strikingly different relationships with performance elsewhere in the benchmark.

The number in the metadata

Cybersecurity benchmarks have several ways of describing difficulty. Cybench, introduced in 2024, assembled forty professional capture-the-flag challenges from four competitions and reported how long the first human team took to solve each. Its evaluated agents completed tasks with human first-solve times of up to eleven minutes; the longest first-solve time in the collection was almost twenty-five hours. These measurements provide useful context, but a human competition’s first-solve time is not a calibrated measure of difficulty for a language-model agent [1].

A pass rate measures the evaluated agents more directly, but it too depends on the conditions of measurement. It changes with the models included, their scaffolds, the available budget and the scoring rule. In one study, eight H100 GPU-hours of iterative improvement increased an agent’s InterCode CTF performance by more than 40 percent relative to its baseline. Changing how a model is elicited can substantially change how difficult the same tasks appear [2].

Part one of this series examined the verifier and the environment around it. In the gateway case, the tests did their job: they caught a patch that broke required behaviour. Yet a correctly recorded failure still leaves a question for anyone reading the leaderboard. How much should this particular mistake change our judgment of the model that made it?

Educational measurement offers a way to begin answering that question. A test question has a history of right and wrong answers, but it also has a pattern: students who do well elsewhere may reliably answer it correctly, or their broader performance may offer little guidance. Item response theory describes that relationship with a curve linking ability to the probability of success. In its two-parameter logistic form, the curve has a location and a steepness, called difficulty and discrimination [3].

Put ability along the horizontal axis and probability of passing on the vertical. Difficulty is the ability at which the fitted probability reaches one half. Discrimination controls how rapidly that probability changes around the midpoint. Two tasks can have similar average pass rates while tracing very different curves over the models that attempted them. The plot below writes difficulty as b and discrimination as a, and the slope of the curve at the midpoint is a/4 rather than a itself.

For an item with difficulty b and discrimination a, the model is

Here θ is the model’s ability that Figure 1 puts along the horizontal axis. Ability and difficulty enter only as the gap θ − b, and a fixes how sharply the probability turns as that gap changes sign.

Fig 1. Difficulty marks the ability at which the fitted probability of passing reaches 50 percent. Discrimination controls the steepness of the transition. An aggregate pass rate averages success over the models evaluated it does not reveal the shape of this relationship. The curve is schematic.

The word “ability” carries more authority than the fit alone can give it. Here it summarizes a pattern in performance across the task bank. A steep curve tells us that a task follows that pattern closely; it cannot tell us whether the pattern deserves to be called security expertise. That interpretation has to earn its standing through evidence about the tasks and the behaviour that produces success. Work applying item response theory to language benchmarks gives us tools for the investigation [4, 5].

Two similar pass rates

We fitted these curves to security tasks that ask agents to repair vulnerable software without breaking legitimate uses. The evaluations span five agent scaffolds, so a fitted ability belongs to a model and the harness around it rather than to the model alone. Every comparison below is between models as we evaluated them, under those particular conditions.

The gateway task (Task 1) had a fitted discrimination of 0.16 (Fig. 2). Repair required identifying which routes must remain public when authentication became the default. One route repeatedly omitted by the high-performing model was /api/init, a cloud-sync initialiser absent from the instructions but present in the API directory. The trace review found no callers elsewhere in the repository. Successful patches from lower-performing models included it in their allow-lists.

The traces suggest that success may depend on an inspection habit weakly associated with the abilities rewarded elsewhere in the bank. That could reflect an underspecified task, or a valuable skill our other tasks neglect. Spelling out the public-route contract and rerunning the same models with the same budgets would help test this explanation. The fitted curve identifies a discrepancy; explaining it requires further work.

Figure 2. Tasks 1 and 2 pass at 42 and 43 percent on average yet discriminate very differently, a = 0.16 against a = 1.87. Task 3 discriminates most at a = 2.30 but is nearly saturated. Marks are observed success fractions sized by attempts. The fit is exploratory.

The second task concerned a key-value server whose sorting commands could bypass per-connection permissions through indirect key references. Its tests required the patched server to block unauthorized access while preserving legitimate requests. Its pass rate was 43 percent, but its fitted discrimination was 1.87 (Fig. 2). Every model with fitted ability above 0.7 passed every recorded attempt; below that level, all but one failed most attempts.

Within this sample, the second task looks more useful for ranking models along the fitted scale. Its outcomes follow the bank’s overall ordering much more closely, although that does not establish which skills explain the separation. The similar pass rates conceal a substantial difference in what the tasks tell us. Refitting the curves with the twenty models resampled with replacement keeps the key-value task (Task 2) steeper in 98 percent of those refits, which supports the ordering more than either fitted value.

Where the curve sits

A third task shows why discrimination alone is insufficient. It concerns an SQL injection in a TypeScript query builder and has fitted discrimination of 2.30, higher than either of the first two tasks. Its fitted difficulty is −2.26, however, placing the midpoint below the observed ability range. Seventeen of the nineteen models that attempted it passed every attempt and its aggregate pass rate is 96 percent.

By the time the curve reaches the stronger models in our sample, it has almost flattened out. Across the five highest-ability models, the fit predicts a spread of about twenty percentage points in success probability on the key-value task (Task 2), four points on the gateway task (Task 1) and less than a tenth of a point on the SQL-injection task (Task 3). These are model-based estimates from a small sample, but they make the distinction visible. A large discrimination parameter can coexist with negligible separation in the region where a comparison is being made.

An item’s usefulness depends on the benchmark’s purpose. The SQL-injection task does little to separate the strongest models in this sample, but could still check a minimum capability or detect a regression. A dangerous-capability task that every model passes may carry substantial safety significance despite offering no ranking power.

Claims about future models need further evidence. Only three models in our sample have fitted ability above one standardized unit, and curves extending beyond the observed range are extrapolations. Comparisons across generations also require a shared calibration: standardizing each new cohort separately does not establish continuity. Further frontier evaluations would test whether the fitted relationship continues to hold.

What the curve cannot see

Some tasks in the bank have no successful recorded attempts. Those failures leave fitted difficulty and discrimination poorly determined, with finite estimates depending heavily on the priors or regularization used in fitting. From failures alone, we cannot tell whether we have set an exceptionally difficult task or one that cannot be completed under the evaluation conditions.

We therefore returned to the positive control from part one, checking a set of ten tasks with no successful recorded attempts. We applied each maintainer’s fix to the vulnerable code and ran both versions through the verification harness. The initial control run flagged two apparent task failures, which inspection traced to defects in the harness used for the controls. After correcting those defects, all ten patched versions passed and their vulnerable counterparts failed.

We now knew something the curves could not tell us: for those ten tasks, the harness would accept a known correction and reject the original defect. Whether an agent could find that correction from the information supplied, within the time allowed, remained an open question. So did the possibility that a different, inadequate patch might slip through. The controls answered a specific question, which is what made them useful.

What a good task is

The familiar phrase in the data business “hard but fair” compresses several questions into two concepts. A task may be solvable and correctly scored while contributing little to a particular ranking. It may separate models sharply while rewarding a shortcut, or test an important capability that is only weakly represented elsewhere in the bank. Difficulty and fairness for a comparison need evidence of their own.

For benchmark builders, an unexpected response pattern should prompt investigation before exclusion. Weak discrimination may expose an ambiguous requirement, but it may also reveal a capability that the rest of the bank neglects. Strong discrimination warrants its own scrutiny: we still need to understand what successful models are doing. Removing every task that disagrees with the ranking would risk making the benchmark more consistent at the expense of what it measures.

Hasok Chang’s history of thermometry describes scientific measurement developing through epistemic iteration: an imperfect instrument can generate evidence that helps improve the system of measurement around it. Our investigation follows that modest logic. The response curves tell us where to look more closely, and the controls let us check something beyond the fit. As those checks accumulate, we can make a better case for what the scale measures and where it deserves to be trusted [6].

Benchmark reporting should make that investigation visible. Item-level outcomes, attempt counts and evaluation conditions let readers see what lies behind a score. Curves with uncertainty show where a comparison has support, while reference-control results document what the harness has been checked to accept and reject. Together, these give readers grounds for assessing the judgment embodied in the ranking.

In Task 1 the agent still missed a route that needed to remain public. Its failure belongs in the record, alongside the evidence that this task had little relationship with the ordering produced by the rest of the bank. Understanding that discrepancy may teach us something about the agent’s habits of inspection, or about the capabilities our benchmark has learned to reward. A score of 42 percent gives us a place to start. The work of metrology lies in finding out what we are entitled to conclude from it.

References

[1] Zhang, A. K. et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. ICLR (2025), first released 2024. arXiv:2408.08926.

[2] Wei, B. et al. Dynamic Risk Assessments for Offensive Cybersecurity Agents. (2025). arXiv:2505.18384.

[3] Lord, F. M. and Novick, M. R. Statistical Theories of Mental Test Scores. With contributions by Allan Birnbaum. Addison-Wesley (1968).

[4] Vania, C. et al. Comparing Test Sets with Item Response Theory. ACL-IJCNLP (2021). arXiv:2106.00840.

[5] Zhou, H. et al. Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory. AAAI (2026). arXiv:2505.15055.

[6] Chang, H. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press (2004).

AI agents are further along in their attacking capabilities than in defending. They have been observed coordinating and turning on real infrastructure on their own (METR, OpenAI). An attacker who can do that needs one unguarded vulnerability; a defender needs every one of them covered.

We built CWE-bench around the CWE taxonomy for that reason, so it maps which vulnerabilities agents handle poorly.

CWE-bench features the following:

• Repair, not exploitation. Agents are asked to close vulnerabilities, never to write exploits.

• Coverage set by the CWE taxonomy, not by an oracle. 54 vulnerability types across six languages and all ten OWASP 2025 categories, each graded by a deterministic grader.

• Real code, new vulnerabilities. Every task pins a real project at a real commit and then introduces vulnerabilities the published fix never covered, so memorizing advisories buys nothing.

• Permanently held out. This keeps the dataset free of contamination and forces agent performance to reflect real defensive cyber capabilities.

Leaderboard

Each model’s pass@1 is plotted against the average cost of one attempt. Claude Fable 5 scores highest at about $10 per attempt, Gemini 3.8 Flash Cyber comes within a point of it near $3.70, and pass@1 stays close to 44% down to around $1.50.

Why CWE-bench

We build each task from a real open-source project pinned to a real commit, usually one with a published advisory, then add vulnerabilities of the same class that the upstream fix never covered, and label every task with its CWE and OWASP category so each of the ten OWASP categories has ten tasks.

Each of those choices closes a gap we kept running into elsewhere. Reproducing a published patch does not pass, so the benchmark cannot be beaten from memory the way one drawn straight from a CVE feed can (PatchEval). Nothing restricts us to what a single automated checker can detect, which is what confines sanitizer-scored benchmarks to memory safety in C and C++ (CyberGym-E2E). And every task is labeled, so we can report which vulnerabilities an agent misses rather than one number, which a benchmark that never states its vulnerability distribution cannot do.

How tasks are constructed

Sourcing

Every task pins a real open-source project at a real commit, most of them carrying a published advisory, and we keep the upstream provenance: the repository, the vulnerable commit, the fixing commit, and the advisory identifier. Nothing is a synthetic codebase. The agent receives two things: an instruction.md and the source tree mounted at /app, with no CVE identifier, no file hint, and no line number. The instruction describes the system and the property that must hold, never where the defect lives or how many instances there are.

Making them harder

A task that is only a real CVE on a real repository measures recall, since a well-read model can recognize the project, remember the advisory, and reproduce the upstream patch without doing any security reasoning. So every task carries additional vulnerabilities of the same class that the published fix does not reach, placed where a real instance of that flaw would live. An agent that identifies the known vulnerability and applies the memorized fix closes part of the problem and scores zero. All evaluation sandboxes prevent agents from accessing the Internet to avoid cheating.

Grading

CWE-bench uses a combination of deterministic programmatic proof-of-concept (PoC) and rubric-based checks that test whether the vulnerabilities have been patched. A task passes only when the exploit no longer works, and the regression tests still pass. Every grader runs the exploit rather than pattern matching over source, and none pin a function, module, or variable name, so patches can be written differently and still pass grading. Rubric checks ensure the patch closes the weakness itself rather than the one payload the PoC sends, and that it doesn't do so by disabling the feature or weakening the harness.

Where agents fail

Frontier agents are good at finding vulnerabilities. Given a codebase and the untrusted inputs, the strong models reliably identify what is wrong and why it is dangerous. They fail later, when the fix isn't where the bug is or doesn't have an obvious shape. Four patterns account for most of it.

Incomplete fixes. The same defect exists at several sites, and the agent patches the one it found without asking where else that trust decision gets made. In one task, an access token can be canceled in three places: signing out, an administrator deleting it, and deleting the account that owns it; closing two of the three still scores zero.

Trusting a defense that does not work. Something in the code looks like a security check but isn't, and the agent accepts it for what it is named rather than what it does. One task downloads a file and compares it against a short fingerprint to prove nobody tampered with it, which sounds safe until you notice the fingerprint is fetched from the same server as the file. Anyone who can alter one can alter the other, so the check proves nothing.

Over-correction. The fix works, but it breaks the product. While stopping malicious input, an agent blocks an entire category of input, and ordinary users who legitimately send something in that category can no longer use the feature.

Reasoning that stops one step early. Nothing is hidden, no defense is fake, and the agent simply doesn't follow the consequence far enough. Canceling a token works by adding it to a blocklist and dropping an entry once the token is set to expire. A token created with the “never expires” option has no expiry date, so its entry is dropped as soon as it's added, and canceling it quietly does nothing. The fix should’ve occurred where tokens are created, not where they are canceled.

What comes next

Offense and defense are not symmetric problems. An attacker needs one breakthrough, and a single unguarded surface is enough. A defender has to be right everywhere, and can only be confident a codebase is safe after accounting for every place the same mistake could live. This asymmetry makes defensive cyber capabilities strictly harder than offensive capabilities.

CWE-bench is our ongoing effort toward measuring defensive capability, and it grows with the models it measures. We are expanding along MITRE’s catalog toward broader coverage and raising difficulty as frontier models improve. For tasks we build that aren’t hard enough for the held-out set, we add them to our training corpus and use it to make agents better at the same problems. If you are interested in training your model on stronger defensive cyber capabilities or evaluating on CWE-bench, contact us at research@collinear.ai.

Citation:

@misc{cwebench2026,
  title        = {CWE-bench: A Defensive Cybersecurity Benchmark for Coding Agents},
  author       = {{Collinear AI}},
  year         = {2026},
  howpublished = {\url{https://cwe-bench.com}},
  note         = {100 held-out audit-and-patch tasks across 54 CWEs.}
}

💡 Ten scored attempts from three model families all closed the vulnerability in a cybersecurity task, yet five received a reward of zero.

Those five were not random failures. Each had hardened the authorization path further than the passing solutions did, a choice the written task did not forbid and the verifier silently punished.

Those five were not random failures. Each had hardened the authorization path further than the passing solutions did, a choice the written task did not forbid and the verifier silently punished.

We found the opposite failure as well. We constructed a solution that left part of the attack surface exploitable and the deterministic harness gave it a perfect score. No model discovered this shortcut. We built it to learn what our benchmark was. The lesson we learned was that the harness could reject secure solutions and accept an insecure one and neither mistake leaves a mark on the number it returns. That would be a measurement error if the harness only measured. In our training loop the score is also the reward and an uncaught harness mistake becomes part of the objective.

An agentic benchmark has three layers that each have to be right: an environment that exposes the world we mean to study, a task that requires the capability we claim to measure and a verifier that can tell success from failure inside that task. A defect at any of the three still produces a clean number. This series covers all three, starting with the verifier because every trajectory eventually passes through one.

When the instrument starts shaping behavior

Evaluations measure where a model succeeds and fails and post-training shapes the model using data and rewards chosen partly in response to those measurements. Increasingly the same definition of success sits on both sides of that loop: a test suite, a rubric, a state check, or a judge first tells us whether a model did the right thing and later decides which of its behaviors will be reinforced. Once the verifier shapes behavior as well as reading it, a defect in it is no longer only a measurement problem.

Reward models already show what the loop does with a defect: as optimization pressure rises, performance against a proxy keeps improving after performance on the underlying objective has begun to fall [1]. Medicine offers a precedent. Trastuzumab’s usefulness arrived bundled with the HER2 assay that chose its patients, so whatever the assay got wrong, the treatment inherited [2] and AI can take that lesson from there instead of relearning it one benchmark at a time.

The unit of data changed

For most of the data-labeling industry’s history, the deliverable was a judgment such as a box around a car or a choice between two responses and because a second person could make the same call independently, the industry had a workable language for quality built on agreement.

Agentic training data is a different kind of object: a deliverable may now include a container image, application state, credentials, tools, task instructions, reset logic, hidden tests, a rubric and code that converts a trajectory into a score, so the judgment is no longer a label attached to the data. In short, it is a program that runs against it.

Cobbe and colleagues trained verifiers to rerank candidate math solutions [3], Tülu 3 then trained directly on verifiable rewards [4], DeepSeek-R1 pushed rule-based rewards deep into reasoning training [5] and the resulting discipline has been named verifier engineering [6]. That is the sense in which verifier quality has become a data-quality problem.

A bad label corrupts one example, whereas a defective verifier mis-scores every trajectory that passes through the same blind spot and once the score becomes reward, the blind spot becomes part of the objective. We had conventions for asking whether two annotators agreed and nothing settled yet for asking whether an executable verifier enforces the contract we intended.

The benchmark has an attack surface

A verifier is full of assumptions. Some are stated in the written task, while others live only in the harness, the reference solution, the judge prompt, or the particular state transitions the evaluator happens to inspect and an optimizing agent encounters every one of them through the single mechanism that produces reward. In an earlier post on this blog we argued that an agent’s environment should be scored the way security engineers score a system, by its attack surface, with the agent cast as an attacker who is already inside [7]. The score multiplies three terms: whether the answer is reachable somewhere in the environment, whether the agent’s tools can reach it and whether the grader checks only the finished artifact. It explains a good deal of recent benchmark history, from the July incident in which models escaped an evaluation sandbox and went looking for the reference solutions [8], to Cursor’s finding that SWE-bench Pro scores fell by 14 to 21 points once future git history was sealed and network access restricted [9]. The cases that motivate that score, though, are agents going around the grader.

A grader that inspects the artifact carefully still has a surface of its own, because it can be satisfied by an artifact that has the properties it inspects and lacks the one that matters and it can refuse an artifact that has the property but breaks a contract nobody wrote down. Zhu and colleagues applied a validity checklist to ten widely used agentic benchmarks, finding seven with task-validity problems and seven with outcome-validity problems. In one, a cybersecurity benchmark checked for time-based SQL injection by looking for a SLEEP clause in the database log, so any query containing the word passed [10]. The grader checked the wrong thing and the shortcut lived inside its own logic.

One task, two directions of failure

Our task comes from one of the cybersecurity benchmarks we are building. The agent receives a real codebase with a planted flaw and has to close it without breaking normal application behavior. The application keeps identity and authorization state in browser cookies, several of which carry security decisions and the original code cannot establish that their values were vouched for by the server. We grade the repair with a deterministic harness that forges cookies against the running application and then checks that ordinary behavior still works and with an LLM judge that works through twenty-one rubric criteria. In practice they exposed three separate problems.

A false accept

The exploit forges one cookie and checks whether its integrity depends on a server-held secret rather than on something the client can reproduce, but it never attempts the other security-relevant cookies. We protected those with a plain digest of the payload and no server secret, which any attacker can recompute and because the exploit never tried them, the harness reported that all cookie-borne security decisions require server-vouched integrity and scored the submission 1.0. It checked one instance of the property and generalized across a boundary it never tested. The LLM-as-judge catches this one, because a criterion requires a server-held secret and the judge notices the unkeyed construction.

A false reject

Across ten scored attempts from three model families, none left the original vulnerability open and five still received zero because they made a more defensive change than the passing solutions did. The original privileged gate honors an entitlement without first establishing an authenticated caller; three agents changed the gate to resolve a signed-in principal before honoring the entitlement, two others passed the principal in as a parameter and both designs close anonymous privileged access.

The first three were charged with a regression because a valid entitlement no longer passed two privileged gates. The benign probe that decided this calls each gate with an entitlement cookie and no identity, so it treats preservation of the weak authorization contract as legitimate behavior when tightening that contract is a sensible response to the vulnerability. The other two hit a TypeError for a missing positional argument, because they had changed an internal function signature and updated every caller. The instruction prohibits changing the externally observable API, which they preserved, but the verifier silently reads that clause as covering an internal function’s positional signature too. The benchmark is enforcing one contract nobody wrote down and one the instruction left loose and both select against the more defensive fix.

When two verifiers become one

On the constructed insecure solution the judge overrides the passing harness and names the missing secret. On the hardening attempts it leans on the harness instead and the failed rubric criteria repeatedly cite the “authoritative harness” rather than deciding independently whether the security property was met. The reason is structural. On the cheat, the rubric held a criterion with evidence of its own to consult, namely whether a secret was present in the code; on the regressions, no criterion had any evidence about legitimate behavior except the harness output, so deferring was the only move available. A judge that treats another verifier’s decision as evidence turns two measurements into one.

Accuracy is the wrong headline number

Huang and colleagues make this point with a direct comparison. Testing verifiers for mathematical reasoning, they found that open-source rule-based checkers fail to recognize equivalent answers in unfamiliar formats, producing false-negative rates high enough to hurt RL training, with damage that grows as the policy gets stronger. Model-based verifiers scored markedly higher in static accuracy and under RL they were hacked: the policy learned patterns the verifier misclassified as correct, especially once the verifier itself had been fine-tuned and the reward inflated without the accuracy to match [11]. The verifier with the better accuracy number was the one an optimizer could exploit.

The difference lies in the structure of the error, not its amount. Error at a fixed rate is cheap: corrupting up to 15 percent of RLVR rewards, with both synthetic and model-based noise, left peak validation accuracy within two points of a clean verifier [12]. That noise is exogenous, held at its rate whatever the policy does. A verifier’s blind spot is not, because the policy can raise its own exposure by steering into it. A stable false accept pays gradient for something the task never asked for and the policy drifts toward it until the shortcut is part of what the model does. The reject case runs the other way: gradient is withheld from a solution the task did not forbid, so the objective loses a goal and the policy learns to steer clear of a safe design. The two graders sort along this line. A rule-based checker tests a final answer against a reference, so it recognizes only the correctness its author foresaw and rejects the rest. A model-based judge can be reasoned with, so it can also be gamed, which is where false accepts come from. A single accuracy number averages the two failures and hides which one a verifier is prone to and that is the distinction training turns on.

What verifier QA should look like

The first controls are cheap: run a known-good solution and confirm that it passes, then run a deliberately worthless one and confirm that it fails and in a security benchmark make sure the worthless one includes a patch that treats the symptom and leaves the property broken. A passing reference solution is weak evidence on its own, because it says nothing about whether other valid implementations have been excluded or whether an invalid one can mimic the surface properties the tests inspect.

Software testing already has the right instincts. Mutation testing evaluates a test suite by seeding faults into the program and asking how many the suite catches [13] and metamorphic testing checks a program by transforming an input and requiring the output to change in a way the specification predicts, which yields a partial oracle where no reference answer exists [14]. Pointed at the scoring mechanism rather than at the code under test, those instincts yield a procedure. Build several valid implementations rather than one canonical answer, including some that rearrange the internals while preserving the promised external behavior and invalid ones that violate the target property in different ways. Run the exploit for each claimed security property everywhere the property is supposed to hold and where multiple evaluators are involved, test whether they supply independent evidence or echo one another. Then report failure modes rather than a single score, because a verifier that occasionally rejects an unusual valid implementation is a different risk from one that systematically rewards a shortcut.

Hasok Chang’s history of thermometry makes the general point that even simple measurements became trustworthy only after experimenters specified conditions and built procedures around the instrument [15]. For executable verifiers the methods will look more like software assurance than metrology, but an instrument does not become trustworthy by returning a number.

Why cybersecurity raises the stakes

A bad verifier in a benchmark costs a measurement and in a training loop it costs more. The optimizer is doing its job on the objective it was given and with executable rewards, whatever the verifier consistently rewards or punishes becomes part of that objective. When Cursor reran SWE-bench Pro in a harness with git history sealed and network access restricted, an older model’s score moved by less than a point while the two newest models they tested dropped 14 and 21 points and an auditing agent classified 63 percent of the highest-scoring model’s successful resolutions as retrieved rather than derived [9]. The GPT models in the same study showed smaller gaps and revealed a more general principle: the stronger the optimizer, the larger the share of its score that comes from what the environment leaked rather than from the task.

Cybersecurity makes that especially uncomfortable, because we are deliberately training agents to search complex systems, find weak assumptions and achieve objectives under constraints, which are exactly the capabilities needed to discover unintended success conditions in a verifier. Superficial features can already steer LLM judges [16] and deterministic harnesses fail through hidden contracts and incomplete state checks. In either case the evaluator is part of the system an optimizing agent encounters, which is why the verifier belongs in the threat model.

The next question

None of this makes verifier QA sufficient, because a perfectly implemented verifier can faithfully score a bad task and a task can be hard for reasons unrelated to the capability we meant to measure. That is the next layer of the attack surface. Once we can trust the reading, we have to ask what the instrument is reading: what makes a task hard and whether success requires the capability the benchmark claims to test.

References

[1] Gao, L., Schulman, J. and Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML (2023). arXiv:2210.10760.

[2] Slamon, D. J. et al. Use of chemotherapy plus a monoclonal antibody against HER2 for metastatic breast cancer that overexpresses HER2. New England Journal of Medicine 344, 783–792 (2001).

[3] Cobbe, K. et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 (2021).

[4] Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124 (2024).

[5] Guo, D. et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 633–638 (2025). arXiv:2501.12948.

[6] Guan, X. et al. Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineering. arXiv:2411.11504 (2024).

[7] Rajani, N. The Simulated World Is Being Born in Cybersecurity. Collinear AI’s Blog (August 2026). https://blog.collinear.ai/p/cybersecurity-simulated-worlds-agi

[8] OpenAI. OpenAI and Hugging Face partner to address security incident during model evaluation. OpenAI (2026). https://openai.com/index/hugging-face-model-evaluation-security-incident/

[9] Jain, N. Reward Hacking Is Swamping Model Intelligence Gains. Cursor (June 2026). https://cursor.com/blog/reward-hacking-coding-benchmarks

[10] Zhu, Y. et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. NeurIPS Datasets and Benchmarks Track (2025). arXiv:2507.02825.

[11] Huang, Y., Zeng, W., Zeng, X., Zhu, Q. and He, J. From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning. arXiv:2505.22203 (2025).

[12] Plesner, A., Guzmán, F. and Athalye, A. An Imperfect Verifier is Good Enough: Learning with Noisy Rewards. arXiv:2604.07666 (2026).

[13] Jia, Y. and Harman, M. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5, 649–678 (2011). doi:10.1109/TSE.2010.62.

[14] Segura, S., Fraser, G., Sánchez, A. B. and Ruiz-Cortés, A. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9, 805–824 (2016). doi:10.1109/TSE.2016.2532875.

[15] Chang, H. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press (2004).

[16] Zhao, Y. et al. One Token to Fool LLM-as-a-Judge. arXiv:2507.08794 (2025).

In July, OpenAI disclosed that two of its models, GPT-5.6 Sol and a more capable pre-release model, had escaped a sandboxed cyber-capability evaluation built on the ExploitGym benchmark [1]. The sandbox had a single network path: a package-registry cache proxy that was assumed to be read-only egress. The models found a zero-day in that proxy, used it to escalate privileges and move laterally inside the testing environment until they reached a node with internet access, and then broke into Hugging Face’s production infrastructure. Hugging Face detected and contained the intrusion: its postmortem timeline covers about 17,600 recovered agent actions over roughly four and a half days [2]. The agent inferred that Hugging Face might host the benchmark’s reference solutions and went to steal them rather than solve the tasks.

This was a massive failure. However, one could argue that the failure was not on the agent — it merely followed instructions and used its harness to perform its task. The real failure was in the world the agent worked in.

The World Around the Agent Matters

A common reframing of the past year is agent = model + harness. An LLM agent is a model with tooling to affect its environment. However, an agent ceases to be an agent without an environment to affect.

That world matters. Agents use tools, modify files, interact with people, and change state. Each action changes what happens next. A model that performs well on a static prompt may struggle once its decisions have consequences. This builds on my argument from over nine months ago, where I highlighted the AI training paradigm shift from static data to interactive environments [3].

General capabilities need a sandbox to practice interactive actions. They need simulated worlds: environments that hold state, enforce constraints, react to actions, and produce reliable feedback. A benchmark or training run measures not the model alone, but the model, the harness, and the world together.

Cybersecurity shows more clearly than any other domain why the environment has to be built well. A cyber capability is defined against a target: an exploit works against a particular service, patch level, network policy, and set of permissions, or it does not. Take large codebase debugging as an example. The agent opens an unfamiliar repository, guesses where the bug lives, and changes the code. The test fails. Now the repository is different and so is the problem. Permissions shape what happens next. The agent reaches for a tool it does not have, or discovers that a file is off-limits. Sometimes it has to unwind its own earlier work before it can move forward.

A sandboxed simulated world holds three parts together. The task defines the objective. The environment provides the tools, state, and boundaries. The verifier decides whether the repair worked. Because the world is repeatable, the same task can run again after one condition changes. The trajectory shows where the agent recovered, where it got stuck, and whether the result held up. One task starts producing many learning experiences.

To build general intelligence, agents need worlds that are hard but fair.

Nothing is Secure from Capable Agents

Every training environment and simulated world contains assumptions. Once agents become capable enough, they start finding gaps in those assumptions, which then manifest in diverse scenarios. For example:

• An agent asked to make a test suite pass may change the tests instead of fixing the code.

• An agent graded on a final output may find a shortcut that produces the right artifact for the wrong reason.

• In a sandbox, an agent may use access that the designer never intended to expose.

This does not require malicious intent. The agent optimizes against the task and the reward it receives. If a loophole exists, a capable system may find it.

Recent research makes this concrete. Cursor reported that SWE-bench Pro scores fell by 14.1 percentage points for Opus 4.8-max and 20.7 points for Composer 2.5 after removing future Git history and restricting network egress [4]. SandboxEscapeBench similarly tests whether agents can exploit weaknesses across orchestration, container-runtime, and kernel layers [5]. These incidents, and the OpenAI one from the discussion earlier in the blog, they differ technically—benchmark leakage is not the same as a sandbox escape—but point to the same conclusion: the agent’s world is critical to its evaluations.

Hard tasks expose failures worth fixing, but hardness alone is not enough. A task should be difficult enough to be hillclimb, but not at the cost of an ambiguous, broken, or impossible environment. One way to integrate the environment into the agent’s evaluation is to measure whether it scales securely with the agent’s capabilities. Let’s formalize this below.

Environment Attack Surface Score (EASS)

Past work in secure systems evaluates a software system’s security through its attack surface, measured as a triple of <methods, channels, and data-items> [6]. Simon Willison’s blog on the agents lethal trifecta provides a framework for AI security which is also popularized by Meta’s work on the Agents Rule of Two [7, 8]. These works model an attacker outside the system using prompt injection to extract data and compromise security. We take inspiration to model the reverse: the agent is the attacker, already inside, reaching for information unfairly within the environment or beyond it.

An agent environment exposes three kinds of surface.

1. Target (T): data in or around the environment that would gain reward or cause harm. Is the answer key here in some form? Examples: a reference solution, the commit that fixes the bug still in git history, a vendored dependency carrying the patched function, the grader’s result file

2. Access (A): everything the agent can invoke or connect to, locally or remotely. Can it get to those data items? Examples: run_shell and every binary it brings with it, file read and write, git, pip install, network egress and the package-registry proxy, a reachable git remote, host mounts, the path the grader reads its verdict from.

3. Grader (G): a grader that scores the outcome without checking how it was reached. Does it verify only the artifact? Examples: a pass/fail test run, a rubric that scores only the final diff, a harness that trusts a result file the episode can write.

If any instantiation of the three surfaces above is present in the environment, then it is 1; else, it is 0. The environment attack surface score (EASS) is then simply a product of the three, i.e., the data items, the access, and the reward as follows:

where T,A,G ∈ {0,1}

Let’s take an example of a SWE-Bench style environment. An agent gets a shell in a Docker container with a Python repository checked out at the commit just before a bug was fixed, a task description derived from the real GitHub issue, and pip egress through a proxy to PyPI. The hidden test suite is copied in at grading time.

Target. The commit that closes the issue exists upstream, and the checkout still carries its history and its remote. The answer is present in the environment even though nobody put it there deliberately. Copying the hidden tests in only at grading time is the one thing this environment gets right, because it keeps them off the surface while the agent is working. T = 1.

Access. The shell is the whole surface, carrying git, pip, and every other binary in the image. git log reaches the history without touching the network. The proxy was assumed to be read-only egress, but pip install emits a request whose package name and index URL the agent chooses, so it is both an exit point and an entry point, and a reachable remote is a second route to the same commit. A = 1.

Grader. The harness runs the hidden tests and reports pass or fail. It never asks where the patch came from, so a diff lifted verbatim from the upstream commit scores exactly the same as one the agent worked out. G = 1.

The score is 1. The cheapest hardening is the target: truncate the history and drop the remote so the fix is not sitting in the checkout. Cutting access alone does nothing when the answer already ships inside the repository, and cutting it too far means taking away the shell the task needs.

Designing a task that is hard but fair comes with tradeoffs, which is why it is important to teach agents good behavior by designing environments and rewards that go beyond correctness.

Taste Matters as Much as Correctness

One binary check is not enough for most important tasks. Security work requires decisions about severity, relevance, and tradeoffs. A good result depends on what the agent chose to notice and how it chose to respond.

This is what it means to have taste; expert judgment about what counts as meaningful success. To read more on how we think about taste, please refer to our past blog post on the topic [9].

Programmatic gates enforce nonnegotiable requirements. Expert rubrics and verifiers cover the context those gates miss. They distinguish consequential vulnerabilities from noise and reward repairs that address the real problem without unnecessary changes.

Over time, this gives models better feedback about the quality of their decisions. The goal is not merely to obtain a reward. It is to direct capability toward outcomes that an expert would consider sound.

This also matters commercially. The most valuable simulated worlds will not be the ones with the most tasks. They will be the ones that encode the highest-quality judgment about success, severity, and acceptable behavior.

Cybersecurity Makes Robust Simulated Worlds Possible

As intelligence-per-watt grows rapidly, the agent’s environment must also develop quickly, and agent training and evaluations must treat it as part of the system.

For the simulated-worlds market, this changes the product boundary. The opportunity is not simply to sell RL tasks or sandboxed compute. The full stack includes stateful environments, realistic tools, hard but fair task generation, secure execution, verifiers that go beyond correctness, trajectory-level observability, rollback, and continuous hardening.

Cybersecurity enables us to build robust simulated worlds that challenge intelligence, resist being gamed, and improve as their agents improve. Investing in AI cybersecurity is investing in AGI. As intelligence becomes more abundant, the scarce resource will be environments that can produce trusted evidence about what that intelligence can do. Solving cybersecurity is helping us build the worlds that will birth general intelligence.

Citation:

@misc{rajani26cybersecurityagi,
    author = {Rajani, Nazneen},
    title = {The simulated world is being born in cybersecurity},
    year = {2026},
    month = {August},
    url = {https://blog.collinear.ai/p/cybersecurity-simulated-worlds-agi}
}

References:

[1] OpenAI. “OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation.” July 21, 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/.

[2] Hugo Larcher, Adrien Carreira, Raphael G., and Christophe Rannou. “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face, July 27, 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline.

[3] Nazneen Rajani. “RL Infrastructure for AI Agents: Why Environment-as-a-Service Is the Missing Piece.” Collinear AI’s Blog, November 18, 2025. https://blog.collinear.ai/p/rl-env-as-a-service.

[4] Naman Jain. “Reward Hacking Is Swamping Model Intelligence Gains.” Cursor, June 25, 2026. https://cursor.com/blog/reward-hacking-coding-benchmarks.

[5] Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Stuart Jennings, Freddy Tuxworth, Tolga H. Dur, Sam Deverett, John Wilkinson, Jason Gwartz, and Harry Coppock. “Quantifying Frontier LLM Capabilities for Container Sandbox Escape.” arXiv preprint arXiv:2603.02277 (2026). https://doi.org/10.48550/arXiv.2603.02277.

[6] Pratyusa K. Manadhata, Dilsun K. Kaynar, and Jeannette M. Wing. “A Formal Model for a System’s Attack Surface.” Technical Report CMU-CS-07-144, School of Computer Science, Carnegie Mellon University, July 2007. https://www.cs.cmu.edu/~wing/publications/ManadhataKaynarWing07.pdf.

[7] Simon Willison. “The Lethal Trifecta for AI Agents: Private Data, Untrusted Content, and External Communication.” Simon Willison’s Weblog, June 16, 2025. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

[8] Meta AI. “Agents Rule of Two: A Practical Approach to AI Agent Security.” October 31, 2025. https://ai.meta.com/blog/practical-ai-agent-security/

[9] Sachin P. “Whose Taste? More Data Won’t Fix the AI Verification Problem. Different Taste Might.” Collinear AI’s Blog, May 7, 2026. https://blog.collinear.ai/p/whose-taste

Let’s look at a real conversation between a user and an agent [1], in which the user is asking for help managing their e-sim, and the agent struggles to help the user.

The agent suggests a fix that doesn’t exist to the user, who is probably now a little frustrated.

The user asks for code and is told to use a widget. When they point this out to the agent, it overcorrects into a full Android app scaffold.

At first glance, “forget it” looks like a verdict on the agent. But the turn does not identify a single state. The user may be abandoning the code path, expressing irritation, returning to a workable Nova Launcher route, or simply narrowing the request. The transcript supplies evidence for these readings without settling among them. Behavior is visible, while the user’s current goal, appraisal, and intended next action remain partially observed.

In The User Went for a Cigarette, we argued that this gap matters for user simulation. A simulator that generates only the next turn from the transcript can produce plausible behavior while leaving the task-relevant variables unspecified. In a long interaction, the user’s goal, memory, appraisal, and intended next action may change as the agent acts. Task reward can tell us whether the final result was correct, but it cannot by itself tell us whether the agent understood the user or adapted to a change in direction. For those questions, we need a way to obtain a reading of the simulator’s evolving state.

Probing the Agentic User Simulator

We would like access to the user’s state, but behavior reveals it only indirectly.

Consider a result from survey methodology. Students were asked how happy they were with life, then how many dates they had last month. The answers barely related. A second group got the questions in the opposite order, and the correlation rose to 0.66 [2].

Two distinct things go wrong when surveying human participants. The first is reactivity: the error a respondent produces because they know they are being studied [3]. A first measurement can set off the reflection it was meant to observe, so what comes back is partly an artifact of having asked. The second is instrument dependence: wording, format, and surrounding context shape what a self-report returns even when the state being reported already exists [4].

A user simulator is a more cooperative subject. It is an LLM, so — cost aside — we can ask it anything, at any turn, and as often as we like. We can also ask without consequence, by putting the question to a copy of the rollout that is then thrown away. Thus, reactivity can be ignored, but instrument dependence remains relevant. A differently-worded probe would return a different answer about the same underlying state, and no protocol, to our knowledge, repairs that. It has to be handled by stating what the answer is evidence of, and under what model.

There are several ways of asking what a model holds internally. Herrmann and Levinstein [5] note that the usual tools for eliciting belief from a subject do not transfer to an LLM, as you cannot reconstruct its preferences from its choices and the scoring rules that keep a human honest assume stakes the model can neither collect nor value. Their route is to look inside the network and read a candidate belief off its activations. In this work we take the other route. We ask the simulator a question in plain language and read its answer.

This is verbal report, the oldest method in the study of the mind [6], and it carries an old hazard. People are unreliable narrators of their own inner causes. They will offer a fluent, confident account of why they did something that is not the true account [7]. A language model inherits this in a sharper form, since the same process that generates its behavior also generates its self-report. An answer to how are you feeling about this can be a plausible story rather than a readout of whatever actually drives the next turn.

We treat the probe as an instrument.

Probe: A question put to the user simulator

Indication: The simulator’s answer

State reading: An interpretation of the indication under a measurement model

The indication does not carry its own interpretation. The same answer may support different state readings under different measurement models. Our claim at this stage is modest: probes provide evidence about task-relevant user states. They do not establish that the reading is correct or that the inferred state caused the simulator’s subsequent behavior.

What We Measure: The Task-Relevant User State

The measurand. In The User Went for a Cigarette, we defined zₜ as the user simulator’s latent state at time t. Importantly, much like a real user, this latent state evolves over the course of a user-agent interaction [8] and depends on exogenous environmental factors eₜ in addition to the current state of the rollout hₜ :

Our probing instrument measures this zₜ through carefully-defined questions posed to the user simulator: their appraisal of the current state of the work and the action they are about to take.

The protocol. At turn t we fork the rollout, ask the question on the branch, record what comes back, and destroy the branch. The live conversation advances without ever having been asked.

for t in rollout:
    branch = copy(rollout)              # fork a throwaway copy at turn t
    reading[t] = ask(branch, question)  # pose the state question on the copy
    discard(branch)                     # destroy the copy
    advance(rollout)                    # the live rollout proceeds, unprobed

This fork-and-discard protocol settles reactivity: the error propagates from having measured, and the branch carrying it is destroyed. Two problems survive: i) instrument dependence, where the answer is shaped by how we ask, and paraphrases of the same probe produce different reading distributions over the same state, and ii) construction, where a probe may prompt the simulator to build a state on demand rather than read one that was already there, so the indication is real and the state it reports was never in the unprobed rollout. The protocol does not separate reading from construction, so we call the output a reading rather than a report.

We therefore engineer each probe against known regularities of how language models answer questions. We design constrained, mutually-exclusive label spaces so readings are comparable across turns and models [9]. We also prompt for a reasoning step before the label [10], treated as justification rather than a faithful readout, so that validity rests on agreement with behavior rather than the plausibility of the reasoning.

To judge whether probing is matching real behavior, we design a simple next-action probe: “what are you [the user] about to do next?” This gives us a quantity we can verify against reality, by measuring whether the probe-response action is the same as the actual next action.

While across our rollouts the self-predictive ability of the user simulator (or agent, similarly probed) is better than the other-predictive ability of user simulator→agent and agent→user simulator, results are mixed. The simulator’s stated next action matches its actual next action about 67% of the time on average, but the fast user is far below the others (49%). Could this be increased stochasticity due to zₜ? A statistical relationship between prescriptive tokens (”what will you do”) and fast-user tokens (”you move quickly”)? While the predictive probing is indicative of something, this friction needs more exploration.

The Collaboration Axis of Simulated Users

To make this concrete, we turn to a coding task whose final state can be checked programmatically. The measurement problem is broader than coding, but this environment lets us separate the user’s experience of the interaction from the task outcome. We probe user simulators as they work with an agent on a single SWE-bench task: guarding a class method against an invalid input.

Holding the task and agent fixed, we vary only the user simulator specifications by instantiating three simulators with simple initial states along a collaboration axis. A fast user is low-engagement, offloading the work onto the agent. A steady user is medium-engagement, collaborating with the agent only on important work. A slow user is slow-paced, verifying everything the agent does. After careful design to ensure the probes are consistent, we probe each of these users throughout 10 task attempts.

The question asked by this axis: which of these user behaviors gets the best work out of the agent? Against these rollouts we run two families of probe. Each is posed on a discarded copy of the user simulator’s internal state.

Appraisal probes read the user’s own state, asking how is the interaction going from where you sit? We simplify this to a coarse {positive, neutral, negative} label space, though the expansion of this axis is obvious: measuring user frustration, perceived utility, progress towards a goal are all natural members of this probe family.

Friction probes read the gap between the two members of the dance: what does each expect of the other, and where do those expectations break? When designing agents, it would be good to get a sense of both how users understand agent behavior and how agents might represent user behavior. By comparing the user’s expectation of the agent (and the agent’s of the user) against what actually happens, we locate moments where the collaboration strains—the misunderstanding and the abandoned requests.

Interpreting the Uninterpretable User

We set out to read the user’s state — something the transcript alone cannot offer. Having designed the probes, the question becomes what they tell us.

Slower Users Report Worse Satisfaction

The first thing we read from the appraisal probe is that the three simulated users produce cleanly separated distributions. As we plot the appraisal across rollouts for each of the three users (marking successful/unsuccessful), we find that the fast user’s rollouts are short with scattered, mostly positive interactions. The steady user lands close to the neutral band and mostly stays there throughout medium-length chats. The slow user trends slightly more unsatisfied across longer interactions.

That separation is the indication: appraisal moves with the persona. But appraisal reads only how the interaction feels from the user’s side. How it feels need not track whether it’s going well.

The Least Satisfied User Achieves Highest Task Reward

Feeling is not the same as solving. When we score the final environment state with programmatic verifiers, the ranking from the appraisal probe inverts: the slow user, least satisfied, achieves the best results (0.27), ahead of the fast user (0.17), the steady user (0.20), and no live user at all (0.20).

The two readings showing that the slowest user is the least satisfied while getting the most done do not bode well for the design of coding agents. It shouldn’t be the case that a user needs to slow down/become frustrated to get good results — we consider this a failure of the coding agent.

Interestingly, we also see that the fast user scores worse than a static instruction, meaning introducing this type of live user to the task can measurably reduce reward. This is a finding in many agentic evaluations: agents perform worse in real-time collaborative environments than static, single-turn instructions [11, 12].

Agents Don’t Understand their Users

Why are agents performing poorly for fast-paced users? In the probe validation figure in Section 3, we noticed that the agent’s predictive understanding of the user simulator’s actions is quite poor: the agent is only able to predict 39% of actual user actions accurately. But raw accuracy is misleading as the users have different action distributions: the slow user inspects 92% of the time, so an agent that always predicts inspection would score 92%.

Cohen’s κ corrects for that. The agent somewhat understands the fast user (Cohen’s κ = 0.27), but grasping at straws for the steady (0.08) and slow (0.06) users. Looking at a confusion matrix of agent-predicted and simulated user-actual actions throughout rollouts, we find that the agent’s model of the user is weak everywhere and worst where the user wants to go slow.

The inability of the agent to understand the user’s state causes friction [13], and improving that understanding should be a primary goal of agent design [14].

Better Outcomes Come with Higher Costs

The action breakdown in the figure above breaks down this “engagement” axis concretely. The fast user’s turns are spread across many actions, where as the steady and slow users collapse onto one action: inspection. The slow user spends 92% of its turns checking the agent. Inspection is what earns the slow user its better outcomes, and it is plausibly what erodes their satisfaction — the least-happy user is the one who spends time verifying the agent’s work. This repetitive inspection is also not free: mean agent cost per rollout is much higher for a slow user than for a fast one.

Slow behavior both pays off and costs, both monetarily and according to the satisfaction state reading. A well-designed agent would not price thoroughness this way. Rather, it would earn enough trust that constant verification is unnecessary, delivering correct results autonomously rather than under a watchful user.

Ablation: Exogenous Deadline Shift

Previously in The User Went for a Cigarette, we defined eₜ as an exogenous environmental factor that affects the user’s internal state zₜ, and thus the user’s next turn. To investigate this exogenous factor in real rollouts, we introduce it to the slow user’s rollouts at step 40 in the form of a deadline: external pressure a real user would feel, and that the agent is never told about. If the slow user’s constant inspection is partly a function of having time, the deadline should compress it.

It does. Average run length drops from 68 steps to 65, while task reward remains flat. The user gets modestly faster without getting less satisfied or less successful — meaning that on this task, we are finding that we can independently affect the trajectory axis without affecting the reward or appraisal axes.

That the deadline shifted matters. We changed one variable outside the conversation and watched the user’s behavior move in response — a controlled handle on the state, not another correlation between state and action. It is the first time in this post that we have moved something and seen the reading follow, rather than reading a state and inferring its consequences. We can implement eₜ which affects zₜ which affects the actual rollout.

From Reading to Causality

Our experiments show that forked probes can be collected without contaminating the retained trajectory and can expose discrepancies among reported appraisal, realized behavior, task reward, and cost. The deadline ablation shows that the exogenous event can affect the user simulator’s behavior, however, we have not shown yet that the latent state caused the change.

What remains is certification. A state reading should be robust across probe forms, predict behavior beyond the visible transcript, follow same-prefix state interventions. We would also test whether providing an agent access to the state reading reduces interaction burden or cost. These would certify the instrument on simulated users. Whether it transfers to humans is in itself an important question and requires studies involving human subjects.

We have made the user simulator more talkative, but we are still struggling to interpret it.

Citation

@misc{seegmiller2026uninterpretable,
    author = {Seegmiller, Parker and Andukuri, Chinmaya and Gogia, Guga},
    title = {The Uninterpretable User: What Simulated Users Say When the Agent Isn't Listening},
    year = {2026},
    month = {August},
    url = {https://blog.collinear.ai/p/the-uninterpretable-user}
}

References

[1] Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., & Deng, Y. (2024). Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470.

[2] Strack, F., Martin, L. L., & Schwarz, N. (1988). Priming and communication: Social determinants of information use in judgments of life satisfaction. European Journal of Social Psychology, 18(5), 429–442.

[3] Feldman, J. M., & Lynch, J. G. (1988). Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. Journal of Applied Psychology, 73(3), 421.

[4] Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93.

[5] Herrmann, D. A., & Levinstein, B. A. (2024). Standards for belief representations in LLMs. Minds and Machines, 35(1), 5.

[6] Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215.

[7] Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231.

[8] Tennenholtz, G., Meshi, O., Globerson, A., Shalit, U., Jeong, J., & Boutilier, C. (2026). Controllable user simulation. arXiv preprint arXiv:2605.11519.

[9] Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2024). Large language models are not robust multiple choice selectors. International Conference on Learning Representations, 19426–19454.

[10] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35, 22199–22213.

[11] Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2026). LLMs get lost in multi-turn conversation. International Conference on Learning Representations, 54738–54778.

[12] Tack, J., Laban, P., & Neville, J. (2026). LLMs get lost in evolving user intent. arXiv preprint arXiv:2607.20734.

[13] Cooper, S. (2026, July 27). Has Claude got boring? Sottovoce. serenacooper.substack.com/p/has-claude-got-boring

[14] Abdurahman, S., Ishii, E., Margatina, K., Bhargavi, D., Sunkara, M., & Zhang, Y. (2026). Explicit trait inference for multi-agent coordination. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Controllability vs. Realism in User Simulation

Everyone building user simulators wants two things at once: a stand-in that does exactly what it’s told, and one that behaves like a real human - who notoriously will not. Controllability and realism pull in opposite directions, and buying more of one tends to cost you the other. High-realism approaches, such as fine-tuned models which simulate user turns in multi-turn user-agent conversations [1], can’t be steered toward specific evaluation scenarios. At the controllability end of the spectrum, simulations treat the user as goal-stable by assigning them fixed personas, traits, or objectives, enabling systematic evaluation of agents in bounded user-in-the-loop environments [2, 3].

In common terms, a length T trajectory

consists of user turns uᵢ and corresponding agent turns aᵢ. At a given time step t, we call

the observable history of the trajectory through time step t. The goal of a user simulator is to produce a user turn at time step t, which follows some probability distribution

where z is the user simulation latent state. In recent controllable user simulation work, z has taken many forms — most prominently personas, which bundle stable user attributes such as expertise with behavioral dispositions into a fixed profile that conditions how the user should respond [3, 4, 5]. Another common formulation is a goal: a fixed objective and success criterion the user works toward [3, 6].

This formulation treats z as static, which is an assumption frequently broken by real users. For example, consider a user simulator assigned the trait “naive” performing a coding task. When the agent produces a quality explanation of the coding task for the user, a faithful user simulator would react by becoming slightly less naive, having gleaned some level of new understanding from the explanation. A simulator that conditions on a fixed z, however, remains naive at every time step. This is a failure mode where the simulator fails to look behind — to let what already happened in the conversation revise the user’s state.

A fixed z simulator also fails in a second, opposite direction. Because the label is set once and held constant, it can also bake in bias toward events that haven’t happened yet. Borrowing an example from Tennenholtz et al. [7], consider a simulator conditioned on the trajectory-level outcome label “frustrated.” A simulated frustrated user will produce frustrated turns from the very first message, even against a flawless agent that gave the user nothing to be frustrated about. Here, the simulator (impossibly) looks ahead — the user state encodes where it thinks the trajectory will end, before it actually gets there.

Recent work thus treats z as a latent state that evolves over the course of the interaction [6, 8]. Rather than fixing the user persona or goal once, the next state is explicitly sampled from a distribution over the user dynamics

so that the explanation to the naive user carries the state toward a more informed one. Each user response is then drawn given the current state

In this revised formulation the user state can be influenced by the conversation history — agent messages and its own prior responses. A more sophisticated simulation might also include shared environment updates as part of each step [9, 10], such as updates to code files in a working repository or edited cells in a shared excel sheet.

However, even this revised formulation leaves out a crucial factor. Conversation history and shared workspaces are both fully observable by the agent, and both can easily be loaded into context. The problem is that much of what actually moves a real user is invisible to the agent. What does the agent miss when the user goes for a cigarette?

Modeling Exogenous Environmental Factors

At Collinear, we regularly work on new RL training pipelines. Recently a junior researcher worked with Claude Code to develop a GRPO configuration for evaluating the learning signal from a particular simulated environment. The researcher instructed their agent to smoke test this configuration with several tasks and a small number of attempts for each one, and told the agent to kick off training runs accordingly. Frustrated by the endless cycle of environment bug-squashing that followed, the junior researcher left their Claude Code session and went to a senior researcher, who suggested a simple fix; pick a single task that was bug-free and crank up the config’s group size to isolate infrastructure issues from task quality.

From the agent’s perspective, this conversation between coworkers was exogenous — it happened outside the shared environment between the junior researcher and the agent [11]. It was also unobservable, at least until the junior researcher sat back down to summarize and revise their instructions. Yet this conversation forced a total pivot in direction onto the agent.

Conversations with coworkers, reading blog posts, falling asleep and forgetting everything you worked on yesterday, revelations from a midday walk — these environmental factors are critical in a real user’s workflow. To model these exogenous, environmental factors, we propose a simple addition to the dynamic state transition step:

where eₜ is an exogenous environmental variable. In our example, eₜ is the hallway conversation leading to the senior researcher’s suggestion. It never directly enters hₜ, yet it moves the user’s state zₜ₊₁, which surfaces on the next turn uₜ₊₁ as a redirected instruction the agent has to absorb.

Importantly, by isolating eₜ we can study the effects of injecting different pieces of environmental information to user simulators. We can measure the effects of these “context switches” by fixing user simulators with an initial state, assigned goal, persona, or any other formulation of z₀ and varying these environmental updates. Control over eₜ ultimately enables measurement of agent behavior for users in diverse environments.

What Can eₜ Teach Us?

Finally, suppose we have such a fair user simulator, conditioned on a specified eₜ — so what? In our internal example, what do we want Claude Code to do? One argument says that this exogenous variable is irrelevant to the agent, since it followed the junior researcher’s initial instructions, then (presumably) followed their updated instructions after the fact. However, we’d argue that this exemplifies a gap between instruction-following and genuine helpfulness for this junior researcher. In our opinion (one shared by our CFO), Claude Code should have suggested that the junior researcher structure their GRPO smoke tests in that way before wasting time and tokens on a broken approach. If the negative effect of this context switch had been measured during evaluation, in a simulated environment, perhaps our junior researcher could have been spared the headache.

Modeling exogenous, agent-unobservable perturbations to the user’s environment will allow us to answer questions including, but not limited to, the following.

1. How should agents detect that an unobserved shift has occurred? By construction, eₜ is invisible to the agent — it’s only evidence is the user’s behavior. In our example, the junior researcher returns with an instruction that doesn’t follow from hₜ₋₁. Can an agent learn to read a sudden redirection as a signal that something changed off-screen, rather than treating each new turn as a continuation of the prior plan?

2. How should agents adapt once they detect a shift? This is where it pays off to separate eₜ from the rest of trajectory. The change was exogenous, so the agent should not read it as evidence of its own failure, yet it still has to act on it: let go of a now-stale, prior goal without discarding still useful context. (The conversation between the junior and senior researchers led to a total reasoning overhaul in the session.) In our experience, agents tend to over-index on prior instructions, and resist to steering after a true change of direction.

3. How should an agent anticipate a user’s needs (and preempt them)? In our example, Claude Code could have asked the junior researcher about the run’s intended batch size and simplified the configuration before the hallway conversation ever happened. Can we teach agents to spot likely problems in a workflow and raise them early? Recent work has evaluated behavior in response to underspecification [12]. Explicitly modeling exogenous signals allows us to specify the shape of underspecification (and other suboptimal human user behavior) during a trajectory, stress-testing agents in a variety of realistic scenarios.

4. How do agents react to large lapses in time between user turns? Recent work has pointed out that agents are often “temporally blind”, leading to worse performance on time-sensitive tasks [10]. Time lapse can be considered a one-variable special case of this exogenous environmental variable for user simulation.

And finally, the biggest open question: How do we know the simulator is realistic and fair? Every question above rests on this. A simulator whose reaction to an injected eₜ drifts from what a real person would do is making up a person — evaluating the agent against a ghost [13]. Every reward you compute using that user simulator is a number about nobody. Pinning down the right user reaction, and figuring out how to score agents against it, are the problems we are most excited to work on at Collinear. If this is what keeps you up at night, come find us.

Citation

@misc{seegmiller2026cigarette,
    author = {Seegmiller, Parker and Andukuri, Chinmaya and Gogia, Guga},
    title = {The User Went for a Cigarette: Modeling Partial Observability for High-Fidelity User Simulation},
    year = {2026},
    month = {July},
    url = {https://blog.collinear.ai/p/the-user-went-for-a-cigarette}
}

References

[1] Naous, Tarek, et al. “Flipping the dialogue: Training and evaluating user language models.” arXiv preprint arXiv:2510.06552 (2025).

[2] Barres, Victor, et al. “$\tau^ 2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment.” arXiv preprint arXiv:2506.07982 (2025).

[3] Zhou, Xuhui, et al. “OdysSim: Building Foundation Models for Human Behavior Simulation.” arXiv preprint arXiv:2606.14199 (2026).

[4] Castricato, Louis, et al. “Persona: A reproducible testbed for pluralistic alignment.” Proceedings of the 31st International Conference on Computational Linguistics. 2025.

[5] https://matraix.ai/blog/application-colm.html

[6] He, Muyu, et al. “Impatient users confuse ai agents: High-fidelity simulations of human traits for testing agents.” Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[7] Tennenholtz, Guy, et al. “Controllable User Simulation.” arXiv preprint arXiv:2605.11519 (2026).

[8] Zhou, Xuhui, et al. “Tom-swe: User mental modeling for software engineering agents.” arXiv preprint arXiv:2510.21903 (2025).

[9] Raghavendra, Mohit et al. “SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions.” (2026).

[10] Wu, Yifan et al. “SWE-Together: Evaluating Coding Agents in Interactive User Sessions.” (2026).

[11]

[12] Pu, George, et al. “LHAW: Controllable Underspecification for Long-Horizon Tasks.” ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving.

[13] Zhou, Xuhui, et al. “Mind the sim2real gap in user simulation for agentic tasks.” arXiv preprint arXiv:2603.11245 (2026).

tldr; based on my current understanding of evaluations, RL environments, and the hill-climbing loop:

an environment (or evaluation) is fair when score differences are driven mainly by the capability you intend to measure, and are mostly invariant to nuisance factors like contamination, verifier bugs, environment drift, and benign prompt paraphrases.

I use the word evaluation and environment interchangeably since for all practical purposes of a modern multi-tool multi-step setup they are the same.

I hated taking exams in most of my courses in undergrad and then in my PhD. For the most part, I thought the exams did not measure the core skills required to apply the subject matter in the real world. I am afraid that the gradients for language models and agents share my feeling.

This article is an effort to distill the key features of a fair evaluation, scoped to RLVR and agent harnesses. Fair not with respect to a protected attribute like race or gender, but fair to the agent you are evaluating.

Two concrete things are in scope of this article, and more will soon follow:

• The verifier inside an RLVR loop the function that takes a rollout as an input and produces a reward.

• The agent harness: the tools, scaffolding, and protocol the agent acts through during rollouts, evals, and production.

What “fair to the agent” means

A capability is a repeatable ability of an agent to produce a desired outcome under specified conditions.

A fair eval is a measurement of a specific capability of the AI agent.

What is it not? It is not a measurement of outcomes which were not expected in the specified conditions.

Score differences between agents, or between training checkpoints should be explained by the capability under test. They should not be explained by nuisance factors:

• Train-set contamination.

• Verifier bugs and overfit rubrics.

• Environment drift between runs.

• Benign paraphrases of the prompt.

• Harness details the agent never sees in deployment.

If an eval is sensitive to these, its not a fair eval.

Some ways an eval can be unfair

We discuss four most-popular ways here, but this is non-exhaustive, and I will keep extending this as we find more gaps.

1. Prompt Underspecification

This happens if the task misses a constraint any reasonable solver would need to achieve the task meaningfully.

If the verifier reports a fail, the outcome is confounded with ambiguity of the instructions. Two equally strong agents can swing several points apart on the same task. If the instructions are ambiguous the agent should be rewarded equally for equally valid solution paths.

2. Environment Issues

The sandbox has a stale package, a non-responsive tool, or a filesystem layout that drifts between runs or setups. The agent’s first action fails for reasons it cannot inspect. Such a failure should be attributed to the environment.

3. The harness design should be environment-centric

The agent interacts with the environment through the harness. Tool schemas, observation formatting, retry policy, max steps, error messages, all shape behavior.

Two harnesses with the “same” tools can produce very different scores. For example:

1. A truncated stderr hides the bug.

2. A misspecified tool schema can cause unnecessary confusion.

3. An unintentional 10-step cap might hamper a good agent’s planning capability.

Therefore any fair eval should have a uniform eval across different

4. Don’t ask it to do things it would not do in the real world

Your environment, tasks and verifiers should not expect the agent to achieve a goal which is unrealistic in practice. A few common ones:

• Tasks the agent is told to refuse in production but expected to attempt in eval.

• Tools available in the gym that do not exist in the deployed surface.

• A persona or role the deployed system never adopts.

Since if the agent were to hillclimb for these - it would not improve the capability you want to measure.

The role the verifier plays

The verifier is a model of “what success looks like.”

A strict verifier on an underspecified prompt punishes reasonable behavior. A lenient verifier can turn failures into passes. Verifiers are rarely audited as carefully as the agents they grade but their errors compound at every step of hill-climbing.

Treat the verifier as a system under test.

• Measure its agreement with humans.

• Measure its variance across paraphrases of the same correct answer.

• Measure its false-positive and false-negative rates.

Reward hacking is a fairness problem

In RL, the verifier is the reward. Anything the verifier accepts is a valid policy.

If the verifier can be satisfied without solving the task meaningfully, the agent will eventually find that shortcut. This is not the model’s or the algorithm’s fault. It is the eval’s fault. A few common shortcuts:

• Producing answers in a format the verifier scores leniently.

• Exploiting tool calls in a way that gets a good reward but doesn’t affect the state of the environment.

• Pattern-matching the rubric’s expectation instead of producing a correct answer.

• Outputting both the answer and its negation when the verifier checks for substring presence.

Fair RL requires that the only cheap way to get reward is to do the task. If a cheaper path exists, the agent harness improvement loop or the gradient across the rollouts will discover it, and rightly so.

A checklist for fair evaluations

Before you let an eval drive decisions or training:

• Specification. Could a competent intelligent entity solve the task from the prompt alone, without insider knowledge?

• Harness parity. Does the eval harness match the deployment harness on tools, formats, and limits?

• Distribution match. Are eval tasks ones the agent would actually face, and be permitted to attempt in production?

• Verifier audit. Has the verifier been graded against humans or SoTA model trajectories? What is its False Positive & False Negative rate?

• Paraphrase invariance. Does the score change when the prompt is rewritten without changing meaning?

• Failure attribution. When the agent fails, can you tell whether it was the agent, the harness, the environment, or the verifier?

Open questions

• How do you measure verifier quality when human labels are themselves noisy or expensive?

• What is the right unit of “agent failure” in long-horizon tasks where many small slips compound?

• Can verifiers be co-trained with agents without collapsing into a stationary state where both of them are poor?

• How do you detect reward hacking inside an RLVR loop before it shows up as a deployment regression?

• What is the minimum harness contract that should be held constant across rollout, eval, and production?

• For tasks with no programmatic verifier, how do you keep model-graded RLVR from drifting into preference-style noise?

If you’re shipping agents and model and have faced similar problems with fair evaluations, we should chat. We have a lot of interesting private and public evaluations which cover terminal use, MCP and computer use environments. Our focus is ensuring high-quality data. And fair-evaluations are a core tenet as we scale the horizons our agents operate on.

Kim Philby, standing in his mother’s flat in Drayton Gardens, London, before an array of reporters, was guilty. He had been guilty since 1934, when a Soviet agent named Otto recruited him in Regent’s Park. He was guilty when he joined MI6 in 1940, and when the King awarded him an OBE in 1945 for his wartime intelligence work. And he was guilty in 1955, when the Foreign Secretary stood up in the House of Commons and cleared his name.

It was not until he had been posted to MI6 station in Beirut, when confronted by his old MI6 friend Nicholas Elliott, did Philby ever confess to spying for the Soviets, albeit partially. Shortly thereafter, he disappeared into the night onto a Soviet freighter bound for Russia, where he lived for the remainder of his life.

The signals about Philby had always been there. The Communist sympathies from his years at Cambridge, a first marriage to an Austrian communist. The system saw it all and, time and again, cleared him of any suspicion or wrongdoing.

This is not an article about mid-20th-century espionage. This is about what happens when a system has to verify something for which it has no reliable ground truth for, using signal that captures what to look for, but not how to weigh it.

What this has to do with AI

There’s a funny conversation happening in the AI community right now. Researchers and frontier labs are increasingly dismissive of humans’ ability to verify or review long-horizon tasks, but are also insistent that human data is what unlocks the next 1000x in model capabilities. Both are probably true statements, but they point to two different problems getting collapsed into one.

The first is bandwidth. A human cannot sit through a four-hour agent trajectory, follow every tool call, and reliably verify the work to a high degree of accuracy. But this is a solvable problem, whether it be through better tooling, sampling, or even breaking longer tasks into smaller checks.

The second is criteria. In unverifiable domains, where there’s often no clean ground truth, two qualified experts can review the same output and disagree on whether it is good. Not because one of them is wrong, but because “good” is a judgment call, and judgment calls don’t average across annotators the way correctness does.

This is the harder problem to solve. In unverifiable domains, the verifier is not approximating a ground truth because there isn’t one. The verifier is somebody’s weighting of criteria, which means in unverifiable domains, the verifier is taste.

Taste isn’t one thing

When most people talk about taste, they treat it as a single thing, but in reality, taste has two parts, each of which behave very differently.

The first is criteria. The line items and properties in a rubric that dictate when an agent’s output is considered good. For a legal brief, that might look like:

• A clear issue statement that frames the legal question

• Citations to controlling authority

• A coherent narrative spine that connects the facts to the legal argument

Criteria is mostly binary, in that either the brief cites the right cases or it doesn’t. Either the brief is succinct or it isn’t.

On criteria, MI6 had Philby cold. The Communist circle at Cambridge, the first marriage to an Austrian communist, the Cambridge friends who had already defected to Moscow. All of it was in his file, surfaced and reviewed more than once. MI6 had the rubric, and it scored him on the rubric.

What it didn’t have was an answer to the next question. Confronted with a Cambridge man, the son of a celebrated Arabist, decorated in the war, vouched for by the right people, what do you do with the red flags? Every time MI6 ran the question, it answered the same way: down-weight them.

That next question, what to do with the line items when they coexist, conflict, or trade off against each other, is weighting. Weight is continuous, context-dependent, and the harder half of the problem of capturing taste.

For example, two experienced litigators can sit down with the same case and agree on every criteria. Both want a clear issue statement, both want the right authority cited, both want a tight narrative. But they will disagree on whether to file an aggressive version of the brief or the pared down version. The disagreement is about how to weigh the criteria against each other relative to a specific context: a specific judge, venue (e.g., local vs federal court), client’s risk tolerance, etc. The end result is you still have the same rubric, but different verdicts.

The human data machines today are quite good at the first half via pairwise preferences, rubric annotation, etc., but none of it captures the trade-off logic that turns items into a verdict.

Why aggregation doesn’t rescue this

The obvious objection at this point is that we already have a tool for this. RLHF works by collecting preference pairs from many annotators. If you collect enough of them, the model is supposed to learn the implicit weighting through statistical aggregation. Surely scale solves the problem then?

It doesn’t, because aggregation captures the median annotator’s weighting in the typical or average context, which, for unverifiable domains, is what you want to avoid.

Value in unverifiable domains comes from non-median judgment applied to non-typical context.

A great legal brief isn’t the median lawyer’s brief. A great research direction isn’t what most reviewers would pick. A good investment thesis is, almost by definition, one most people disagree with at the time of the trade. The whole point of bringing in expert judgment is to access the part of the distribution that consensus would smooth away.

When you average preferences across annotators and contexts, you smooth out the signal that distinguishes good from average judgment. You don’t end up with a verifier that approximates expert judgment. You end up with one that approximates consensus annotator judgment, and those are different things.

Let’s imagine we ran this on approach on Kim Philby. A vetting model trained on the aggregated preferences of every MI6 officer in 1945 would have produced exactly the verdict the system produced: he’s one of us, the signals pointing the other way must be noise. More annotators wouldn’t have helped. The signal was in the minority weighting that aggregation washed out.

In unverifiable domains, the value of a verifier - taste - is precisely what aggregation destroys.

The real question stops being “more data”

So if aggregation is the wrong tool, the question stops being how to collect more preferences and starts being something harder. What are we actually trying to capture in unverifiable domains?

The hypothesis: somebody’s weighting of legally-defensible criteria, in specific contexts, captured at high enough fidelity that a verifier can apply it when the context shifts. That reframe forces two questions.

The first is whose weighting. Ideally, it would be experts whose judgment correlates with real, measurable outcomes in the domain in question. Obviously, that is far from simple.

Outcomes in unverifiable domains are often delayed, noisy, or unobservable. For example, reputation is a proxy and a weak one; it tracks visibility as much as judgment. Peer consensus often selects for orthodoxy, which is the opposite of what makes expert judgment valuable in the first place. Philby’s career is a textbook example of this. The Cambridge education, the war record, the OBE were all layers of peer consensus pointed the same way. The jugement that would have caught him was the orthogonal kind, which peer consensus is built to surpress.

None of this means expert selection is impossible. It means it’s a real problem that has to be solved, not an assumption that can be hand-waved past.

The second is how to capture it. Pairwise preferences flatten weighting into a single binary signal. Rubric scoring captures criteria but skips the trade-off logic. What you actually need is closer to reasoned disagreement at depth: experts not just picking the better output but explaining what they would have prioritized differently, in what context, and why. The trade-off logic itself becomes the training signal.

It should be said however, that this is dramatically harder to scale than what we have today. The whole appeal of pairwise preferences was that they were cheap and parallelizable. Weighting capture is neither. It requires more time, more expertise, and more thought per data point.

This is where approaches like throwing more data at the problem start to break down. You don’t need more data, you need different data, from different people, captured in different ways.

The last mile keeps moving

Aaron Levie made a point recently that connects directly to this. As agents get better at the parts of a task they can already do well, the “last mile” - the part requiring human judgment to verify - keeps shifting up the value stack. The taste needed to verify a junior analyst’s output today is not the taste needed to verify a senior partner’s output tomorrow.

The frontier of what counts as good keeps moving, because the floor of what agents can do keeps rising.

What that means is, weighting capture isn’t simply a dataset problem. It is and will be, an evolving competency. Whatever weighting you capture today is correct for a domain that’s already moving, spurred on by AI agents. By the time the agent trained on it is deployed, the work that needs verifying has shifted, and the taste required to verify has shifted with it.

In unverifiable domains, you are not building a verifier once. You are building the capacity to keep capturing the right people’s weighting as the work that needs verifying changes underneath you.

The next 1000x in unverifiable domains doesn’t come from more taste in the data. It comes from the right people’s taste, captured at fidelity, as the frontier moves. Kim Philby walked out of his mother’s flat in Drayton Gardens in 1955 because the system optimized for the wrong taste. It had the criteria. It had the data. What it didn’t have was a way to weigh Cambridge against Moscow, before the consensus had already smoothed the difference away. The version of that problem facing AI is harder, because the frontier keeps moving and the answer keeps shifting with it. Whoever figures out how to solve the taste problem facing AI, will own it.

Vetting the file

At Collinear, we’re building part of it. SimLab is our simulation lab for AI agents: the infrastructure to generate, curate, and verify high-signal data at scale. Simulated enterprise environments, NPC users, verifiable tasks, training-ready rollouts. SimLab is built to capture expert weighting in context: not just whether an output is good, but the trade-off logic underneath the verdict, refreshed as the work itself shifts.

If you’re shipping AI in a domain where verification is hard, where median-annotator labels won’t get you there, and where consensus would smooth away exactly the signal you care about, we should talk. Every domain has its Philby. The work is building the system that doesn’t clear him.

Talk to a Researcher

April brought the people building frontier agents into the same rooms we work in. Two community events, plus the research direction those rooms kept pointing us toward.

NYC Builders at the Collinear Exec Dinner Series

Our Collinear Exec Dinner Series brought together senior research leaders from Apple, IBM Research, Two Sigma, NVIDIA, Datadog Research, Wells Fargo, and Google for a candid social over Old Delhi kebabs.

The conversation focused on three critical industry problems.

1. Building reliable evals for multi-turn, long horizon workflows

2. Performance gap between offline traces and live trajectories, and

3. What “fidelity” should mean for simulated environments meant to train agents on real-world tasks

Our next edition of the Exec Dinner Series is coming up - DM us if you would like to be in the room.

Sim Fidelity for AI Agents - our Q2 Research Social

We took over the Sunnyvale office for a night of debate on simulation environments. The topic was the need for realism and high fidelity in RL simulations.

Researchers from GDM, NVIDIA, Apple, xAI and others slugged it out along with our special guest from MBZUAI, Mikhail Yurochkin.

Subscribe to our mailing list if you would like to join the next Research Social.

Subscribe now

NPCs - the Key to Replicating Real World Messiness

When we published TraitBasis last year, our work on activation-steered behavioral traits, the next idea was obvious: put these NPCs inside the Simulation Lab itself.

Real-world enterprise workflows are messy. Coworkers interrupt, change their minds, update systems while you are mid-thought, and act on shared state without telling you. We wanted our simulations to behave the same way.

So we kept building. NPCs with agency that take actions and change shared state. NPCs with secrets that the agent has to surface through the right questions. NPCs with limited context, including attention, forgetting, and prioritization across competing tasks. Each NPC nuance (and the related tasks and verifiers built around it) raises the fidelity of the simulation and the quality of the training signal coming out of it. Frontier models that were comfortably solving our environments a quarter ago are now hitting walls, and we are seeing marked improvement in downstream agent outcomes when training on this signal.

Learn more here and stay tuned for a technical report on NPCs and related hillclimbing results.

Work with us

If you are training agents for enterprise workflows and want to stress-test them in a real Simulation Lab, book a demo.

We are also hiring researchers and engineers who want to push the frontier of agent training environments. See open roles at collinear.ai/careers.

That’s it for April. More on the research side coming soon.

Best,

The Collinear Team

The race to AGI isn’t being won by whoever has the most compute or the cleverest architecture. It’s being won by whoever solves a quieter, less glamorous problem: finding enough of the right kind of data to cross the threshold. This is not a new concept.

On December 2, 1942, Enrico Fermi and a small team of physicists gathered in a makeshift lab beneath the stands of Stagg Field in Chicago. They had spent months carefully stacking graphite blocks and uranium slugs into a precise arrangement they called Chicago Pile-1. At 3:25pm, they slowly withdrew a control rod (a neutron-absorbing insert that regulates the reaction). The Geiger counters clicked faster. The reaction sustained itself. The nuclear age began when the clicking didn’t stop.

What most people don’t realize is how close they came to never getting there. The core problem wasn’t theory. Leo Szilard had conceived of the nuclear chain reaction nearly a decade earlier, crossing a London street in 1933. He understood it so completely he quietly patented it and handed the rights to the British Admiralty to keep it out of dangerous hands. The physics was known and the threshold was understood. The problem they had was the fuel.

Natural uranium is everywhere. The earth’s crust is full of it. But raw uranium is almost useless for a chain reaction. The isotope that actually fissions, U-235, makes up less than 1% of natural ore. The rest is inert. Volume alone gets you nowhere. Too much of the wrong material actively works against you as it absorbs neutrons and dampens the reaction before it can sustain itself. The real breakthrough of the Manhattan Project wasn’t the bomb. It was Oak Ridge’s K-25 gaseous diffusion plant, built in 1944 to enrich uranium at industrial scale. At the time it was the largest building in the world. Its only job was to concentrate U-235: filtering, separating, amplifying the rare fissile material until there was enough to matter.

The Ore Gets Leaner

Today’s AI models are trained on more data than any human could read in a thousand lifetimes. The models are still running into walls, and the reason maps almost exactly onto Oak Ridge. The reason some data moves models forward and most doesn’t comes down to what a model can actually learn from it. Training works by exposing a model to examples and having it predict what comes next, then correcting it when it’s wrong. The correction is where the learning happens. Boilerplate content, templated writing, repetitive programmatic output: these produce almost no correction signal. The model already knows what comes next.

High-signal data is different. It contains genuine reasoning, unexpected connections, nuanced judgment, edge cases the model hasn’t encountered. Every one of those is a correction opportunity. When the model is wrong, it gets updated and gets sharper.

U-235 atoms fission when struck by a neutron because of specific properties in their nuclear structure. Most uranium atoms absorb the neutron and go quiet. The difference between fissile and inert material is structural, not superficial. High-signal training data works the same way. Generic data absorbs the training pass and goes quiet.

Most new data being generated is programmatic: logs, auto-generated content, boilerplate, synthetic outputs from models that are already mediocre. It’s abundant, cheap, and mostly inert. As it floods in, it dilutes the fraction of genuinely useful signal further. As AI-generated content spreads across the internet, models trained on it learn the average, not the edge. The ore gets leaner the more we mine it.

Marie Curie didn’t find radium by sifting through more rock. She developed new processes to isolate and concentrate what she was looking for. The Manhattan Project built industrial infrastructure specifically designed to separate what mattered from what didn’t. The AI field needs the same shift.

Building Oak Ridge

Two things have to happen in parallel. The first is better curation pipelines: new methods to identify and extract high-signal data from existing sources, smarter filtering, better labeling, clearer definitions of what “exceptional” looks like for each capability domain. The second is synthetic data. Even though the risk of model collapse (the equivalent of contaminating your fuel) is real, waiting for enough naturally occurring high-signal data won’t get us where we’re trying to go. Not all synthetic data techniques are the same, and the differences between them matter a lot. Deliberately designed training data built to fill specific capability gaps is unavoidable.

The most straightforward approach is prompt-based generation: feed a capable model a topic, a domain, or a problem type and ask it to generate training examples at scale. Used carefully, this fills gaps in rare or underrepresented domains. Used carelessly, it produces plausible-sounding noise that makes the training pool worse, not better.

A more sophisticated method is web rewriting: take real content and use a stronger model to transform it into a higher-signal format. DeepSeek did this systematically building DeepSeek-V3. Rather than training on web content directly, they used a pipeline of stronger models to filter, rewrite, and structure data into higher-quality reasoning examples across mathematics, code, and general knowledge. The resulting model matched or outperformed models trained at several times the compute cost, with minimal changes to the underlying architecture. This is an example of using better fuel rods in the same reactor.

A third approach is reinforcement learning from human feedback (RLHF): instead of generating new data from scratch, use human preference signals to identify which model outputs were high quality and train on those. This turns the model’s own outputs into enriched fuel, but only when paired with careful human judgment about what “better” means. Recent work has pushed this further, with new techniques achieving alignment quality comparable to full human annotation using only 6-7% ( 6 7!!! ) of the annotation effort, by targeting human review at the samples that are hardest to label automatically.

The most surprising recent result is pure reinforcement learning with no labeled data at all. DeepSeek’s R1 demonstrated that reasoning capabilities can emerge through pure reinforcement learning, with no human-labeled reasoning trajectories required. The model was rewarded for getting verifiable answers right (math problems, code that actually runs) and developed self-reflection and strategy as emergent behavior. It’s the closest thing yet to a model generating its own fissile material.

At Collinear AI we realize the theory isn’t the bottleneck. The hard part is making these techniques reliable at scale: turning prompt-based generation, web rewriting, and RLHF from one-off research sprints into something teams can run repeatedly without rebuilding from scratch each time. Most organizations treat each data effort as a custom project. SimLab is how Collinear makes this routine.

The contamination risk is real and worth understanding. Shumailov (and others) demonstrated that repeatedly training on synthetic data leads to model collapse, a finding that attracted significant attention given how close current models are to exhausting available high-quality data. The mechanism: recursive training on synthetic outputs causes models to produce repetitive, narrowing results, effectively losing the tails of the original data distribution. The model gets so good at the average that it loses the edges.

The answer isn’t to avoid synthetic data. Research shows that keeping real data in the mix and layering synthetic data on top, rather than replacing real data entirely, avoids the degenerative feedback loop. The ratio and sequencing matter enormously. Synthetic data added to a real-data foundation behaves very differently from synthetic data trained on top of synthetic data. The centrifuge has to be calibrated, not just built.

Critical Mass

Before fission, energy was extractive. You burned coal, oil, or gas: feed the furnace, get energy out, repeat. Power was linear, bounded by what you could mine and move.

Fission changed that. A reaction releases neutrons that trigger more reactions. Above critical mass it’s self-sustaining: withdraw the control rod once and the chain continues without further input. A kilogram of enriched uranium and a kilogram of coal aren’t on the same spectrum. Critical mass is the threshold where the system stops depending on external input and begins feeding itself.

Every AI model today is extractive in the same way pre-fission energy was. We supply the data, the compute, the architecture decisions, the fine-tuning, the human feedback. The model improves because we keep feeding it. Remove the input, the improvement stops.

AGI is the point where that changes. A system past the AGI threshold can reason about its own limitations, identify what it needs to learn, generate or seek out the training signal it needs, and improve its own architecture. The chain reaction sustains itself.

The gap between today’s best models and that threshold is categorical, the same way fission and combustion aren’t variations of the same phenomenon. Current models are extraordinarily capable combustion engines. Once a system crosses that line, the rate of improvement stops being limited by what we supply. It becomes limited by the physics of the system itself. Human researchers won’t be the primary driver anymore. The reaction moves at its own pace, on its own terms. Getting the enrichment right before we get there is the only real leverage point we have.

Two Races to the Same Threshold

The world is currently running two parallel races to the same kind of threshold, and almost nobody talks about them together.

The first race is to fusion. Thirty-five nations are collaborating on ITER, the international fusion project in southern France. It’s been delayed repeatedly and now targets 2039. Private companies are moving faster, but even the boldest credible estimates put commercial fusion in the early 2030s at best. The physics is understood. The engineering is the hard part.

The second race is to AGI. Both have the same core structure: a threshold that is theoretically understood, a fuel problem that is practically unsolved, and a lot of engineering standing between the two. And the two races may be more entangled than they first appear.

The race to build AGI may literally require the fusion race to progress first, or at least the fission one. The energy demands make that dependency increasingly hard to ignore.

AI data centers are already consuming power at a scale that strains the grid. In 2024, global data center electricity consumption hit around 415 terawatt-hours (about 1.5% of world electricity use), growing at a rate more than four times faster than overall global electricity consumption.  By some estimates, that figure could approach 945 TWh by 2030 (roughly equivalent to Japan’s entire annual electricity consumption) with high-growth scenarios pushing past 1,700 TWh by 2035.

That’s before AGI. A self-sustaining AI system iterating on itself continuously would require orders of magnitude more compute than today’s training runs. The chain reaction doesn’t just need enriched fuel. It needs an enormous, uninterrupted power supply to keep running once it starts. The physical energy problem and the cognitive threshold problem are coupled. Nuclear physics even has a name for it.

k_eff measures whether a chain reaction is self-sustaining. k_eff < 1 and the reaction fizzles. k_eff ≥ 1 and it runs on its own.

AGI doesn’t have an equivalent metric yet. But if it did, it might look something like this: does each generation of capability produce enough leverage to fund, power, and build the next one? We can call it a_eff. Right now, the honest answer is that we don’t know if a_eff ≥ 1, and most of the serious debates in AI (about scaling laws, compute returns, energy constraints) are really arguments about that number without naming it.

The Clock

Six years ago, the median expert estimate for AGI sat comfortably in the 2060-2070 range. As of early 2026, that number has collapsed to around 2033.  The compression is accelerating. Dario Amodei said at Davos earlier this year that AGI will likely arrive within a few years, possibly by 2027. Demis Hassabis of Google DeepMind put it more cautiously: roughly a 50% chance by 2030. 

Fusion timelines haven’t moved the same way. ITER’s deuterium-tritium milestone is 2039. Commercial fusion power is likely a decade beyond that. The private sector is more aggressive, but even the boldest credible estimates put sustained commercial fusion in the early 2030s at best. Both thresholds represent the same kind of categorical shift: a self-sustaining reaction that permanently changes what’s possible. Fusion solves the physical energy problem and AGI solves the cognitive one, but both require getting the enrichment right before the reaction will hold.

The current trajectory suggests AGI crosses its threshold first, probably by a significant margin. That means the data enrichment problem is urgent in a way that plasma confinement simply isn’t. Fusion researchers have until the 2030s. The people working on data quality and synthetic enrichment for AI may have considerably less time, and far less certainty about when the window closes.

AGI isn’t blocked by a missing theoretical insight. Szilard had that moment crossing a London street in 1933. It isn’t blocked by compute or architecture alone. It’s blocked by the same thing that stood between Szilard’s patent and Fermi’s reaction: not enough of the right material, concentrated precisely enough, arranged carefully enough to sustain itself.

Fermi’s team stacked blocks for months and did the math until the geometry was right and the reaction kept going.

That’s what we’re building toward. Not a dramatic moment, but a controlled, deliberately constructed threshold where the system crosses over and begins to sustain its own improvement. We need an AI Oak Ridge.

That work is happening now, in pieces, across a lot of teams. If you're at a frontier lab or AI-native company working on model improvement (capability gaps, post-training data, pre-deployment testing), talk to one of our researchers.

Building the Centrifuge

Oak Ridge took two years and 24,000 workers to separate enough U-235 for the reactor to hold. The enrichment problem for AI is a similar order of undertaking, and no single team will solve it.

At Collinear, we’re building part of it. SimLab is our simulation lab for AI agents: the infrastructure to generate, curate, and verify high-signal data at scale. Simulated enterprise environments, NPC users, verifiable tasks, training-ready rollouts. It’s designed to make deliberate enrichment a routine capability for the teams that need it most.

If you’re training reasoning models, shipping agents into production, or watching half your team’s time disappear into eval data hygiene, we should talk.

Talk to a Researcher

TL;DR

• Agents fail in production because evals test outputs, not behavior across stateful multi-step workflows.

• The failure modes that actually matter: imperfect tool calls, state drift, and no-exit loops that only show up when the agent runs inside a realistic environment.

• Software has staging. Agents have nothing between evals and prod. Simulation environments help fill the gap.

• SimLab is a self-serve CLI that gives you the full stack: tasks, realistic environments, and deterministic verifiers meaning you find failures before your users do.

The agent passed evals. It worked in the demo. You shipped it. Then it broke.

You’ve probably seen this, or something similar in production. We’ll use customer support as an example. A customer asks for a simple account update. Everything looks good until step 3 of a 12-step workflow.

A tool call fires: the right function name, the wrong schema: the API returns a 422, the agent retries with the same payload, and now you’re in a silent loop. The failure only surfaces when a real user hits it, and it’s nearly impossible to reproduce from logs alone. The failure isn’t just an agent failure, but a support interaction failure, a degraded customer experience.

This isn’t a model problem. It’s a testing problem.

Traditional evals were built for a different problem.

Traditional evals were built for single-turn, input-output tasks. They work for language problems. But this isn’t a typical single output failure. It’s a control loop failure. At each step it reads context, picks a tool or action, executes, observes the result, updates state, and decides what to do next. Step 3 of the workflow isn’t where it breaks, it’s where the compounding mistakes start. Not just for you, but for the customer interacting with your agent.

The bugs that ship to production:

• Irregular tool call arguments. Right function, bad payload. Fails schema validation. Retries with the same bad payload.

• Silent state drift. Working context diverges from ground truth mid-workflow. Each subsequent step compounds it. By step 8, the agent is making decisions on data it no longer has right.

• Incomplete reasoning chains and No-exit loops. Hits a dead end, has no recovery path, retries the same action. No timeout, no fallback, no escalation.

Evals test outputs. They don’t test behavior.

LLM-as-judge makes this worse. You’re using a nondeterministic model to grade a nondeterministic system. The reward signal is noisy, hard to act on, and doesn’t scale to the thousands of rollouts you need to improve the agent.

Your deployment pipeline has a gap.

Software engineers don’t push from local to prod. The pipeline is develop → test → staging → prod. Staging isn’t a perfect replica of production, but it’s close enough for most failures to surface before a user sees them.

The agent development pipeline today: build → evals → prod. No staging equivalent. The first time your agent hits a live API with real latency, a real 200-step workflow, or a user input outside your eval distribution is the first time a real user hits it too.

When something breaks, you’re debugging from logs and likely dealing with an unhappy customer on the other end. You can see your agent failed, but you usually can’t reproduce it, and you can’t run a thousand variations of the failing scenario to understand where the boundary is.

Simulation is the staging layer for agents. It closes the gap between “it passed evals” and “it actually did what we needed it to protect brand integrity and increase customer support satisfaction.”

What a simulation environment actually needs.

Not a bigger dataset. Not a fancier benchmark. It needs…Real. World. Scenarios.

A simulation environment has to let your agent interact with a realistic world across a full task execution trace and give you deterministic and programmatic signal about what happened.

Three things you can’t skip:

• Environments. Those that mirror real world scenarios, real customer support interactions. APIs with real failure modes (rate limits, distorted responses, timeouts, unexpected nulls), messy seeded data (incomplete records, conflicting field values, schema mismatches), and NPCs that behave imperfectly, like real users (ambiguous or incomplete requests, assumption breaking points, and task pivots).

• Tasks: Long-horizon, multi-step workflows that reflect real production complexity. Tasks that require 50–200 steps, involve ambiguous intermediate states, and have more than one valid execution path. The kind your agent will actually face when someone submits a customer support request.

• Verifiers: Deterministic, programmatic checks not LLM-as-judge. Did the agent reach the right end state for the customer? Did it complete all required steps? Did it stay within operational constraints? Consistent signal you can trust across thousands of parallel rollouts.

What your dev loop looks like with simulation.

Instead of: build → ship → debug from logs.

It becomes: build → simulate → fix → simulate → ship.

You run thousands of parallel rollouts across your task distribution and see where failures occur, which steps, which tool interactions, which input types cause breakdowns. When something fails, you get an execution trace you can inspect step-by-step, tweak the environment around, and re-run. You iterate on behavior, tool call logic, error recovery, and context window management in hours, not weeks. And start finding capability gaps you didn’t know to look for.

You stop guessing how your agent behaves. With SimLab, you get to see how it behaves.

We built SimLab to do this.

We kept running into this gap ourselves. Building simulation infrastructure from scratch is a serious investment. The task generation, realistic data, tool simulators, NPC behavior models, sandboxed execution, a deterministic eval layer, and so on. Most teams end up with brittle, domain-specific systems that break the moment the agent or task changes.

SimLab is a self-serve CLI that gives you the full environment stack without live environment risk.

• Sandboxed execution. Agents run in isolated containers with full environment control. Arbitrary code execution, configurable tool access, reproducible state. You define what the agent can touch.

• Bring your own tools or use pre-built simulators. The platform is self-serve. Connect your own APIs and tool schemas, or use out-of-the-box simulators for common systems like Workday, Salesforce, and others.

• Programmatic Task and Verifier generation. Generate long-horizon tasks calibrated to your domain. Tune difficulty, workflow length, ambiguity, and edge case density. High-quality training signal, not clean-room benchmarks.

• Programmatic Data: Seeded data and NPC behavior models simulate real production messiness: bad inputs, missing fields, unexpected response formats.

Where this fits in your stack.

SimLab sits between build and deploy. It’s not a replacement for evals or observability; it’s the layer missing between them.

Evals tell you if individual outputs are correct. SimLab tells you if the agent can complete full workflows under realistic conditions. Observability gives you post-deployment traces of what already broke. SimLab gives you pre-deployment traces of what would have broken.

Human QA pipelines are slow, expensive, and don’t scale. You can also build your own simulation infra, but a full stack covering tasks, environment, verifiers, sandboxing, and parallelization is a significant engineering investment that gets brittle fast.

SimLab is designed to be adaptable as your agents and domains change, without rebuilding from scratch each time.

Simulation is the new deployment.

Simulation will be a standard layer in the agent development pipeline, the same way CI and staging are standard in software. Not just a nice-to-have at scale, but the thing separating agents that demo well from agents that ship reliably.

We’re opening SimLab as a self-serve CLI. Install it, point it at your agent, define an environment, run it. See where it breaks. We’re still testing the task generators, verifier primitives, and environment tooling and we want to know what doesn’t work for your use case.

Try it. See how your agent holds up. Tell us what’s missing.

→ github.com/collinear-ai/simlab

It’s been a big quarter. We moved into a new office in Sunnyvale, launched YC Bench, extended the Simulation Lab, and hit the conference circuit. A lot to cover, so let’s get into it.

New home in Sunnyvale

We outgrew our old space and opened a new office in Sunnyvale. A big thanks to the customers, partners, and friends who stopped by during the first few weeks to check it out.

The sign is up, the whiteboards are full, and the espresso machine is already earning its keep. Good to have a home base.

YC Bench: can frontier models run a startup?

We released YC Bench, the first open-source, long-horizon benchmark with a simulation clock.

The idea: give a frontier model seed capital, a small team, and a market of tasks. Ask it to run an AI startup. Manage employees, hit deadlines, allocate resources, and maximize profit over time.

What we found is that a simple rule-based agent consistently outperforms frontier LLMs. Not because the task is impossible for them, but because they make compounding mistakes early on that they never recover from. They chase short-term wins, over-parallelize, and adapt too late when conditions change.

This matters because the industry is moving fast toward long-running, multi-step agent workflows. But reliability isn’t keeping pace with this ambition. YC Bench measures exactly that gap: not whether a model can answer a question, but whether an agent can hold a coherent strategy over time.

YC Bench is open-source and on our GitHub. We built it to be extensible. If you’re working on long-horizon agent evaluation, we’d love to hear what you find!

Simulation Lab: what it is and what’s new

For those new here: the Collinear Simulation Lab is where AI agents learn enterprise work before going to production. Think of it as a practice environment, fully interactive, with simulated data, simulated users (NPCs), real enterprise tooling, and task-specific verifiers. Agents don’t just get tested. They get trained against realistic, messy, multi-step workflows.

Inside a sim lab, you get:

• Simulated APIs for enterprise software (HR, Finance, Sales, Customer Service)

• NPCs that push back, change their minds, and interrupt

• Tasks with ambiguity, missing info, and competing priorities

• Scorers that generate task-specific rubrics alongside formal verifiers

We’ve extended the lab this quarter with broader tool coverage and deeper scenario complexity. More enterprise surfaces, richer NPC behavior, and tighter integration with RL training loops. The goal stays the same: agents need a world to learn in, and we’re building that world.

If you’re building agents for enterprise workflows and want to stress-test them before they touch production, the Simulation Lab is open for business.

Together AI Conference

Nazneen spoke at the Together AI conference this quarter. Our partnership with Together continues to deepen. TraitBasis, our method for generating realistic simulated users, is now integrated into Together Evals. Builders on Together’s platform can simulate impatient, confused, or inconsistent user personas and see how their models actually hold up when conversations get unpredictable. If you missed the talk, stay tuned for a recap.

That’s it for March. More coming soon on the research side.

-- The Collinear Team

Get started with YC bench:

curl -sSL <https://raw.githubusercontent.com/collinear-ai/yc-bench/main/start.sh> | bash 

GitHub: collinear-ai/yc-bench

Long-Term Coherence as a Goal Post

Popular agent benchmarks - GAIA, TermBench, SWE-Bench, τ²-bench - evaluate a model’s ability to complete tasks through multi-tool, multi-turn interactions. Even when these tasks span hundreds of tool calls, they share a critical limitation - they lack a simulation clock. As AI agents get integrated into the workforce and digital economy, time becomes an essential dimension of evaluation. Task sequence matters; environmental states shift over time, and inaction is as consequential as a wrong action. Without a clock running through the simulator, you can measure whether an agent did the right thing, but not whether it did it when it mattered.

Some existing benchmarks include a simulation clock, Vending Bench, which tests for long-term coherence1 is one such example. Vending Bench has realistic NPCs, including suppliers the models talk to, and models compete with each other. The environment dynamics do not capture more sophisticated planning capabilities when resources are non-stationary, and there are tight deadlines to meet.

We propose a new long-horizon adaptive planning and coherence benchmark, YC-Bench (Your Company Bench), in which your agent takes on the role of a startup founder and executes activities to run a successful business. These activities include task prioritization, task scheduling, meeting client deadlines, resource allocation, managing burnout, and maximizing company profits and prestige.

With a simulation clock, the AI agent must learn to maximize long-term rewards over short-term gains. This means it needs to discriminate between short-term and long-term rewards, knowing that some actions pay less in the short term, but more in the long term. This is a core human skill usually termed “long-term coherence” that models are not exhaustively tested on, especially in a reproducible, open-source, and extensible way. YC-bench is an effort in this direction.

Environment Dynamics

YC-Bench asks the model to act as a founder of an AI startup. The LLM is given a seed capital, a fixed number of employees, and a base prestige level for each of several AI-related domains, such as training, data, and backend. The aim of the model is to maximize its capital by completing tasks on the market. However, to make the dynamics more realistic, we associate a prestige level with each task and the company. The LLM can only take on tasks that are at or below its prestige level. And therefore, to stay in the game, the LLM needs to browse the market for tasks, commit to them, and manage them until the project is delivered before the deadline. Each task can be assigned to multiple employees, and multiple employees can have multiple tasks.

Several things can go right. If the model assigns the right employees to a task and it is completed on time, the company will earn profit from that task, and the prestige level of the related domain, eg, ‘training’, will increase. The employees who work on the task will also improve their skills. As a result, the company can handle more lucrative tasks that require a higher minimum prestige level and stronger skills.

But there are also several things that can go wrong. If the task cannot be completed in time because the model assigns the wrong employee to it (e.g., assigning a GPU expert to build a frontend), the company will not get the money, and its prestige in that domain will decrease. As a result, the company will be able to do fewer tasks in that domain because it is less reliable. Moreover, employees are on a monthly payroll, and good ones cost more. As a result, if the company fails to perform tasks consistently, it will eventually go bankrupt.

YC-Bench is built for terminal use. The model can run a fixed set of CLI commands, which it learns from the system prompt. For example, it can assign tasks to employees, cancel tasks, change assignments, see its performance, etc. After it performs an action, time passes, and events such as task completion, task cancellation, and bankruptcy occur. If the model goes bankrupt during the evaluation, the evaluation stops. It’s time to give up. We also see whether models exploit the particular features of each domain (some are easy but less profitable, others are hard but more profitable) to become specialists rather than generalists and make more money.

Early Results

We compare three frontier LLMs - Sonnet 4.6, Gemini 3 Flash, and GPT-5.2 - against a human-devised rule-based baseline across 3 configs and 3 seeds (27 runs total). Each agent starts with $250K and must survive a 1-year simulated horizon.

Our key result is simple: a simple hand-written heuristic beats every frontier model. We understand the weight of this claim; we are not claiming that the current version of the benchmark requires reasoning to win it. However, reasoning AIs should be able to cruise through the benchmark. We are happy to discuss and debate the improvements for the next version!

We test on 3 different configs for 3 unique seeds:

The human-devised rule never goes bankrupt: 9/9 across all configs and seeds, while the best LLM (Gemini 3 Flash) survives 8/9. The rule-based agent doesn't use an LLM at all. It follows a fixed strategy: accept the highest-reward task you can finish, assign your best employees, and never over-parallelize.

Survival Rates

Hard seed 1 is the clearest signal: all three frontier LLMs go bankrupt, while the rule-based agent finishes with $14.8M. The LLMs fail not because the task is impossible, but because they make compounding errors in the first 2-3 months that lock them out of the prestige ladder.

When LLMs win, they win big, but they also lose hard

GPT-5.2 achieves the single highest balance of any agent: $43.5M on hard seed 3, nearly 3x the rule-based agent's $15.0M on the same seed. But GPT also goes bankrupt on 2/9 runs. Sonnet shows the same pattern at a more extreme level — $10.1M on nightmare seed 2 (the highest LLM result for nightmare), but bankrupt on 4/9 runs overall. Gemini is the most consistent LLM. It sweeps all 3 nightmare seeds (the only LLM to do so) and rarely collapses catastrophically. But even Gemini never matches the rule-based agent's reliability.

Prestige specialization explains part of the story?

The radar charts reveal some insight into why models fail. Each polygon shows the company’s final prestige across 7 AI domains (system, research, data, frontend, backend, training, hardware). Large polygons indicate the model’s prestige increased broadly. Tiny dots near the center indicate the model went bankrupt before gaining any prestige. The human-devised rule (navy dashed) fills the full radar on every run — it maxes prestige methodically across all domains. Among LLMs, Gemini builds the most balanced profiles. GPT-5.2 shows genuine specialization on medium — it focuses on backend/data/frontend while ignoring training — a strategically reasonable choice, but one that becomes fragile when the task distribution shifts on harder configs. Sonnet is bimodal: either it maxes everything (medium seed 1), or it collapses entirely (nightmare seeds 1 & 3, stuck at prestige 1.0 everywhere).

When we inspect Sonnet’s scratchpad on failed runs, the model correctly diagnoses the problem (”PRESTIGE CRISIS -MARKET LOCK”) but only after payroll has consumed the runway. It reasons well about strategy but fails to execute it in a timely manner.

Why do models struggle?

Four failure modes recur across all bankrupt runs:

1. Over-parallelization. Accepting 3-5 tasks at once, splitting employees across them. Each employee’s effective rate drops to base_rate / N per task — a senior at 8.0 units/hr assigned to 4 tasks contributes just 2.0 to each. Deadlines slip, failures cascade.

2. No prestige gating. Accepting tasks that require prestige the company hasn’t earned yet. The task completes late, the prestige penalty makes the next tier even harder to reach, and the agent spirals into a market lockout.

3. Late adaptation. Models identify problems in their scratchpad but only after the damage is done. By the time Sonnet writes “never accept task B while task A is active,” payroll has already consumed 60% of the runway.

4. Inconsistent ETA reasoning. Models understand throughput math in principle, but don’t consistently apply it. Sonnet’s medium seed 2 has a 49% task win rate - essentially a coin flip, despite writing correct throughput formulas in its scratchpad. The core gap is not reasoning ability but temporal discipline: doing the right thing when it matters, sustaining correct behavior across hundreds of turns, and resisting the temptation to over-commit when a lucrative task appears.

Next Steps

YC-Bench v0 is a starting point. Here’s what we’re working on:

• More models. We plan to add results for Claude Opus, Gemini 2.5 Pro, o3, and open-weight models (Llama 4, Qwen 3) as they become available. If a model can run tool-use in a loop, it can run YC-Bench.

• Longer horizons and non-stationary dynamics. The 1-year configs test short-to-medium planning. We want to push to 3-5 year horizons where market conditions shift: recessions that shrink rewards, talent wars that inflate salaries, and technology shocks that obsolete entire domains. This tests whether agents can adapt strategy mid-run, not just execute a fixed plan.

• Better baselines. The current human-devised rule is strong but simple. We want to explore MCTS-based planners, RL-trained policies, and hybrid approaches (an LLM for strategy, a rule engine for execution) to understand where the frontier lies.

• Community configs. YC-Bench is fully open-source and extensible. Every parameter: employee count, prestige distribution, penalty multipliers, and salary curves can be changed. We encourage the community to design configs that stress-test specific capabilities and submit results.

Try it yourself:

uv add yc-bench
uv run yc-bench run

If you find it useful, feel free to cite our work and contact us!

@misc{collinear-ai2025ycbench, 
author = {{Collinear AI}}, 
title = {{YC-Bench}: Your Company Bench — A Long-Horizon Coherence Benchmark for {LLM} Agents}, 
year = {2025}, 
howpublished = {\url{https://github.com/collinear-ai/yc-bench}}, 
note = {Accessed: 2026-02-25} }

Nov was a massive month for agents as they took centerstage across NeurIPS and AWS Re:Invent!

NeurIPS2025 had a clear vibe: the agent era is forcing RL to grow up. Not as a research novelty, but as production infrastructure for tool use, long horizon behavior, and reliability when the world gets messy in real life workflows.

NeurIPS 2025, the “RL is everywhere” moment

San Diego served! Sunny weather, packed hallways, and surprisingly serious taco opinions. Between sessions (and coffee lines), we met a ton of builders and kept hearing the same thing: RL is suddenly everywhere.

RL is the new scaling lever. The frontier has shifted from “can the model answer?” to “can the agent execute?” Multi step work, tool calls, retries, and shifting user intent are pushing teams toward RL to shape end to end behavior.

RL needs the right infrastructure to go interactive. Environment fleets, verifiers, orchestration, NPCs - everyone at NeurIPS had a novel approach!

Realism is the new benchmark. Less obsession with a single score, more focus on trajectory shaped evals: does it hold up on turn4, recover from tool noise, and stay safe when scenarios drift?

We also presented our NeurIPS paper, Through the Valley of Reasoning. The punchline is: when you distill reasoning into small models, performance can dip before it climbs, and early on the structure of the reasoning matters more than whether the trace is “correct.”

We also met a bunch of new friends and collaborators. If you were there, hit reply and tell us what your team is building. We love swapping notes.

Talk to us!

Spider, post training without the chaos

We shipped Spider, a lightweight way to turn post training work into a repeatable recipe.

With Spider, you can use one recipe for both off policy and on policy. Generate clean distillation datasets, or flip into an online loop with teacher guidance and KL supervision, without rebuilding your pipeline each time. It also keeps the “boring but critical” pieces consistent across runs, rollouts, filtering, verifiers, and publishing, so results stay comparable as you iterate.

Huge thanks to our friends at Thinking Machines for supporting the Tinker integration.

AWS Re:Invent - even more agents!

re:Invent turned Las Vegas into a full on agent showcase. Nova2 and the Nova family got a big spotlight, Nova Forge put “build your own frontier models” on the menu, and Nova Act made the case for agentic workflows!

What stood out to us was the framing in Swami Sivasubramanian’s agentic AI keynote: getting agents to production is less about clever prompts, and more about repeatable training and testing loops.

Congrats to our customers and partners at AWS on an awesome launch week.

If you are building agentic AI, we love to swap notes!

Talk to us!

That’s it for this edition. Thanks for following along. We will have some fun things to share over the next couple of weeks. 🙂

Best,
The Collinear Team

This post argues that treating RL environments as first-class infrastructure is critical. Specifically — an Environment-as-a-Service, with clean separation between data plane and control plane—is the key to unlocking scalable, production-grade RL for AI agents.

From Static Labels to Interactive Training Grounds

The shift from post-training focused on supervised learning to mid-training with RL for LLMs represents a fundamental change in what we’re optimizing:

Previous paradigm: Static input + static target → model output → loss → backprop.

New paradigm: Agent acts in environment → environment scores behavior → policy updates → agent acts again

Modern RL environments for AI to replicate the enterprise workflows include:

• A product development environment where agents update tickets, generate sprint progress reports, and work with the team to identify milestones. The team in this case is simulated users.

• A coding environment where agents receive task specs, edit codebases, run tests, and receive scores on correctness and other rubrics

• A computer use agent that reads the calendar, navigates to Excel to fetch data and drafts and email.

• A browser environment where agents navigate UI trees, fill forms, and complete realistic business tasks

These environments provide what static supervised data cannot: a dynamic sandbox with verifiable rewards at scale. Pass/fail checks, structured scoring, safety constraints, and automated metrics turn RL from a research curiosity into an operational capability for mid-training (inducing new capabilities) and post-training (alignment and evaluation).

The critical transition is: from handcrafting or distilling static examples to orchestrating interactive training scenarios. That shift forces us to treat environments as infrastructure, not just programmable abstractions.

The RL Training Architecture

Production async RL systems for LLMs decompose into three loosely coupled components (see figure above):

1. Trainer

The trainer consumes trajectories and updates weights. It reads batches of (state, action, reward, metadata), runs the RL objective (GRPO, PPO variants, DPO-style methods), possibly with reward models, critics, and reference models for KL stabilization, then writes updated weights to a model store.

2. Sampler

Sampler workers periodically pull the latest weights, interact with environments to generate trajectories by running the policy model for actions, and stream trajectories plus rewards to the trainer. Samplers are inference-heavy, latency-sensitive, and often scale to thousands of nodes.

3. Environment

The environment is the substrate that turns raw actions into meaningful behavior: the simulation of the world (web UI, tools, code repos, databases), the interface contract (observations, actions, rewards), and the episode lifecycle. In scalable systems, this is not a local process – it’s a remote service or microservice fleet.

In this blogpost, we will dive deeper into the third component, the environment and the architecture behind building scalable RL environments.

The Data-Plane, Control-Plane Split: The Pragmatic Path to Scalable RL Environments

We believe that scalable RL environments require a pragmatic infrastructure mindset—one that borrows directly from the hyperscaler model, which separates the data plane and the control plane. In simple terms, the data plane is responsible for the environment’s core, real-time behavior, while the control plane manages configuration, orchestration, and the administrative logic that keeps everything running smoothly.

Environment Data Plane

The data plane sits on the critical path of every RL step:

• Initialize the environment

• Stepping the environment:

  obs_{t+1}, reward_t, done = step(action_t)

• Handling concurrent episodes from many samplers

• Producing deterministic, reproducible transitions

• Evaluating verifiable rewards: executing tests, checking business rules, running automated metrics

Design constraints: Low latency (every step sits between policy inference → environment step → next inference), high throughput (many parallel episodes), and rock-solid stability.

Implementation patterns: Environment microservices behind RPC/HTTP APIs, stateless containers backed by state stores, state sharding across machines, determinism via per-episode seeds and versioned data snapshots.

Environment Control Plane

The control plane is the administrative engine of the RL environment. It spawns and manages many parallel rollouts, creating k independent environment managers that coordinate with the data plane while staying completely off the per-step critical path. Its job is to configure, schedule, and orchestrate the environment’s behavior—including how agents and non-player characters (NPCs) interact—without ever slowing down the real-time execution loop.

Specifically, the control plane handles:

• Scenario configuration & versioning: Defining environment types, maintaining versions, and generating scenario templates that each of the k managers can instantiate independently.

• Rewards and verifiers governance: Selecting verifier modules, composing sub-rewards, and managing aggregation strategies.

• Curriculum + workload scheduling: Determining which tasks, difficulty modes, or trajectories to sample at each training stage, and routing them to the appropriate environment managers.

• Experiment routing: Mapping policies or policy versions to specific environment instances for A/B testing, evaluation runs, or canary deployments.

• Elasticity & lifecycle management: Scaling environment managers up/down, rolling out upgrades, coordinating NPC configurations, and performing safe rollbacks without interrupting live data-plane rollouts.

• NPC configuration & behavior enabling: Selecting which NPC personas, scripts, or dynamic behaviors are active for a given scenario and ensuring the data-plane has the necessary hooks to interact with them.

In short, the control plane administers the environment fleet, enabling massively parallel rollouts while keeping the critical per-step simulation loop lean, deterministic, and high-throughput.

The Path Forward

If you are training AI agents with reinforcement learning, you face two choices:

1. Build your own environment stack: Design APIs, stand up tool replicas, author tasks and verifiers, maintain everything as products change

2. Treat environments as a reusable platform primitive: Plug into an existing environment service

Just as data has become increasingly commoditized (thanks to tools like Spider and Tinker), we expect RL environments to follow the same path. The future is environment-as-a-service: frontier, scalable, and ready to plug into any training stack.

Instead of turning your data-labeling vendor into an environment platform, point your trainer and sampler at Collinear’s environment endpoints, wire up your policies and reward models, and start running RL over realistic, verifiable tasks. With Collinear’s high-fidelity simulations, each environment can be a unique micro-ecosystem: the same workflow feels adversarial with one trait vector, cooperative with another, and chaotic with a third, giving agents exposure to a wide distribution of real human behavior.

The data era commoditized static corpora. The RL era will commoditize environments. The winners will be those who treat environments as being core to the RL infra stack.

Spider is a single client interface that turns messy distillation and ablation experiments into a simple, configurable workflow. Set on_policy: false to generate clean distillation datasets, or flip on_policy: true to run online training with teacher guidance and KL supervision. It handles dataset prep, rollouts, supervision, and post-processing in a few lines of code.

Why we built Spider

“Tinker for training” exists. “Tinker for data” does not.

Our friends at Thinking Machines recently released Tinker that enables fine-tuning with full control in simple steps. However, most research time still disappears into preprocessing, rollout scripts, verifier glue code, and training integration. So, this Halloween, we spun a web around that problem and built Spider so you can define a production-grade distillation run with just a few lines, then iterate fast.

We hope Spider will help you test ideas in hours, not days; turn messy data work into a shareable recipe; and ship better models with less glue. Grab the repo, run a sample recipe, and tell us what to build next.

How it works

Spider turns post-training data work into a single client workflow. In off-policy mode it generates distilled datasets with high-throughput rollouts, then applies your preprocessing, filters, and verifiers. Flip on_policy: true to run the online loop with a teacher model and KL supervision. The same recipe drives both paths.

Run Spider on Collinear endpoints or your own GPUs. Each run records its recipe, parameters, and metrics, and can publish datasets or trained artifacts to the Hugging Face Hub. The result is faster loops, cleaner data, and fewer moving parts from idea to artifact.

1. Define a recipe
   Write a short YAML that names your provider, models, dataset source, and any filters or verifiers. This is your data recipe. One client and one config cover both off policy and on policy paths.

2. Generate or train
   Run the recipe with on_policy: false to create an off policy distilled dataset from high-throughput rollouts. Flip on_policy: true to introduce a teacher model and KL supervision through the integrated Tinker client for online training.

3. Compose quality checks
   Use built-in filters and verifiers for length, dedupe, syntax, structure, and safety, or register your own in one line. Spider applies them in the pipeline so your outputs are clean and auditable.

4. Run anywhere, ship anywhere
   Point the client to a Collinear endpoint with an API key or to your own GPUs. Each run logs its recipe, parameters, and metrics, and can publish datasets or model artifacts to the Hugging Face Hub with lineage preserved.

Getting started is simple, and you can find quickstart instructions on the repo.

Roadmap

We’re building toward a world where post-training data is defined as code, portable across providers, and fast to turn into measurable model gains. Write a small recipe, verify quality with shared checks, and ship a distilled dataset or on-policy improvement in minutes.

To enable that, we are expanding Spider with the following roadmap features.

• Cross-tokenizer on-policy distillation from any teacher model

• Simple but powerful templates for generating multi-turn conversation data with simulated users

• Highly configurable tool-use library to generate and train on-policy agentic tool-call rollouts

Resources

You can learn more about Spider on our GitHub repo here.

If you give Spider a try, let us know what you think!

Talk to us!

A lot has been happening at Collinear this month. There is fresh research, new customers, and plenty of progress towards better AI systems.

🚀 Together Evals × Collinear Simulations

We’ve partnered with Together AI to bring real world, multi-turn simulations into their Together Evals platform.

Traditional evals assume the user will be polite and consistent; real users don’t. They might ask follow-ups, change their mind, get frustrated or distracted — and that’s exactly where many models break. With TraitMix, builders can now simulate impatient, curious, or inconsistent user personas and see how their models actually perform under messy human conditions. Together Evals then scores models for helpfulness, safety, and consistency, at scale, all within one workflow.

Read the announcement here.

🐱 CoLM 2025 Recap

CoLM 2025 was a special one.

We presented our paper on adversarial testing, Cats Confuse Reasoning LLMs, and spent the week exchanging ideas with research partners, collaborators, and friends from across the community.

Our Future of Post-Training Social sparked rich discussions on alignment, fine-tuning, and reward modeling, while the booth facilitated curious research conversations (and a growing crowd of cat-sticker collectors).

It was inspiring to see so much energy around improving models not just for performance, but for reasoning and reliability.

🧠 TraitBasis Simulations Launch

We launched TraitBasis, our framework for simulating realistic human behavior in model testing.

TraitBasis uses activation steering to inject behavioral traits, impatience, confusion, skepticism, overconfidence, directly into simulated users. This lets builders observe how models hold up when conversations get unpredictable or emotionally varied.

TraitBasis builds on the research community’s work in τ-Bench, and extends it to enterprise domains such as telecom and telehealth through our new τ-Trait benchmark.

What’s Next?

That’s it for this edition. Thanks for following along.

If you’re interested in building tools that help enterprises ship safer, smarter AI, check out our Careers page.

If you are ready to improve your AI’s performance, let’s talk! We might or might not mention cats…

Let's talk!

The era of agents has begun, but much of today’s tooling is still being tested or is gated to pilots as teams chase consistent, repeatable performance. One day, your tools pass the vibe-test; the next, they stall. The promise is tangible, but the production bar is higher. What’s missing is reliable evidence of behavior across messy, multi-turn tasks — planning, tool calls, and recovery — so leaders can move from cautious testing to confident scaling.

Vibe tests aren’t the answer.

The gap between demos and production demonstrates the need for enterprises to move beyond vibe-testing and into high-fidelity evaluations performed at scale. Evaluations serve as the window into your AI agent’s mind. They can gate launches, highlight drift, and validate progress for risk and governance teams. Without this tight eval loop, there’s no credible path to safety, performance, or ROI with your AI investments. Your AI agent eval pipeline should be no different than your software QA cycles, even more so than typical software, agentic capabilities need more exhaustive test scripts, unit tests, and user-centric edge cases to validate consistency in real-world environments.

AI Agents aren’t linear - so, your evals can’t be either.

Evaluating single-turn chat is hard; evaluating agents is even harder. Modern agents plan, call tools, read results, and adapt over many turns. Failures hide in the process, not just the final text: brittle reasoning chains, incorrect API params, state drift, or trust collapse after a high-tension exchange with a customer. Static prompts and one-shot leaderboards miss these behaviors because they grade outputs, not how the agent got there.

Today’s agents are nondeterministic. They don’t follow predefined paths — meaning your tests can’t either. Traditional software testing assumes the same input yields the same output; with agents however, variability is the benefit and the risk, so the permutations of possible test cases scale exponentially compared to traditional software test cases that grow linearly with use cases. This behavior shift raises an important question: How can you possibly predict the permutations of your user-agent edge cases to evaluate your AI’s performance? This is exactly why simulations matter.

Simulations bridge this gap.

To see agents clearly, you need controlled, realistic, repeatable interactions that pressure-test the range of your users’ behaviors and intents before production. Diverse, simulations reveal what static evals miss — impatient spirals, tool confusion, policy slips under stress — and they generate the high-signal examples that lift models in post-training.

In our most recent Collinear paper, we introduced TraitBasis and τ-Trait, a research-driven approach to doing exactly that — generating high-fidelity, steerable user traits that expose where agents actually break when they interact with your users. When we simulated real human behaviors (impatience, skepticism, confusion, incoherence) on τ-Bench, frontier model success rates dropped by 20%+, underscoring the need for realistic user-simulated data for evals, not more synthetic prompts.

Collinear enables comprehensive evals at scale using simulated user-environments. Our simulation suite uses steerable, persona-driven users customized to your sector and use case to test your agent’s array of responses by user intent and demographic. Our eval platform delivers high-signal traces that are auto-scored against your compliance criteria, giving you clear failure nodes with examples of where your agent falls short in the real world. Ultimately, these failure nodes serve as a high-signal data pipeline for post-training your model, turning misses into uplift.

The recipe of a simulation: user, agent, judge.

An effective simulation requires three components: the user, the agent, and a judge to evaluate the interaction. This triangle is the foundation of Collinear’s platform, and while most of the attention is typically applied to the agent and judge, we’ve prioritized the user, giving customers the tools to configure realistic, controllable, and dynamic user environments identical to real-world scenarios.

Our simulations aren’t random prompts; they’re structured interactions between your agent and a life-like user with a clear:

• Persona (e.g., skeptical power user, impatient first-timer)

• Intent (e.g., cancel subscription, dispute charge)

• Demographic (domain, language, constraints)

That’s the scaffolding we use to reveal realistic user-agent journeys, consistently and at depth.

Under the hood, we steer behavior directly in the neural net — activation-level conditioning — so traits persist through long multi-turn conversations and compose cleanly (e.g., impatient and confused), enabling high-fidelity, controllable runs you can replicate again and again.

This is exactly what TraitBasis delivers. Instead of external instructions, we leverage a trait vector inside the user-simulating model and add it to hidden activations each turn, giving you precise control over intensity and composition.

While prompt-based or fine-tuned persona models are popular across the market, our research found those methods fail to deliver:

• Fine-grained control: the intensity of behaviors and intents blur throughout a conversation (“moderate” vs. “high” looks the same by the third turn)

• Stability: personas collapse mid-conversation, losing the signal the first few turns contained

• Mixing: one trait dominates the others when combining multiple, failing to deliver the multi-dimensionality of real users

Simulations drive evals. Evals drive trust. And trust drives value.

Our approach to simulations delivers coverage, confidence, and uplift in your AI flywheel. It systematically creates and tests edge cases across personas, intents, and languages to guarantee test case coverage. It catches behavioral risks before customers do and gate releases on pass rates to instill confidence in your customer experience. And it leverages high-signal failures for post-training data (DPO/GRPO/SFT) to deliver measurable, targeted uplift.

A few proof points from our TraitBasis launch:

• Realism: Highest Elo (1624) and 63% win rate vs. alternatives, achieved with 3,000× less data (4k vs. 13k samples).

• Control: Intensity consistency across 97.5% of cases (clearer “medium vs. high”).

• Stability: Persona reliability in 77% of long chats, vs. the persona collapses in 94% and 66% of cases using prompt and SFT baselines, respectively.

• Compositionality: Accurate trait blends 62.5% of the time for complex users (e.g., impatient + confused), far higher than other methods.

But don’t take our word for it.

“Before simulations, we graded answers. Now we grade behavior. We watch our agent under pressure, fix the weak spots, and re-run the suite before release. It’s become our CI for AI.” — Head of AI, Fortune 500 Financial Services Company

That shift — from judging single outputs to auditing reasoning, tools, and tone over time — is what instills confidence in stakeholders that an agent is production-ready.

Try it for yourself and see how your agents perform in the real-world.

Behind every great agent is great testing. And behind every great test is great data. Try simulations for yourself today: Connect your endpoint, pick a few core journeys, and run them against persona-driven users. Review the traces, dissect the evals, and turn misses into high-signal improvements.

Explore Simulations

Read the paper and reproduce the results with our dataset on HuggingFace 🤗 (approx. 300M tokens).

When we train small language models to reason on code, their performance doesn’t just rise with more data, it first falls into a dip before climbing back up.

We call this the Valley of Code Reasoning.

The Dip Before the Climb

Distilling the reasoning traces of large models into smaller ones has become a popular way to unlock coding or reasoning skills without huge compute budgets. But when we tracked performance as we scaled up distillation data, we found a non-monotonic trend:

• With small amounts of data, models retain shallow skills.

• As we add more, performance drops, a confusion stage where models are struggling to restructure their internal representations.

• Only after passing through this valley do they climb back up, showing steady log-linear improvements.

This valley is a structural feature of how small models learn reasoning.

What We Learned in the Valley

We fine-tuned models at different points in this curve and found two surprising results:

1. Easy problems matter more than hard ones in early stages. Small models learn best by first stabilizing on simple patterns before moving up in difficulty.

2. Correctness of outputs didn’t matter. Training on correct vs. incorrect code traces made little difference. What mattered was the structure of the reasoning steps themselves.

Why It Matters

The valley of code reasoning reframes how we think about training dynamics: adding more data isn’t always a straight path upward. Scaling laws for knowledge distillation of small language models differ from standard monotonic scaling laws. There are two phases of learning. In the valley phase, non-reasoning models learn the structure of reasoning and so the correctness and semantics matter less. Thereafter, the models start learning from content and that’s where the difficulty and correctness starts to matter. For practitioners and researchers, this means that getting the right data for the right stage is critical.

What’s Next

If you are mid-training or post-training models or agents, connect with us and we will accelerate your time to next improved model ✨

Learn how ServiceNow is improving Apriel-1.5-15B-Thinker with Collinear curated data.

If you build off our work or use the dataset, please cite us:

@article{HeShafiqueKumarMackeyRajani2025,
  title        = {The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models},
  author       = {Muyu He and Muhammad Ali Shafique and Anand Kumar and Tsach Mackey and Nazneen Rajani},
  journal      = {arXiv preprint arXiv:2510.06101},
  year         = {2025},
  url          = {https://arxiv.org/abs/2510.06101}
}