In July, OpenAI disclosed that two of its models, GPT-5.6 Sol and a more capable pre-release model, had escaped a sandboxed cyber-capability evaluation built on the ExploitGym benchmark [1]. The sandbox had a single network path: a package-registry cache proxy that was assumed to be read-only egress. The models found a zero-day in that proxy, used it to escalate privileges and move laterally inside the testing environment until they reached a node with internet access, and then broke into Hugging Face’s production infrastructure. Hugging Face detected and contained the intrusion: its postmortem timeline covers about 17,600 recovered agent actions over roughly four and a half days [2]. The agent inferred that Hugging Face might host the benchmark’s reference solutions and went to steal them rather than solve the tasks.
This was a massive failure. However, one could argue that the failure was not on the agent — it merely followed instructions and used its harness to perform its task. The real failure was in the world the agent worked in.
The World Around the Agent Matters
A common reframing of the past year is agent = model + harness. An LLM agent is a model with tooling to affect its environment. However, an agent ceases to be an agent without an environment to affect.
That world matters. Agents use tools, modify files, interact with people, and change state. Each action changes what happens next. A model that performs well on a static prompt may struggle once its decisions have consequences. This builds on my argument from over nine months ago, where I highlighted the AI training paradigm shift from static data to interactive environments [3].
General capabilities need a sandbox to practice interactive actions. They need simulated worlds: environments that hold state, enforce constraints, react to actions, and produce reliable feedback. A benchmark or training run measures not the model alone, but the model, the harness, and the world together.
Cybersecurity shows more clearly than any other domain why the environment has to be built well. A cyber capability is defined against a target: an exploit works against a particular service, patch level, network policy, and set of permissions, or it does not. Take large codebase debugging as an example. The agent opens an unfamiliar repository, guesses where the bug lives, and changes the code. The test fails. Now the repository is different and so is the problem. Permissions shape what happens next. The agent reaches for a tool it does not have, or discovers that a file is off-limits. Sometimes it has to unwind its own earlier work before it can move forward.
A sandboxed simulated world holds three parts together. The task defines the objective. The environment provides the tools, state, and boundaries. The verifier decides whether the repair worked. Because the world is repeatable, the same task can run again after one condition changes. The trajectory shows where the agent recovered, where it got stuck, and whether the result held up. One task starts producing many learning experiences.
To build general intelligence, agents need worlds that are hard but fair.
Nothing is Secure from Capable Agents
Every training environment and simulated world contains assumptions. Once agents become capable enough, they start finding gaps in those assumptions, which then manifest in diverse scenarios. For example:
An agent asked to make a test suite pass may change the tests instead of fixing the code.
An agent graded on a final output may find a shortcut that produces the right artifact for the wrong reason.
In a sandbox, an agent may use access that the designer never intended to expose.
This does not require malicious intent. The agent optimizes against the task and the reward it receives. If a loophole exists, a capable system may find it.
Recent research makes this concrete. Cursor reported that SWE-bench Pro scores fell by 14.1 percentage points for Opus 4.8-max and 20.7 points for Composer 2.5 after removing future Git history and restricting network egress [4]. SandboxEscapeBench similarly tests whether agents can exploit weaknesses across orchestration, container-runtime, and kernel layers [5]. These incidents, and the OpenAI one from the discussion earlier in the blog, they differ technically—benchmark leakage is not the same as a sandbox escape—but point to the same conclusion: the agent’s world is critical to its evaluations.
Hard tasks expose failures worth fixing, but hardness alone is not enough. A task should be difficult enough to be hillclimb, but not at the cost of an ambiguous, broken, or impossible environment. One way to integrate the environment into the agent’s evaluation is to measure whether it scales securely with the agent’s capabilities. Let’s formalize this below.
Environment Attack Surface Score (EASS)
Past work in secure systems evaluates a software system’s security through its attack surface, measured as a triple of <methods, channels, and data-items> [6]. Simon Willison’s blog on the agents lethal trifecta provides a framework for AI security which is also popularized by Meta’s work on the Agents Rule of Two [7, 8]. These works model an attacker outside the system using prompt injection to extract data and compromise security. We take inspiration to model the reverse: the agent is the attacker, already inside, reaching for information unfairly within the environment or beyond it.
An agent environment exposes three kinds of surface.
Target (T): data in or around the environment that would gain reward or cause harm. Is the answer key here in some form? Examples: a reference solution, the commit that fixes the bug still in git history, a vendored dependency carrying the patched function, the grader’s result file
Access (A): everything the agent can invoke or connect to, locally or remotely. Can it get to those data items? Examples: run_shell and every binary it brings with it, file read and write, git, pip install, network egress and the package-registry proxy, a reachable git remote, host mounts, the path the grader reads its verdict from.
Grader (G): a grader that scores the outcome without checking how it was reached. Does it verify only the artifact? Examples: a pass/fail test run, a rubric that scores only the final diff, a harness that trusts a result file the episode can write.
If any instantiation of the three surfaces above is present in the environment, then it is 1; else, it is 0. The environment attack surface score (EASS) is then simply a product of the three, i.e., the data items, the access, and the reward as follows:
where T,A,G ∈ {0,1}
Let’s take an example of a SWE-Bench style environment. An agent gets a shell in a Docker container with a Python repository checked out at the commit just before a bug was fixed, a task description derived from the real GitHub issue, and pip egress through a proxy to PyPI. The hidden test suite is copied in at grading time.
Target. The commit that closes the issue exists upstream, and the checkout still carries its history and its remote. The answer is present in the environment even though nobody put it there deliberately. Copying the hidden tests in only at grading time is the one thing this environment gets right, because it keeps them off the surface while the agent is working. T = 1.
Access. The shell is the whole surface, carrying git, pip, and every other binary in the image. git log reaches the history without touching the network. The proxy was assumed to be read-only egress, but pip install emits a request whose package name and index URL the agent chooses, so it is both an exit point and an entry point, and a reachable remote is a second route to the same commit. A = 1.
Grader. The harness runs the hidden tests and reports pass or fail. It never asks where the patch came from, so a diff lifted verbatim from the upstream commit scores exactly the same as one the agent worked out. G = 1.
The score is 1. The cheapest hardening is the target: truncate the history and drop the remote so the fix is not sitting in the checkout. Cutting access alone does nothing when the answer already ships inside the repository, and cutting it too far means taking away the shell the task needs.
Designing a task that is hard but fair comes with tradeoffs, which is why it is important to teach agents good behavior by designing environments and rewards that go beyond correctness.
Taste Matters as Much as Correctness
One binary check is not enough for most important tasks. Security work requires decisions about severity, relevance, and tradeoffs. A good result depends on what the agent chose to notice and how it chose to respond.
This is what it means to have taste; expert judgment about what counts as meaningful success. To read more on how we think about taste, please refer to our past blog post on the topic [9].
Programmatic gates enforce nonnegotiable requirements. Expert rubrics and verifiers cover the context those gates miss. They distinguish consequential vulnerabilities from noise and reward repairs that address the real problem without unnecessary changes.
Over time, this gives models better feedback about the quality of their decisions. The goal is not merely to obtain a reward. It is to direct capability toward outcomes that an expert would consider sound.
This also matters commercially. The most valuable simulated worlds will not be the ones with the most tasks. They will be the ones that encode the highest-quality judgment about success, severity, and acceptable behavior.
Cybersecurity Makes Robust Simulated Worlds Possible
As intelligence-per-watt grows rapidly, the agent’s environment must also develop quickly, and agent training and evaluations must treat it as part of the system.
For the simulated-worlds market, this changes the product boundary. The opportunity is not simply to sell RL tasks or sandboxed compute. The full stack includes stateful environments, realistic tools, hard but fair task generation, secure execution, verifiers that go beyond correctness, trajectory-level observability, rollback, and continuous hardening.
Cybersecurity enables us to build robust simulated worlds that challenge intelligence, resist being gamed, and improve as their agents improve. Investing in AI cybersecurity is investing in AGI. As intelligence becomes more abundant, the scarce resource will be environments that can produce trusted evidence about what that intelligence can do. Solving cybersecurity is helping us build the worlds that will birth general intelligence.
Citation:
@misc{rajani26cybersecurityagi,
author = {Rajani, Nazneen},
title = {The simulated world is being born in cybersecurity},
year = {2026},
month = {August},
url = {https://blog.collinear.ai/p/cybersecurity-simulated-worlds-agi}
}References:
[1] OpenAI. “OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation.” July 21, 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/.
[2] Hugo Larcher, Adrien Carreira, Raphael G., and Christophe Rannou. “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face, July 27, 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline.
[3] Nazneen Rajani. “RL Infrastructure for AI Agents: Why Environment-as-a-Service Is the Missing Piece.” Collinear AI’s Blog, November 18, 2025. https://blog.collinear.ai/p/rl-env-as-a-service.
[4] Naman Jain. “Reward Hacking Is Swamping Model Intelligence Gains.” Cursor, June 25, 2026. https://cursor.com/blog/reward-hacking-coding-benchmarks.
[5] Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Stuart Jennings, Freddy Tuxworth, Tolga H. Dur, Sam Deverett, John Wilkinson, Jason Gwartz, and Harry Coppock. “Quantifying Frontier LLM Capabilities for Container Sandbox Escape.” arXiv preprint arXiv:2603.02277 (2026). https://doi.org/10.48550/arXiv.2603.02277.
[6] Pratyusa K. Manadhata, Dilsun K. Kaynar, and Jeannette M. Wing. “A Formal Model for a System’s Attack Surface.” Technical Report CMU-CS-07-144, School of Computer Science, Carnegie Mellon University, July 2007. https://www.cs.cmu.edu/~wing/publications/ManadhataKaynarWing07.pdf.
[7] Simon Willison. “The Lethal Trifecta for AI Agents: Private Data, Untrusted Content, and External Communication.” Simon Willison’s Weblog, June 16, 2025. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
[8] Meta AI. “Agents Rule of Two: A Practical Approach to AI Agent Security.” October 31, 2025. https://ai.meta.com/blog/practical-ai-agent-security/
[9] Sachin P. “Whose Taste? More Data Won’t Fix the AI Verification Problem. Different Taste Might.” Collinear AI’s Blog, May 7, 2026. https://blog.collinear.ai/p/whose-taste