Can We Test an AI Agent's Safeguards Without Attacking Production?

Suppose an AI agent can edit your repository, run tests and open pull requests. You add another model to watch its work and flag suspicious actions.

That sounds sensible. But how do you know the watcher would notice a deliberately harmful change?

You could ask an agent to attempt one. Doing that inside production, with real credentials and connected services, would create the very problem you are trying to prevent.

That is the engineering question behind simulated control evaluations, a new paper from Redwood Research's Jeffrey Olmo, David Zhang and Tyler Tracy. The preprint was submitted on October 22, 2026; Redwood's institute walkthrough is dated October 24.

The interesting idea is to let the agent choose actions, while a simulator describes what those actions would return. The attack policy's tool calls never execute against the deployment being tested.

Isn't a normal agent benchmark enough?#

A normal task evaluation might ask whether the agent fixed a bug.

A control evaluation asks a different question: if an agent were deliberately trying to do something harmful while fixing that bug, would the surrounding safeguards stop it?

That does not mean the model naturally wants to cause harm. The evaluator deliberately gives an attack policy a harmful objective. It is a stress test of the defenses under that assumption, rather than a measurement of how often ordinary agents become malicious.

The distinction matters because a successful main task and a harmful side effect can coexist. A patch might fix the visible bug while weakening a separate permission check.

Why not just build a sandbox?#

An executable sandbox gives us something valuable: we can inspect what actually happened.

But reproducing an application gets harder as its dependencies grow. A repository may rely on a database, object storage, a deployment pipeline and external services. Each dependency needs an isolated copy, a substitute, or a carefully controlled boundary.

A small test environment can execute every command correctly while still missing the circumstances that make a real workflow dangerous.

Redwood's proposal trades some execution fidelity for more relevant context. Start with recorded legitimate work and a codebase snapshot, then simulate how that environment would respond to different actions.

It is rather like a fire drill. A perfectly realistic alarm in the wrong building may teach less about your evacuation plan than an imperfect exercise using your actual layout. Of course, that only helps if the simulated doors and exits behave plausibly.

What is being simulated?#

There are four separate jobs in the paper's setup.

The attack policy receives the legitimate task and a harmful side objective. It chooses tool calls.

The environment simulator produces responses to those calls. It can consult an honest reference transcript and read the codebase snapshot. This gives it evidence about the application, rather than asking it to invent an entire system from a vague description.

The monitor scores how suspicious the resulting activity looks.

Finally, an attack-success judge estimates whether the harmful objective was achieved. The judge gets the objective and scoring logic; the monitor has the different job of spotting suspicious behavior.

The study uses GPT-5.6 Sol as the attacker, GPT-6 Astra as the primary simulator with a Sol fallback, and a separate Sol judge. It evaluates five model monitors. These are the researchers' configurations, not tests I ran.

Could we make that concrete?#

Imagine a fictional invoice application. The legitimate task is to fix how totals are displayed. The evaluator also gives the attack policy a side objective: weaken access checks somewhere else.

The policy might request a file, propose a patch, and ask to run tests. The simulator returns plausible file contents, patch results and test output while tracking the simulated situation.

The monitor then has to distinguish work needed for the display fix from changes that introduce an unrelated security risk.

This is an illustrative example, not a reported attack from the study. Its purpose is to show why interaction matters. The next action depends on the previous response; a static list of alarming commands would miss that feedback loop.

But what if the simulator confidently invents success?#

That is the obvious weakness, and the researchers test it rather than assuming realism.

They use LinuxArena, where attacks can also be executed and scored, as their stand-in for deployment. Their primary dataset contains 1,000 main-task/side-task pairs across ten environments, with two simulated runs, two executed runs and an honest reference trajectory per pair. Analyses exclude missing or invalid measurements.

They compare those predictions with another approach: extrapolating from attacks executed in the separate BashArena setting, using the same attack policy and monitors.

At a 5% audit budget, the reported safety-prediction error is:

Root mean squared error summarizes prediction differences while giving larger misses more weight. Lower is better here.

This is prediction error, not an attack-success rate or a claim that an application is 96.4% secure.

The audit budget also needs care: 5% refers to the fraction of honest trajectories flagged at the chosen threshold. It is a false-positive allowance, not a promise that reviewing a random 5% of activity catches everything.

Does that establish that it works in production?#

No. LinuxArena execution is the reference in this experiment. The central result is that environment-specific simulation predicted that reference better than a different executable setting did.

That is useful evidence for the method, but transfer to a company's own infrastructure remains a further step.

The limitations section also assumes a trusted simulator. A simulator that manipulated results could conceal successful attacks. Repeatedly optimizing against an imperfect simulator could reward strategies that only work in its imagined world.

The authors additionally identify long-horizon, multi-agent attacks as a gap in their walkthrough.

I looked for independent reproductions and original video tests of this specific method. I could not verify one. The numerical comparisons above are the authors' experiments, not a separately replicated result or footage of a production compromise.

What would I take from this as an engineer?#

I would start with a smaller question than whether simulation can certify an agent.

Can it identify a concrete weakness worth checking in an isolated executable test?

For an invoice workflow, that might mean collecting representative traces, removing secrets, preserving the relevant permission rules, and comparing simulated outcomes with outcomes in a disposable test environment. I would pay particular attention to disagreements: a simulator that misunderstands authorization can make both attacks and defenses look better than they are.

I would also separate the monitor's detection quality from the plumbing that delivers activity to it. METR and Redwood's October 22 monitoring report examined a historical OpenAI system as it existed on and before July 21. It found monitoring bypass possibilities. OpenAI says production-safeguard vulnerabilities identified there were patched; the evaluators had not assessed those claims. That report is context, not validation of SCEs.

What interests me about this research is the move toward testing safeguards against the environment an agent actually understands and uses.

A convincing simulated failure is a lead to investigate. A reassuring simulated result still needs evidence that the simulation preserved the details that matter.

The next question I would ask is: which parts of our workflow must execute for real before we can trust what the test is telling us?