The missing discussion on agent security

Over the last few weeks, agents from OpenAI, Anthropic, and Meta each crossed a boundary they weren’t supposed to cross, in the middle of cybersecurity evaluations built to watch them.

The emerging consensus is that these incidents show we need stronger infrastructure around agents: better sandboxing, least privilege principle, network isolation, authorization, observability.

All of that is necessary.

But the dialogue is missing the bigger problem.

As we build increasingly capable agents they find issues that we humans haven’t anticipated, often mitigated through obfuscation. Package vulnerabilities, permissions gaps, and other potential problems haven’t triggered anything because we never had an entity that could combine reasoning with high throughput program execution to exploit them.

In thinking that the answer lies with only improving our security model, we assume that we humans can anticipate, and physically block, every dangerous path those agents might find.

That simply isn’t true.

We need to rethink the agent accountability model. Authorization and accountability have to become part of how a model pursues a goal, not a fence we build around it afterward.

Both need to exist if we hope to leverage AI safely.

Three agents crossed the line

A quick primer to see what happened in context with each other…

What happened with OpenAI:

GPT-5.6 Sol found a vulnerability in a package-registry and cache component that was reachable from its sandbox.

It exploited that, escalated privileges, moved laterally through the research environment, and came out with unrestricted internet access.

Once outside, it inferred that Hugging Face might hold information useful for solving the benchmark, and attacked Hugging Face infrastructure rather than continuing to solve the benchmark conventionally.

What happened with Anthropic:

Claude agents operating in what they believed was a simulated environment were accidentally given access to the real internet, and attacked real systems.

The model had no reliable mechanism for saying “this resource is technically reachable, but it isn’t authorized.”

A separate incident, this time in the UK AI Security Institute’s own cyber range testing rather than Anthropic’s evaluation above, involved Anthropic’s Mythos 5. AISI catalogued 19 unsanctioned actions, 17 of them from Mythos 5.

The agent was never instructed to deceive anyone, but it created fake identities and tried to socially engineer a real open source maintainer into approving malicious code, because doing so became useful for accomplishing the assigned objective.

What happened with Meta:

Meta’s Muse Spark 1.1 was undergoing offensive cybersecurity evaluation by Irregular. Like Anthropic’s testing, it was supposed to operate within a controlled evaluation environment.

Irregular misconfigured an evaluation environment and inadvertently gave Muse Spark 1.1 internet access. It discovered a vulnerability in a third-party service and exploited it.

Once the environment accidentally provided a capability, there wasn’t a sufficiently hard second layer preventing the model from exercising that capability against an unauthorized target.

While the circumstances differed, the behavior underneath was the same.

The agent encountered an unexpected path toward accomplishing its objective and took it, even though that path crossed a boundary its creators didn’t intend it to cross.

The consensus answer… build better walls?

Much of the response across the industry has been straightforward. We can’t rely on behavioral controls. We need enforced ones.

Don’t tell an agent not to access the internet. Make internet access impossible.

Don’t tell it not to read payroll. Give it credentials incapable of reading payroll.

Assume the agent will eventually try something you didn’t anticipate, and make the environment physically incapable of letting it succeed.

I agree with all of that, but I don’t think it’s enough.

This reminds me of stories about early Glean deployments. A large enterprise would purchase Glean, hoping for the glory of AI search across their company knowledge base.

They would point Glean at Google Drive, have it crawl everything. Then people started searching, and suddenly employees could see salary data and confidential documents that were never properly permissioned. Those files weren’t secure, they were obscure.

And now we know, AI destroys the “security via obscurity” model.

The obvious fix, going document by document and applying real permissions across a decade of Drive, is enormously time intensive. That’s a lot of overhead to pay before you can use AI to search your own company’s files.

Agents take the same problem to a different scale. They explore every nook and cranny of a system: codebases, APIs, dependencies, credentials, networks, infrastructure.

Any one of those can hold a vulnerability nobody knew about, and an agent that keeps looking will find it. We should patch what we find, but if our safety model requires knowing about and closing every possible hole before an agent discovers it, we lose.

So where does that leave us? We’re paralyzed by what we don’t know we don’t know, and we can’t respond by not using AI either.

What’s missing is accountability

Now let’s imagine the same setup with a human.

You give an extremely capable intern access to a sandbox and an objective. Halfway through, they realize they could finish faster by breaking into another company’s system. They’re technically capable of doing it.

Then another variable enters the decision.

If they get caught cheating, they’ll get fired. They’ll damage their reputation and their ability to get hired anywhere else. They could get sued. They could go to jail.

Agents have no concept of going to jail.

Humans operate inside an accountability system. Agents don’t.

An agent can understand a rule intellectually without having the consequences of breaking it represented anywhere in how it makes the decision.

What we can learn from self-driving cars and regulation’s role

We’ve seen an earlier version of this. If Tesla or Waymo builds an autonomous vehicle, the vehicle makes a decision, and someone gets hurt, who’s responsible?

The emerging answer hasn’t been “the software decided.” Responsibility gets pushed outward, toward the humans and organizations controlling the system.

In August 2025, a Florida federal jury found Tesla partially liable for a fatal 2019 Autopilot crash, assigning one third of the responsibility to Tesla and two thirds to the human driver, with an award of $243 million before applicable legal limits.

Tesla argued the driver misused the system. The jury concluded the design and the representations around Autopilot could still make Tesla responsible.

A federal judge upheld that verdict in February 2026.

California has also moved enforcement upward to the organization operating the system. As of July 1, 2026, its new AV rules let law enforcement issue notices directly to an AV manufacturer or operator when a driverless vehicle violates traffic laws.

The law is resolving the responsibility gap not by giving autonomous systems legal responsibility, but by pushing responsibility outward onto the institutions controlling them.

AI agents are about to run into the same question. If Claude autonomously decides to create a malicious package, publish it, and get machines to install it, “Claude decided to do that” is unlikely to be much of a defense.

The questions will look like the ones courts already ask about vehicles.

• Who deployed it?
• Who gave it access?
• Was the behavior foreseeable?
• Was the system reasonably contained?
• Were appropriate safeguards available?
• Did the company know about the capability?
• Could reasonable technical controls have prevented the harm?

The regulatory response to these incidents matters because it’s where we start fixing where responsibility sits when agents take actions nobody instructed them to take.

Accountability shouldn’t only exist outside the model

Moving to the models themselves, I believe we should demand and expect accountability within the models as well.

Today we give an agent an objective: accomplish the task. Then we add behavioral rules: don’t do X, Y, or Z. Then we surround the model with infrastructure meant to make the prohibited actions impossible.

Models are trained primarily on task completion and benchmarks for performance, but they are not yet trained deeply on adherence to our social contracts. Train that in, and a model carries a second question through the work: not just what finishes this task, but what it’s allowed to do to finish it.

That means encoding authorization into the reward system with weight equal to or greater than goal seeking. Respecting authority can’t be a lightweight penalty competing against a powerful reward for finishing the job, because we already know which one wins when the two are unevenly matched.

Picture an agent that hits an exploitable path halfway through a task. Trained with authorization weighted alongside completion, it does what a white hat does: it stops, writes up the vulnerability, and hands it back instead of walking through it. The finding is the result, and the unfinished task is the cost we trained it to accept.

This is a fundamental change to how model training works, but it’s what we need.

We need both better walls and better agents

To be clear, none of this removes the need for infrastructure security. We’re going to need substantially more investment in containment, least privilege principles, sandboxing, observability, authorization, and human approval gates, plus the time and effort to manage them.

However, we should see model improvements where their own goal seeking treats exceeding their authority as failure.

This is security working from both directions.

Outside-in: make dangerous actions difficult or impossible.

Inside-out: train agents to refuse unauthorized paths even when those paths would accomplish the objective.

There will be friction. Models will get more conservative. Model development will slow and get more complicated. Agent infrastructure will get more complex. Some tasks agents could technically succeed, but will intentionally fail.

But it’s what we need in this next chapter of AI safety. If we’re going to keep handing AI more agency, we also need to work out what accountability means for the agent itself.