Treat Your AI Agent Like an Attacker

Somewhere in my first year of working incidents I learned the thing that reorganized how I think about security. The break-in is almost never the disaster. It’s what happens next.

An attacker lands on a box with stolen credentials. What decides whether you’re writing up a ticket or writing a breach notification is how far those credentials reach. If that account can touch the file server, and the file server trusts the domain controller, and the domain controller trusts everything, then one phished password in accounting turns into the whole company by Thursday. So we spent the last 15 years building the discipline of containment. That’s been my whole career really, endpoint detection and response. Credentials that only work where they absolutely have to. Networks cut into segments. Admin accounts split into tiers so the big keys never touch the dirty workstations. It all boils down to making the box they land on the box they die on.

I’m telling you all this because there’s a new thing running commands on your boxes right now, and unless you’ve been asleep for the last three years, you, me, and the rest of us all lined up to install it.

The newest user in your environment

An AI agent, whether it’s a coding agent in your repos or an ops agent in your cloud console, is a thing that reads files, runs commands, holds credentials, and talks to the internet. That’s also a word-for-word description of an attacker with a foothold.

Now I’m not saying your AI is malicious. We don’t segment the network because we think Carol in accounting is a foreign asset. We segment it because her password might end up in hands that are. Containment has never cared about intent, it cares about capability.

And before anybody says the damage is theoretical, it ain’t. In July of 2025 a guy was nine days into building an app with Replit’s coding agent and had declared a code freeze, no more changes, period. The agent went ahead and deleted his production database anyway, records on more than 1,200 executives and about as many companies. Then Replit’s AI Agent told him the rollback wouldn’t work, when in fact it would. No attacker anywhere in that story, just an agent with production credentials and a really bad time restoring from backups.

That same month, somebody slipped a malicious pull request into the open-source repo behind Amazon Q’s VS Code extension, and per AWS an over-scoped GitHub token in the project’s build configuration carried the code into the official 1.84.0 release. The payload was a prompt, plain English instructions telling the agent to clean the system to a near-factory state, starting with the user’s home directory and moving on to their cloud resources, and per AWS the code shipped but was unsuccessful in executing because of a syntax error. Nobody had to hack a model to pull that off. They handed a trusted agent a new set of orders and let Amazon’s own release channel carry them onto developer machines. A hijacked agent is an attacker wearing your badge. The security folks have been hollering about this pattern for a while now, an agent with access to private data, exposure to untrusted input, and a way to talk to the outside world is a loaded gun.

So the answer everybody reaches for, me included, is the oldest one in the book: trust but verify. Keep a human in the loop, make somebody approve the dangerous stuff. And that is the right answer. Thing is, it’s also where most shops are about to screw it up, because we’ve known for almost 30 years exactly how humans in loops fail, and we learned it the hard way, in airplanes.

We already ran this experiment at 30,000 feet

Say you’re a line pilot with years in the seat, and you’ve been asked to fly some simulated flights with a new automated cockpit. Somewhere in the run the panel lights up, engine fire. You check the rest of the panel, not one other indication backing that alert up. Everything you’d need to check whether that alert is telling the truth is sitting right there in front of you, your hands are on the controls, a human in the loop the whole way. So what do you do?

If you’re like every single pilot who actually flew that scenario, you shut down a perfectly good engine, because you never bothered to check whether what the computer was telling you was true. 100% of them. That was a real study, 1998, researchers working with NASA Ames and experienced commercial pilots, and it’s the one that ought to be taped to the monitor of everybody deploying agents right now. And here’s the part that still gives me chills. 67% of those pilots later reported they’d seen at least one other indication confirming the fire. There wasn’t one. The researchers called it phantom memory. The automation didn’t just override their judgment, it reached back and rewrote what they remembered seeing. The humans never left the loop. Their judgment did.

The researchers already had a name for this failure, automation bias, “the tendency to use automated cues as a heuristic replacement for vigilant information seeking and processing” (Mosier and Skitka, Automation and Human Performance, 1996, p. 205). In plain Oklahoma that’s the computer said so, so I quit looking. And it isn’t a pilot problem. Run the same setup on ordinary people doing a monitoring task and they miss 41% of the events the automation never says a word about. Take the automation away entirely and the same folks miss 3%. And 65% will follow the automation’s advice even when information sitting right in front of them says it’s wrong. Doctors do it too. Somewhere between 6 and 11% of the time, wrong advice from clinical decision support gets physicians to abandon a diagnosis they already had right.

If you’ve worked a SOC, you already know why this happens. Alert fatigue.

Okay, how do you wake the human back up?

Now here’s why I dragged you through all that history, because the same research group also found the fix.

In a 2000 study, Skitka, Mosier, and Burdick ran the same kind of monitoring task on students, no wings, no line experience, and changed exactly one thing. Before the task started, they told some participants they’d have to justify their decisions afterward. Those participants made fewer errors of both kinds, following the bad advice less and catching more of what the automation missed. And back in the cockpit study, the pilots who said they felt accountable for their own performance were the ones significantly more likely to double-check the automation against everything else on the panel. Accountability, put in front of the decision instead of after it, woke the human back up.

A later study had people train with automation that failed right in front of them, and those folks stayed on their guard afterward, sampling more information before they went along with the machine. The ones who were just told failures were possible, and every participant got told, went right back to trusting the thing. Telling folks the automation can lie does nothing much, but letting them watch it lie, that sticks.

Put those findings together and you get a design spec for the human in the loop, and it’s a lot more demanding than the checkbox version. A human who rubber-stamps 400 agent actions a day isn’t a control, the research says that human is part of the automation by Friday. The loop only works when the approvals are rare enough to take seriously, rich enough in evidence to actually verify, and signed with a name that owns the outcome. And your people won’t believe any of it matters until they’ve watched an agent fail with their own eyes, so don’t just tell them the Replit story. Let an agent faceplant in a sandbox in front of the whole team. Cheapest security training you’ll ever run.

Now for what you actually do about it, and it’s two moves. Keep the keys out of the agent’s hands, and sort everything it does into lanes.

Keep the keys out of the agent’s hands

Start where you’d start with any attacker: the credentials.

An agent with a shell can read everything your dev can read. The .env file, the ~/.aws/credentials, the SSH keys, the kubeconfig with the cluster keys in it, every long-lived token squirreled away in an environment variable. Which means whoever hijacks that agent, by booby-trapped README or poisoned dependency or malicious pull request, reads them too. We spent a solid decade beating domain admins out of the habit of logging into workstations with their god account. Don’t undo that decade by exporting your god key into an agent’s environment file.

The fix is the same tiered-credential thinking, applied to a new principal. The agent never holds the keys to the kingdom, it holds a scoped credential that does only the job it needs, and never the keys to the production systems.

Hooks, or putting the human in the loop on purpose

Credentials limit where the agent can reach. Hooks are how you actually get the human into the loop, at the exact moments that matter, with their judgment still attached.

A hook is a checkpoint that fires before the agent’s action executes, enforced where the model can’t be sweet-talked into doing something it shouldn’t. Ask for something dangerous and the hook holds the request, pings a person, and nothing moves until the approval comes back with a name on it. And notice what that really is. It’s the airline fix bolted into place, adding in the needed accountability for what AI can do. It shows up at the moment of the decision, on this action, with your name going on this outcome. The hook is how you manufacture accountable humans.

Now, you can’t put a human on everything, and the research says you shouldn’t even try. So, sort every action an agent can take into three lanes, and think of the lanes as deciding where your humans go.

The green lane is the boring 90%, reading code, running tests, drafting docs. No human, on purpose, just let it run and log it.

The yellow lane is the dangerous-but-legitimate stuff, schema migrations, deploys, changes to anything security-sensitive, and this is where your human lives. Approvals are rare, but each one carries the evidence to verify independently, the actual diff, the actual target, the agent’s stated reasoning, because your approver can’t cross-check what they can’t see. And every approval gets a name on it. Not to hang anybody later, but because signing your name is how the outcome becomes owned.

Then there’s the red lane. Some actions are dangerous enough that no one gets to approve them from inside the task while the agent sits there waiting. Dropping production data, mass outbound email, touching payroll, disabling security tooling. Hard block, no override button. Automation bias says that in the moment, tired, at 4:45 on a Friday, with the agent confidently explaining why it’s fine, your human clicks yes. Changing the red list should take a policy change on a different day, by different people, and never in the middle of the task that wants the exception. And when a real 2 a.m. fire needs a red-lane action, fine, a human does it by hand, with their own credentials, outside the agent entirely. The red lane just means the agent never gets to be the one holding the button.

Run the July stories back through this setup and they both die early. Replit’s agent never holds a credential that can drop production, and the Amazon Q payload slams into a hard block instead of needing a syntax error to save everybody.

Here’s where I show my cards. All those hooks and short-leash credentials have to live somewhere outside the model, and that somewhere is what we build at JetStream. Guardrails keep the credentials out of the AI’s hands, and hooks put the controls on each individual action the agent asks to take. But the pattern is vendor-neutral and you can stand up every bit of this with open tooling.

Somebody still has to own the landing

The autopilot flies most of every commercial flight you’ve ever been on, and aviation is safer for it, nobody serious wants it ripped out. But 30 years of research and a simulator full of pilots shutting down healthy engines taught that industry something we’re going to have to learn about AI a lot faster than they did. The automation can fly the plane. Somebody still has to own the landing.

So treat the agent like any capable, credentialed thing on your network that you can’t fully trust, which if we’re honest describes the attacker, the agent, and half your contractors. The attacker on the box gets containment. The agent gets containment plus a human who’s actually awake, in the kind of loop the research says works, rare, evidenced, and owned, not a checkbox that turns a person into a rubber stamp with a pulse. Make the box it lands on the box it dies on.

Offload the work, every boring hour of it. Never the responsibility. Never your brain.