The Attack Surface Is the Workflow
AI risk becomes concrete when a model is connected to credentials, tools, and unattended loops.
The most useful AI safety question is not whether a model has hidden intentions.
It is what the workflow around the model can reach.
That sounds less dramatic than a debate about superintelligence. It is also more useful for anyone building with models today. A capable model inside a narrow sandbox is one system. The same model connected to production credentials, browser sessions, code repositories, databases, queues, and retry loops is another.
The model is one component. The workflow sets the blast radius.
The chat window is not the system
A model can generate text, reason over a problem, write code, or propose a plan. None of those abilities tells you what the surrounding system will do with the output.
The surrounding system may let the model call tools, inspect files, query a database, open a browser session, create tickets, deploy code, or pass a result to another agent. It may also let the model retry after a failed action, change the method after a block, or keep working while a human reviews the results later.
Those details are not implementation trivia. They are part of the security boundary.
A useful review therefore starts with questions that sound almost boring:
What can the agent read?
What can it write?
Which identities and credentials can it use?
Where are those credentials stored?
What is the largest change it can make without approval?
What stops the loop, and can that stop action work independently of the agent?
What evidence does a human inspect before the next consequential step?
If the answers are unclear, the system is not autonomous in some mysterious philosophical sense. It is simply underspecified.
AI changes the cost of persistence
Recent threat reporting has made this distinction concrete. The important change is not that a model suddenly became an independent attacker with its own target list. In the cases described, humans still set objectives and reviewed important results.
The change is that a model can now handle more of the intermediate work.
It can search, classify, rewrite, test, sort, document, and try again. It can move between tools without requiring a specialist to perform every small step. That gives a human directed operation more speed, breadth, and persistence.
Security teams have traditionally gained something when a defense forced an attacker to stop and rebuild. A blocked technique cost time. A failed attempt required another specialist. A new target required another round of research.
A workflow that can notice a block, change its approach, and retry reduces some of that cost. It does not make the system magical. It changes the economics of the operation.
That is why the word autonomous can be misleading. The relevant question is not whether the model has become a person. The relevant question is how much work the workflow can continue doing after a human starts it.
A stolen AI key is a security event
Many teams still treat an exposed AI key as mainly a billing problem. The immediate symptom may be an unexpected invoice, but the larger risk is the identity and access chain around the key.
An attacker may use the key to obtain compute, hide activity inside a legitimate service, operate a wrapper, reach a reseller, or move through an integration that was never designed as a security boundary. A token can also become valuable because it opens access to another system, not because the model itself is special.
This is especially relevant for applications built quickly. A key can end up in a client bundle, a public repository, a screenshot, a build log, or a copied configuration file. An integration can quietly give an agent more access than its author intended. A sandbox can contain production credentials because the boundary was never tested under pressure.
The basic controls are not glamorous, but they are concrete:
Scope keys and tokens narrowly.
Keep them on the server.
Rotate them when exposure is possible.
Log their use.
Separate test identities from production identities.
Make the system fail closed when a credential is revoked.
A credential review is not a detour from AI safety. It is AI safety at the level where a deployment can actually fail.
Human in the loop is not a control model
"There is a human in the loop" can describe two completely different systems.
In the first, a human approves each consequential action. The agent can prepare a change, but it cannot apply the change until a person reviews the exact target, scope, and evidence.
In the second, an agent performs dozens of actions, follows retries, changes tools, and produces a summary. A human reads the summary at the end.
Both systems contain a human. Their blast radii are not comparable.
The phrase matters only when it is followed by a description of the human's position in the loop. What can happen before approval? What can happen after a failed check? Which action requires a fresh decision? What does the reviewer actually see?
A final answer is not the whole operation. If an agent can make consequential tool calls, then those calls, permissions, retries, and intermediate results are part of the output that needs review.
This is also why audit logs should not be treated as decorative observability. A log that records only the final answer cannot explain how the system reached it. Useful evidence includes the identity used, the tool called, the arguments passed, the returned result, the retry count, and the approval boundary crossed.
Refusal is useful, but it is not an architecture
Model refusal is one layer of defense. It can reduce some harmful requests. It can also fail when the surrounding system changes the prompt, the tool, the account, or the path by which the action is requested.
That does not make refusal pointless. It means refusal cannot carry the whole system.
Access control answers what an identity can reach. Rate limits constrain how quickly it can act. Sandboxing constrains where it can operate. Approval gates slow down consequential changes. Monitoring creates a chance to notice drift. Incident response determines what happens after a control fails.
The strongest design assumes that any one layer can be wrong.
A model can misunderstand the task. A classifier can miss the risk. A credential can be stolen. A reviewer can approve the wrong screen. A queue can retry an action that a person thought had stopped.
Good security design does not require every layer to be perfect. It makes failure visible, bounded, and recoverable.
Dual use makes simple filters insufficient
The hardest cases are not always obviously malicious prompts. The same technical knowledge can support a legitimate experiment, defensive research, or harmful work. A classifier sees language. The surrounding system determines identity, permission, context, and consequence.
That is why sensitive domains need more than a content filter. The design has to account for who is asking, what the system can access, which actions are possible, and whether a person can intervene before the point of no return.
This is also where broad claims about AI risk can become unhelpful. Catastrophic scenarios deserve serious attention. They do not remove the need to secure ordinary deployments. The forgotten wrapper, the broad sandbox, the token in a client, and the unreviewed retry loop are already concrete engineering problems.
The near term and the long term are not competing conversations. The same infrastructure choices affect both.
Distillation turns access into a supply chain problem
Another issue is model distillation. A model provider can lose more than a single answer when another system repeatedly harvests its outputs to improve a competing model.
That creates a different kind of security boundary. The question is no longer only whether a user can make one harmful request. It is whether access can be automated at scale, whether identities are real, whether requests are being proxied, and whether the output is being collected as training data.
The details vary by case, and allegations need to remain allegations. The general lesson is still clear: model access is part of a supply chain. Rate limits, account controls, monitoring, output policy, and abuse detection protect not only the service itself but also the capability encoded in its responses.
A useful system has to know the difference between a person using a model and a pipeline quietly turning that model into a data source.
What I would inspect first
I would not start an AI safety review by asking whether a model secretly wants anything. I would start by drawing the workflow.
Put the model in the middle of a page. Around it, write every tool, identity, credential, data store, queue, browser session, and external service it can reach. Mark which actions are read only and which actions change state. Mark where retries happen. Mark where a person must approve. Mark how the system can be stopped.
Then make the first experiment smaller than the proposed deployment:
Use a test account.
Start with read only access.
Log every tool call.
Set a time limit.
Require approval for state changes.
Test the external stop action before adding another tool.
Review the log, not only the final answer.
This does not eliminate risk. It makes the risk legible. It gives the team evidence about what the workflow actually does rather than what the prompt suggests it should do.
Autonomy should be earned through that evidence. It should not be granted because a model sounds confident, because a demo worked once, or because a human remains somewhere in the process.
The next serious AI security boundary may not look like an advanced model. It may look like an ordinary integration with a broad token, a queue that never gives up, and nobody who can stop it from outside.
That is why the workflow deserves as much attention as the model.