Trending Topics
Trending
AI agents can make changes. They cannot own the consequences.
This article is part of our Opinions section, where we invite industry professionals to share their views on the most pressing technology questions of our time.
Imagine this scenario. Your transport management system detects a performance bottleneck and adjusts its own scheduling configuration. It does not wait for a human to approve the change; the organisation has already authorised this category of decision, and the adjustment takes only a few minutes. Everything initially looks fine: shipments keep moving, orders continue to be processed, and customers notice nothing.
Eight hours later, it becomes clear that the change had a side effect. A specific import corridor is now calculating customs tariffs incorrectly. Shipments start to back up, SLAs are being missed, and the client is calling. Your operations team is working backwards through system logs, trying to reconstruct what changed, why it changed, and under whose authority it happened.
This may not describe how every logistics company operates today, but it illustrates a new class of operational risk that emerges when automation and AI stop recommending decisions and start executing them. The AI agent may have made the change. It cannot own the consequences.
The system can be running and still be operationally down
The value AI brings to complex operations is not in question. In well-defined tasks, it can improve demand prediction, routing, capacity planning, exception handling, and the speed at which organisations respond to disruption.
The economic value of avoiding operational failure is already well understood: a 2024 Siemens study of large industrial organisations estimated that unplanned downtime costs the world’s 500 largest companies almost 1.4 trillion dollars annually, equivalent to around 11% of their revenues. But autonomous systems introduce something that looks increasingly similar to downtime without necessarily involving an outage.
The system may remain available, every service may still respond, and infrastructure dashboards may remain green. It can simply start behaving incorrectly: a scheduling rule changes, a pricing threshold moves, capacity is allocated to the wrong corridor, a warehouse process begins prioritising the wrong class of shipment. The platform continues operating, but its behaviour no longer matches the organisation’s intent.
The result can be economically equivalent to downtime: delayed shipments, manual replanning, broken SLA commitments, regulatory exposure, and eroded client trust. The system is technically up. Operationally, it may already be down.
Execution speed has outrun governance speed
Traditional change controls were designed in an era when the slowest actor in the system was a person. An engineer proposed a change, someone reviewed it, the team lead or system owner approved it, and only then was the change implemented. Those steps created time for judgement, and where the workflow was properly designed, they also created a record of who proposed, approved, and executed the change.
AI agents compress that timeline. An agent can detect a problem, select a response, and execute the change before a human would have finished reading the incident notification, and that speed is precisely why organisations want to use it. But when execution becomes faster than the governance model around it, a significant part of the organisation’s control can disappear. The answer is not to make the AI wait for a person every time. It is to redesign how authority and accountability are represented in the system.
The actor is not the accountable owner
When an autonomous change causes an incident, several questions are often collapsed into one: who made the change? In an AI-enabled operating model, that question is no longer precise enough. The organisation needs to distinguish at least four roles.
The actor is the entity that technically executed the change: an AI agent, an automation pipeline, or a person. The delegator is the person who granted the agent authority to perform a defined class of actions. The authorisation mechanism is the policy and control layer that decided whether this specific action was allowed under the current conditions. And the accountable owner is the human role or organisational function that accepted the risk of allowing that category of autonomous action. These roles may be connected, but they are not interchangeable.
An audit record saying “scheduling-agent-17 changed the routing configuration” provides attribution. A record saying “scheduling-agent-17 changed the routing configuration using authority delegated by an operations engineer, under production policy version 12, for a process owned by Transport Operations” provides the beginning of accountability. The agent is the actor. It is not the accountable owner.
Do not let the agent impersonate a person
One of the simplest ways to integrate an AI agent into an existing enterprise system is to give it a user’s API key, and that approach is also one of the most damaging from an accountability perspective. If the agent operates under a human identity, every audit record appears to show that the person performed the action. The organisation loses the distinction between the person who authorised autonomy and the non-human actor that exercised it.
During an incident, the logs might say: “Petr changed the production configuration.” What actually happened may be: “An AI agent changed the production configuration using a subset of permissions delegated by Petr.” Those are not the same event.
A better architecture gives the AI agent its own identity as a non-human principal. That identity is linked to a named human sponsor, but it does not impersonate the person. The agent receives only an explicitly delegated subset of the user’s current permissions, and its effective authority should be the intersection of three things: what the user deliberately delegated, what the user is currently authorised to do, and what the platform and operational policy permit in that context.
If the user loses a permission, the agent should lose it too. If the agent was authorised only for a test environment, it should not be able to act in production. If platform policy prohibits a change during a critical operational window, the user’s delegation should not override that restriction.
This can be thought of as a constrained digital twin of the user: linked to a human source of authority, but operating under its own identity, its own scope, and its own revocable credentials.
Delegation does not transfer accountability to the agent
Pre-authorising an AI agent does not remove the human decision from the system. It moves that decision earlier. Instead of approving every individual action, the organisation decides in advance which categories of decisions the agent may make, which resources and environments it may affect, what operational conditions must be true, which actions require notification, which actions still require human approval, and which business or system owner is accountable for that autonomy.
This is a more scalable form of human judgement, but it is still human judgement. The critical mistake is to treat the AI agent as though it has somehow absorbed responsibility for the decision. It has not. An agent cannot accept business risk, cannot own a contractual SLA, it cannot explain a governance model to a regulator, and it cannot decide the organisation’s risk appetite.
The accountable owner remains the person or role that authorised the operating boundaries within which the agent acts.
Another approval step is not the answer
The instinctive response to autonomous operational risk is to add a human gate: the AI proposes, a person approves. This works only while the volume and speed of automated decisions remain low. Once every AI-generated action requires human approval, the organisation has recreated the same queue that automation was introduced to remove, and the AI becomes fast software waiting on a slow inbox.
The objective should not be to slow autonomous systems down to human speed. It should be to make the boundaries of autonomy structural. A human decides the operating laws, and the platform enforces them automatically.
Four structural properties
A governable autonomous system needs four properties. First, a distinct identity. Every AI agent must act under its own technical identity, and the organisation should be able to distinguish between a human user, an automated pipeline, a specific AI agent, and one agent instance or execution run from another. Auditing only human accounts is no longer sufficient once non-human actors can make operational changes.
Second, bounded delegated authority. The agent must not inherit a person’s full access simply because that person created it; its permissions should be explicit, narrow, and revocable. The system must record who created or sponsored the agent, what permissions were delegated, which resources and environments are in scope, when the delegation expires, and which policies further restrict its behaviour. Delegation should describe the maximum authority the agent may exercise, not a guarantee that every action within that scope will be permitted.
Third, audit evidence created at execution time. The audit trail must be produced as part of the action itself, recording which agent acted, whose authority it was using, which policy allowed or rejected the action, what resource was changed, the previous and resulting state, why the action was initiated, which operational or business owner is accountable, whether the action succeeded, and how it can be reversed. By the time an incident occurs, reconstructing this information from separate logs, repositories, workflow tools, and monitoring platforms is already too late. The evidence must be a natural output of execution, not an administrative reconstruction afterwards.
Fourth, independent stoppability. The organisation must be able to stop a specific agent immediately, not the whole user account, not every automated process, not the entire operating platform. A single compromised or malfunctioning agent should be suspended without unnecessarily paralysing the surrounding system, which requires the agent to have its own credentials, identity, and revocation path. This is the difference between governed autonomy and automation that has simply been released.
Accountability must follow the chain of authority
A useful audit trail should allow an organisation to reconstruct the complete chain: from the business or system owner, through operational policy and human delegation, to the AI agent identity, the authorisation decision, the executed change, and finally the operational outcome.
When any link in that chain is missing, the organisation hasn’t lost a record. It has lost the ability to answer, after the fact, who was actually in control.
More from our Opinions section
About The Author
