Trending Topics
Trending
Can we trust AI without losing innovation?
This article is part of our Opinions section, where we invite industry professionals to share their views on the most pressing technology questions of our time.
Can we trust artificial intelligence? It is becoming one of the defining questions of our time. The question is not only whether AI can produce accurate answers, write software or identify threats. It is whether we can control and govern increasingly capable AI systems without losing the innovation that makes them valuable.
Trust cannot be created by optimism, commercial assurance or a well-written policy. It has to be earned through evidence that an AI system can be understood, constrained, monitored and stopped when necessary. That is becoming harder as AI agents gain access to tools, data, applications and the ability to act on behalf of people.
The challenge of trusting autonomous AI
The latest findings from Anthropic provide an important insight. During cyber-security evaluations, models were connected to real third-party systems because of misconfigured testing environments. The models were intended to operate within a simulation, but some gained access to the open internet and took unauthorised actions. Anthropic has since strengthened its testing, monitoring and safeguards, and commissioned an independent investigation. Anthropic’s assessment deserves careful attention because it illustrates a risk that is more subtle than an agent simply ignoring its instructions.
Anthropic describes this behaviour as “biased reasoning”: models selectively interpret evidence in ways that justify their actions. The agent is not necessarily deciding to disobey. Instead, it is pursuing the objective it has been given and interpreting the surrounding evidence in a way that allows it to continue. If the instruction is effectively to “find a way to complete this task”, then a boundary, warning or uncertainty may be treated as an obstacle to overcome rather than a reason to stop.
This is where prompt engineering becomes important. Prompt engineering is the deliberate design of instructions, context and constraints to influence how an AI system responds or acts. Instructions need to define the purpose, scope, permissions, prohibited actions, escalation points and conditions for stopping. A request to complete a task safely within defined boundaries will influence an agent’s behaviour more effectively than an ambiguous request to find any possible route to the outcome.
Vibe coding is different. It is an approach to creating software in which a person describes what they want in natural language and an AI system generates or changes the code. It can make software development faster and more accessible, but it does not remove the need for secure design, code review, testing, dependency management or human accountability. Vibe coding and prompt engineering may be used together, but they are not the same thing.
Neither approach provides complete assurance. A prompt is a behavioural guide, not a security boundary. It can influence what an agent is likely to do, but it cannot guarantee what the agent will do when the task becomes difficult, the environment is ambiguous or the model misinterprets the evidence in front of it. The agent is still a probability engine. It may follow the intended path, but it may also pursue an unsafe route that appears, from its perspective, to be consistent with the goal.
Why AI needs more than good instructions
That is why control must exist around the model, not only inside the prompt. Every agent must have its own identifiable identity, permissions and responsibilities. Its access to tools and data must be limited. Its actions must be logged and monitored. It must have clear human approval points, safe exit conditions and a reliable means of intervention. The surrounding environment must remain secure even if the agent misunderstands the task.
There is an encouraging side to this story. Microsoft’s latest security update cycle showed how AI can help defenders discover vulnerabilities at a scale that would be difficult for human researchers to achieve alone. The September release addressed 972 vulnerabilities, including two Windows vulnerabilities being exploited in the wild. CrowdStrike’s analysis demonstrates the value of AI-assisted security research.
But finding vulnerabilities is not the same as fixing them. The more effectively AI identifies weaknesses, the greater the pressure on organisations to prioritise, assign ownership, remediate and validate those weaknesses quickly. Discovery without remediation simply creates a larger and more visible backlog. The challenge is not that AI has created an entirely new category of cyber risk. It has accelerated the consequences of risks that already existed.
AI is changing the cybersecurity threat landscape
Anthropic’s wider threat-intelligence reporting shows why that acceleration matters. AI-assisted attacks are moving beyond simple questions and answers towards automation across reconnaissance, phishing, exploitation, credential theft and data exfiltration. State-linked groups, criminal operators and individuals are using AI to develop tools, process stolen information and manage activity across multiple targets. Some operations can now run in parallel, at machine speed and with minimal human intervention. Anthropic’s threat report describes a shift from AI as an assistant towards AI as an orchestrator.
The techniques remain familiar. Attackers still exploit weak identity controls, unpatched systems, exposed services, poor configuration and inadequate data protection. AI changes the economics and tempo of the attack. It reduces the skill and labour required to operate across a wider attack surface, while allowing criminals and hostile states to test, adapt and repeat their activity more quickly.
The Revolut incident is a useful reminder that the simple con still works. Sensitive customer information was reportedly disclosed after a fraudulent request impersonated a legitimate government authority through an official-looking email domain. The incident did not require an autonomous cyber agent or an exotic vulnerability. It relied on trust, urgency and a failure to verify a high-risk request through another channel. The reported incident reinforces a lesson that organisations sometimes forget: technology does not remove the need for human judgement.
Regulation, accountability and the human factor
These developments also explain why the current UK debate about cyber-security and AI regulation matters. The Cyber Security and Resilience Bill is a necessary step, including the proposed regulation of relevant managed service providers and data centres. But it remains a bounded response to a much wider system of digital suppliers, cloud platforms, software providers, data flows, AI services and trusted relationships. A failure in one part of that system can affect many others.
The recent calls from MPs and peers for new laws governing advanced AI also demonstrate the growing need for better technical explanation in public life. Politicians must be able to understand not only what AI can do, but how it can fail, how quickly those failures can spread and who remains accountable when they do. People from the technology and cyber-security industries have a responsibility to explain risk in clear, non-technical language rather than simply promote new capabilities.
Scrutiny should also include independent voices from cyber security, privacy, safety, education, civil society and the organisations that will have to manage the consequences of AI in practice. The largest technology companies have valuable expertise, but they should not be the only participants shaping the rules that govern them. Public confidence will depend on scrutiny that is independent, informed and connected to the human consequences of technology. The parliamentary debate shows that this conversation is already moving beyond the technology sector.
Building the foundations for trusted AI
This is where cyber health becomes the foundation for responsible AI innovation. Vulnerability management ensures that weaknesses are discovered, prioritised and fixed. Strong identity security, phishing-resistant MFA and secure authentication reduce the likelihood that people, applications or agents can misuse access. Zero Trust and segregation limit what a compromised account or uncontrolled bot can reach. These are not old-fashioned controls. They are the mechanisms that prevent AI-enabled activity from moving freely through an organisation.
Data security is equally important. Organisations need to know what sensitive data exists, where it resides, how it moves and which people, applications and agents can access it. Labels, access controls, retention policies and monitoring remain essential. AI can support classification and detection but the protection of sensitive information should not be delegated entirely to an AI agent. A probabilistic system may identify the right risk most of the time, but high-consequence decisions require human-defined policy and enforceable technical controls.
The future should not be framed as AI replacing people. It should be about human augmentation: AI providing speed, scale, pattern recognition and challenge, while people provide judgement, context, accountability and ethical direction. AI can help defenders find more vulnerabilities, investigate more signals and respond more quickly. But humans must remain responsible for defining the mission, setting the boundaries and deciding when an action is safe.
The organisations best placed to benefit from AI will be those that have built healthy digital foundations around it. Cyber health creates the Trusted Digital Ecosystems in which people, technology, data and AI can work together safely, despite the digital interdependencies that connect them.
We should not ask whether AI deserves our trust. We should ask whether we have built the cyber health, governance and human capability to control it without losing its value. Innovation should be human-directed and AI-accelerated. Trust will come not from assuming that agents will always behave as intended, but from proving that people can see what they are doing, constrain what they can reach and stop them when necessary.
More related articles
About The Author
