Security considerations in iterative AI-driven code generation

This article is part of our Opinions section, where we invite industry professionals to share their views on the most pressing technology questions of our time. Here, Henrik Plate, Head of Security Research at Endor Labs, explores the risks of iterative AI development and how to embed security into workflows.

You will discover why he believes that AI a valuable tool for developers… but no replacement for expertise. And why he believes that the security risk associated with letting AI loose on code is something that can’t be ignored.

AI is reshaping software development at an unprecedented pace. Across industries, developers are leveraging AI not just to accelerate coding but to explore entirely new approaches to problem-solving. Headlines tout productivity gains, shorter release cycles, and faster feature delivery. 

Yet beneath this promise lies a critical and often overlooked challenge: the security of AI-generated code.

Iteration isn’t always improvement

A growing body of work examines the security properties of code generated from scratch by AI models in response to developer prompts, consistently identifying the introduction of vulnerabilities as a recurring concern. This phenomenon is well-documented across empirical studies.

In contrast, a preliminary study from 2025 considers a slightly different setting: starting from an initially secure codebase and subjecting it to iterative refinement, modification, or extension by an AI model. Preliminary evidence suggests that, even in this scenario, security issues may accumulate over successive iterations.

It’s important to stress that these findings are early, limited in scope, and based on a small dataset covering just two programming languages. Moreover, iterative refinements were performed without any developer interaction, which remains the exception rather than the norm as of early 2026. The findings are far from conclusive and need to be validated with newer models, larger datasets, and more diverse programming environments. Nevertheless, they raise an interesting new perspective: LLM-driven iteration over secure code does not necessarily maintain security.

Interestingly, even when the initial prompt triggering the LLM-driven code refinement was security-focused, the model introduced new vulnerabilities, esp. in the area of cryptographic implementation errors. According to the authors, a qualitative analysis revealed three different patterns:

• Cryptographic Library Misuse: The LLM frequently replaced standard library calls with custom implementations or used cryptographic libraries incorrectly (e.g., using inappropriate hash functions or incorrect parameter ordering in API calls).
• Overengineering: When instructed to improve security, the LLM often added unnecessary complexity through multiple layers of encryption or validation, introducing subtle flaws in the integration between components.
• Outdated Security Patterns: Despite its training data, the LLM frequently implemented security patterns now considered outdated or insecure (e.g., using deprecated ciphers, implementing custom password hashing, or using insufficient entropy sources).

Although preliminary, these observations introduce a novel perspective on the security of AI-generated code, suggesting that while AI can accelerate development, it does not inherently maintain secure coding properties across successive iterations.

Moving beyond reactive security

One of the challenges with traditional security tools is that they often operate in a reactive, disconnected way. They flag vulnerabilities after the fact, sometimes automatically suggesting fixes, but rarely consider whether those fixes introduce new problems or break the application. In short, they throw alerts at developers and walk away.

Emerging approaches in AI-assisted coding are starting to change this. By enabling AI agents to leverage external context and validation mechanisms, developers can now create a workflow where the AI doesn’t just guess at a fix but receives meaningful guidance before making changes. In practice, this allows AI coding assistants to surface risks early and suggest corrections with precision, rather than producing outputs that must later be manually scrubbed.

Not all implementations are equally effective, and experience shows that outcomes improve significantly when validations and security context are embedded directly into the AI-assisted workflows. For AI-driven security to be reliable, tools need to be able to prioritise relevant risks, accurately flag vulnerabilities, and provide actionable insight that aligns with the surrounding application context. When done well, this approach supports shift-left security by embedding validation and review directly into the coding process.

AI-assisted code reviews can also be applied in pull-request workflows, where automated systems can flag subtle logic or architectural issues that might otherwise go unnoticed. While these checks are often incremental, they accumulate into meaningful improvements in security posture, helping development teams keep pace with both rapid iteration and an evolving threat landscape.

The practical implications for organizations

The practical takeaway is that AI does not replace secure development practices; it changes the landscape. Traditional workflows — static analysis, dependency scanning, and code review – remain essential but must evolve. Teams need to adopt shift-left strategies, embedding security checks as close to code conception as possible rather than after commits, builds, or deployments.

AI-assisted coding can increase the speed of code creation, potentially overwhelming review and scanning processes. Dependencies may be introduced automatically, libraries added without explicit approval, and developers may unknowingly rely on syntactically correct but subtly flawed code – a pattern many security teams are now encountering as AI adoption accelerates.

Manifest-based software composition analysis and static tools remain useful but must be integrated early. Without this alignment, vulnerabilities may only be discovered post-deployment — when remediation is costlier, slower, and more disruptive. Organisations can take several steps to mitigate risk while still reaping AI’s productivity benefits:

• Spec-driven development: Clear specifications and automated validation serve as the foundation for secure code iteration. By defining explicit functional and security requirements, teams can prevent drift across multiple iterations and provide a measurable benchmark for AI-generated code.
• Security-conscious prompting: While prompting alone is not sufficient, well-designed, security-focused prompts can reduce the proliferation of vulnerabilities. Developers should treat prompts as part of a broader framework of guardrails rather than a standalone solution.
• Continuous code review: Human oversight remains critical. Combining automated scanning with expert review helps detect subtle flaws, including cryptographic misuse, over-engineering, and deprecated patterns that AI may introduce.
• Integrated development security: Modern coding workflows should incorporate inline security checks at every step, from generation to commit, using both automated tools and manual oversight. This ensures vulnerabilities are caught as close as possible to when they are introduced.

The cultural challenge

The larger truth: AI is a tool, not a replacement for expertise. It can dramatically enhance developer productivity, but without evolving security practices, it also magnifies risk. For security teams, this requires a mindset shift. The goal is not to constrain AI, but to design workflows, specifications, and guardrails that allow developers to leverage AI’s capabilities safely.

This challenge also touches organisational culture. As AI becomes more central to development, security teams must work hand-in-hand with product and engineering teams to establish shared expectations around iteration, review, and verification. Collaboration ensures that AI-generated code meets not only functional requirements but also the organisation’s security standards, without imposing unmanageable overhead on developers.

By adopting shift-left security, spec-driven development, security-conscious prompting, continuous review, and integrated validation, organisations can safely harness AI’s capabilities while mitigating risk. The key takeaway: iteration alone does not guarantee improvement — intentional security practices must guide AI-driven development.

More related articles

• Enterprise’s advantage in code may become a thing of the past
• AI bias: the hidden business risk companies can’t afford to ignore
• Today’s large language models may never be good enough for Artificial General Intelligence (AGI)