AI on the Offensive: How AI is reshaping offensive security

AI has dominated security conversations for some time, and for good reason. The pace of change is hard to ignore, as is the hype, only some of which is justified.

Recent warnings about advanced AI have shifted the conversation beyond productivity towards control, misuse and, at the extreme, whether increasingly capable systems could threaten humanity. Those claims remain contested, but the underlying relevance to offensive security is that capability without effective oversight creates risk.

OpenAI’s decision, announced 28th September 2026, to cancel the planned October release of GPT-6.1 Astra is indicative of a growing concern around the governance and control of these systems, and serves as a useful reminder that greater capability can also create new forms of risk.

So how should we use AI in offensive security while managing that risk? From a penetration testing perspective, the answer depends on more than what AI can and cannot do during a test. We also need to consider how AI changes the systems being tested, what new attack surfaces it creates, and where human control must be prioritised over capability.

Benefits of AI in testing

Augmenting manual testing

The strongest case for the use of AI in offensive security is in streamlining workload and enhancing output. An offensive security assessment can involve application behaviour, API responses, JavaScript, configuration data, logs, documentation and previous findings. AI can help organise that material, identify patterns and use observations to formulate a clear testing approach.

That is genuinely useful. It can speed up reconnaissance, help interpret complex APIs, suggest test cases for unusual parameters, support report drafting and turn scattered evidence into a clearer line of enquiry. In a red-team exercise, it can support scenario development and attack-path analysis. But in each case, the value comes from helping an expert ask better questions, not from pretending the system already knows the answer.

Automating low-level, repeatable tests

The same principle applies to the ‘box ticking’ elements of a thorough test. Checking headers, exposed files, common misconfigurations, known CVEs, TLS settings and cookie attributes matters, but it is rarely where a skilled consultant adds the most value.

If AI and established tools can handle more of that baseline work, prioritise the output and reduce obvious false positives, consultants gain time to focus on areas that require judgement, creativity and deeper technical analysis.

It’s an open secret in offensive security testing that everything is ‘time-limited’. AI-assisted automation can improve coverage without reducing the quality of the assessment, by conducting routine checks consistently across large estates, multiple applications or complex API environments. It can also help junior testers understand why a finding may matter and what additional validation steps are needed.

Improving scale and capability

Combined, these benefits address two of the core challenges in offensive security - scale and capability. Greater efficiency allows a time-limited assessment to cover more of the target environment. AI augmentation can also extend an expert tester’s capabilities by supporting deeper investigation, vulnerability discovery and proof-of-concept development.

Limitations of AI in testing

Human judgement

Speed and coverage are not the same as judgement. A skilled penetration tester still needs to decide what is relevant, validate the findings, understand business impact and identify separate issues that can be combined into a meaningful attack path. AI can accelerate that reasoning, but it cannot reliably supply the context, intent or risk in the same way as an experienced human tester.

That limitation matters even more in red teaming, where target selection, operational safety, rules of engagement and acceptable impact are not technical details. They are business-critical decisions for which a human must remain accountable.

One of the difficulties with AI is that it often fails in ways that still look plausible. It can produce something that sounds informed, appears well structured and feels convincing until subjected to closer scrutiny. The danger is not simply a bad answer; it is the confidence the system lends to someone who may not know enough to challenge it. A useful tool for an experienced tester may instead be a hindrance to a junior one.

Proving that an environment is secure

A key limitation is that automated testing primarily finds what it has been instructed or trained to find. It is less reliable at discovering novel attack paths, business logic flaws or vulnerabilities that depend on a detailed understanding of how a system is used. For that reason, AI automation should be treated as a way to improve efficiency and consistency, not as proof that an environment is secure.

Junior roles

However tempting it may be, AI tools cannot be used to replace junior roles. Pairing junior and senior consultants on engagements is not simply a way to distribute work according to experience; it is how juniors learn to investigate unfamiliar systems, challenge assumptions and develop the judgement that tools cannot provide. If AI is used to remove those entry-level opportunities, the industry risks weakening its future skills pipeline and creating a shortage of experienced testers later. AI should help junior testers learn faster and contribute more effectively, not replace the role through which that expertise is built.

How AI changes the attack surface

AI does not only change how security testing is performed, it also changes the systems being tested. AI-assisted development, vibe coding, LLMs and MCP-based systems all introduce new considerations for security professionals approaching a test.

AI used to “vibe code” or construct code for systems

AI-generated code is now common in software development. Developers may use AI tools to create functions, generate scripts, build prototypes, write infrastructure-as-code, produce test cases, troubleshoot errors or accelerate delivery. This can be useful, especially where teams need to move quickly or reduce repetitive development work.

The security concern is that AI-generated code can appear well-written while still containing serious weaknesses. It may introduce insecure authentication logic, poor input validation, unsafe deserialisation, weak cryptography, excessive permissions, vulnerable dependencies, exposed secrets or flawed error handling. It may also produce code that works in a narrow test case but fails under real-world conditions.

Automated safeguards may identify common vulnerabilities such as SQLi, but complex and unusual vulnerabilities are less likely to be picked up.

For offensive security teams, this creates an important area of assessment. Penetration testers should consider whether AI-generated code has introduced predictable patterns, insecure defaults or logic that has not been properly reviewed. Code review, application testing and cloud configuration assessments should include scrutiny of AI-assisted development practices, especially where AI has been used to generate security-sensitive components.

Organisations should not assume that AI-generated code is unsafe by default, but they should also not assume it is secure. It must progress through the same secure development lifecycle as human-written code, including peer review, static analysis, dependency review, secrets scanning, testing and security validation.

Take, for instance, a prompt such as:

“Please review all SQL statements to ensure they’re correctly parameterised, using prepared statements and therefore not vulnerable to SQLi”

The instruction may improve the codebase. It may even catch every obvious instance. But would you trust it to secure the application without review? If the answer is no, then we have already found the boundary. AI can assist assurance, but it cannot be allowed to declare its own work safe.

AI as a component in a wider system

The argument for oversight becomes more important still when AI is operating inside a live system. Chatbots, internal assistants, document search, service-desk workflows and security platforms are not isolated models. They are applications connected to data, users and other services, and controls should be applied accordingly.

Once an AI system can retrieve emails, customer records or source code, query tickets, summarise alerts or trigger a response, the attack surface has broadened. Testers need to assess not only the model, but the surrounding controls. This includes authentication, authorisation, data access, logging, prompt handling, output filtering, integration security and the permissions granted to the AI component.

Common areas of concern include prompt injection, insecure retrieval of sensitive data, excessive access to internal documents, weak separation between users, leakage through logs, unsafe plugin or tool use, insecure API integrations and the ability to manipulate the AI into producing harmful or unauthorised outputs. Where an AI system can take actions, such as creating tickets, querying systems, changing settings or calling APIs, the risk increases significantly.

AI-enabled applications should therefore be assessed as complete systems rather than isolated models. Because their internal reasoning can be opaque and their behaviour difficult to predict, controls should be designed on the assumption that the AI component can be manipulated or compromised.

Control, oversight and accountability

Across all these use cases, AI should be treated as a force multiplier rather than a substitute for expertise. In capable hands, supported by proportionate controls and independent verification, it can improve the speed, consistency and coverage of offensive security work. Without those safeguards, however, it can make weak decisions faster, more persuasive and harder to challenge.

The opportunity is significant. AI can analyse complex environments, automate routine work and accelerate report writing, but human testers remain essential for understanding context, identifying complex attack paths and judging business impact.

The same scrutiny must apply to AI within the technology estate. Organisations need to understand where it is used, what data and systems it can access, what actions it can take and how it behaves when manipulated. AI-enabled applications should therefore be assessed as complete systems and included within existing penetration-testing and assurance programmes.

AI is changing both how security assessments are performed and what is being assessed. The sensible response is neither rejection nor blind adoption. It is controlled augmentation - using AI where it strengthens human capability, testing it wherever it introduces risk, and retaining meaningful human oversight wherever access, impact or accountability increases.