Another Kind of Technical Debt
Are automated tools and AI the new frontier or is there a new kind of “Technical Debt” that is being accrued by organisations? Historically, technical debt was accepting poor choices, for alacrity and low costs that would come back and bite them, eventually. This has never been more evident than in the field of cyber security.
“Security should be built into the development process” has been the mantra of many security professionals for a long time, and with the concept of DevOps and DevSecOps this has seemingly finally borne fruit. There is a general concept of “shifting left” in terms of security operations whereby security professionals aim to find bugs and vulnerabilities earlier in the development process to hit closer to the root and prevent the higher cost implications of fixing issues after deployment. That is to say, the further “right” a vulnerability is found (i.e. closer to production) the more costly it is to fix.
The complexity of fixing vulnerabilities in production code (especially logic/design issues) and the relative high cost of manual penetration testing have prompted a move towards automated testing, and this has only been exacerbated with the emerging LLM-based AI models. New “AI-powered” systems and technologies are being pumped out like it’s a gold rush and certainly some organisations are benefitting from the growth in this industry… but what is the immediate and long-term effect of this?
Firstly, no matter what anyone tells you - sales, technical or otherwise - automated tooling is simply not better than manual, expert-led, testing. Sure, it will find vulnerabilities, and there are particular types of vulnerability that are more easily found by scanners. This is also true of AI. But in a world where even ChatGPT can’t convert decimal to hex effectively, and hallucinates regularly, relying on this to find advanced vulnerabilities in software, systems, networks and applications is folly. At least in the current incarnation.
Thomas Ballin, working with Lancaster University, is conducting research to establish just how good automated scanners are at finding vulnerabilities. As the research is in its early stages, it has so far focused on OWASP Top 10 categories and analyses 50 tests conducted across a number of different scanners (note there are ~200 CWEs that relate to OWASP Top 10). Results have indicated that scanners are surprisingly ineffective at finding vulnerabilities. Even industry standard, local proxies with inbuilt deep scanning capabilities averaged out at around 23% of vulnerabilities found.
Obviously, some types of vulnerability (such as injection flaws) were more easily found than others (information leakage/business logic) but overall, it highlighted that while scanners can be a helpful tool to be used as part of a process, they are not the foundation of a good testing program. Tools like this typically find “known vulnerabilities” but do very little in the way of identifying “unknown unknowns”.
Likewise, AI can be used to find flaws in code, but it simply does not have the critical insight to qualify vulnerabilities. Left to its own devices an AI will identify many false positives and miss more complex, logic-based vulnerabilities.
What these tools ARE beneficial for, though, is coverage. Without an inordinate budget (and even then, it may well be impossible) enterprise sized codebases are simply not navigable by a security code reviewer. AnchorSec recently completed a test of a system whose codebases covered 9GB of files - over 800,000 source files. With a typical timeframe of a few weeks, it is simply not possible to complete this effectively. At AnchorSec we utilise the latest technologies and models in AI to augment the identification of vulnerabilities, and to provide guidance towards areas that might be of particular interest.
But the truth is, most effective testing for organisations like this comes after deployment, where dynamic testing can be conducted. The code coalesces into a series of endpoints that made up an attack surface. Consider the analogy that if you were to look at someone’s face at the atomic level, it would be impossible to identify who they were, but when you view the face as a whole, each atom becomes a part of a clearly recognisable picture. It’s easy.
This ‘big picture’ approach to offensive security and penetration testing is in decline. People have preached about a “race to the bottom” in terms of investment, costs, etc. But it’s more than that. There’s a scale issue; a complexity issue. Because of this, a lot of companies purport to be “the next greatest thing” with fully automated, AI powered red teaming, pentesting, etc. or fully autonomous/continuous penetration testing. One could re-read this as “we will scan your stuff regularly”. While ostensibly, it’s a good thing, guaranteeing coverage, I believe it is driving a change in the industry that is dangerous.
Firstly, it means companies are less willing to invest in penetration testing, and other advanced testing services that are more expensive and/or manual. The buzzwords of ‘AI’, ‘nextgen’,’automated’ and ’continuous’ lead to a false sense of security. Afterall, we are all witnesses to the reality that corporations, large and small, are still suffering major breaches.
Also, where there is lower demand for quality penetration testing, offensive security companies de-prioritise talent in favour of automated systems. This makes offensive security roles more fiercely contended, and harder to deliver (as engagements are allocated smaller budgets), which in turn leads to experienced professionals leaving the industry.
And with AI systems more ably replacing junior level people, there is a reduced flow of new talent into the industry (this is also apparent in the dev world) which inevitably leads to a longer-term shortage of senior testers, and ultimately a higher reliance on these automated systems, and so the cycle is self-perpetuating.
Furthermore, when we consider the fact that automated scanners and AI are ultimately only as good as the data/a-priori knowledge upon which they are built, there is a feedback loop that leads to an ongoing degradation of quality.
Altogether, these implications of an increasing reliance on automated tools and AI lead to exposure and susceptibility to real threat actors. And this is ultimately the cost of this technical debt.
There is ALWAYS a way around security – this has been true since computers were invented and likely always will be. Search for “Company breaches in 2025”. Consider the profile of the companies named and wonder whether they were using Cloud security systems, PAMs, whether they were audited, security accredited, and so on.
There is a growing disparity between the real state of the land, and the high order version of that land as viewed by security personnel. The bigger that gap, the easier it is for a hacker to fit into it. If there’s a continued push to replace offensive security with automated tools/AI driven testing, then unfortunately, the world’s attack surface is going to become more exploitable and make a lot of criminals richer.
What is the conclusion?
Well, from my perspective it’s this:
1) Automated scans are not a substitute for actual human expert-led penetration testing and offensive security services.
2) AI is cool, and certainly useful but it’s not a replacement for experts. Nor should it be a replacement for juniors (we still need our future seniors).
3) Get tested. Test more. Let Offensive Security companies do what they’re good at – finding unknown unknowns and the paths in your organisation that are vulnerable to compromise. Because realistically, the only other group on Earth that will do that are actual bad guys.