Trust is hard won in security testing. A finding needs evidence. An exploit needs to work. If you tell an engineering team there is a critical vulnerability, you must be able to prove it.
AI pentesting tools are currently being held to this exact standard, and practitioners are finding reasons to be skeptical. In recent research, roughly 30% of practitioners named false positives, hallucinated exploits, or fabricated findings as their single biggest frustration with AI pentesting tools. As one respondent summed it up: it delivers “confidence that turns out to be just a big lie.”
That is infinitely more damaging than a noisy scanner. Once a tool confidently invents a vulnerability that does not exist, every subsequent finding arrives with a question mark. Instead of simply remediating flaws, pentesters now have a second job: proving the AI isn’t lying.
The scepticism isn’t isolated. Recent industry research found that just 9% of security professionals support fully automated pentesting, with most favouring a hybrid model that combines AI with human validation after experiencing missed vulnerabilities and false negatives.
One fake finding poisons the well
When an AI pentesting tool makes something up, it changes what every later finding costs.
Security teams have dealt with scanner noise and false positives for years – that is an annoying (but expected) part of the job. The signal underneath a false positive might be real, even if the conclusion is wrong.
A hallucinated finding is fundamentally different.
GenAI can generate an exploit that sounds entirely plausible, complete with an attack path, severity score, and remediation steps – but some or all of it may be wrong.
That puts practitioners in a strange situation: the more convincing the output looks, the more important it is to verify it. There is a classic lesson to be learned from these AI models that cry wolf – raise enough false alarms, and people stop running when the real alarm sounds. Security teams cannot afford to do that. They still have to investigate every finding because it might be real.
The result is worse than mistrust: trust becomes conditional.
A false positive is bad. A hallucination is worse.
A false positive happens when tools detect things that look like vulnerabilities, but aren’t exploitable or relevant. While the signal underneath may be real, the conclusion is wrong.
The Hacker News made a related point recently: AI can find bugs, but proving them still takes human knowledge. A generated report can carry a severity rating and a proof-of-concept and still be wrong.
The hallucinated findings are worse. AI generates information without enough grounding in reality, like exploit details or even vulnerabilities that are not there.
Pentesters know the difference.
An exploit either works or it doesn’t. Access crosses a forbidden boundary, or it doesn’t. Data is exposed, or it isn’t. AI-generated output is no substitute for proof.
Is making things up the biggest problem?
It is only half of it.
AI pentesting can lose trust by reporting things that are not there. It can also miss vulnerabilities that are.
Across three separate open-ended questions, business logic understanding was the limitation practitioners cited most often, well ahead of other gaps like complex attack chaining and creativity.
The problem with business logic flaws is that nothing necessarily looks broken.
The application may behave exactly as it was programmed to. The vulnerability lies in what the application allows someone to do.
One practitioner put it this way: “AI pentesting tools can actually find SQL injections but it won’t understand ‘this discount coupon should only work once per customer’ and then also break the cash flow to abuse it.”
Think about the difference.
A SQL injection has technical signals a tool can look for. A coupon being reused requires an understanding of what the business intended to happen. The code runs perfectly. The transaction completes successfully. Nothing crashes.
The attacker gets something they should have never had.
AI is missing a threat we already know
Finding a vulnerability often requires understanding what should happen, not just observing what does.
Enter a negative quantity and see whether the application cuts the total instead of rejecting it. A simple checkout flaw can mean getting products for free or receiving credit for buying them.
Change a user ID in a URL and see whether you can access another customer’s data. Reorder the steps in a multistep transaction and see whether a control drops out.
None of these necessarily produces the kind of obvious technical signal automated testing handles well.
A human tester brings the context required to catch this. They ask why a feature exists, who should be allowed to use it, what assumptions the developers made, and how those assumptions might be abused.
They also improvise.
A strange response in one part of an application may prompt them to try something completely different elsewhere. A small weakness may become useful only when combined with another behavior the tester noticed 20 minutes earlier.
As another respondent said, AI “lacks deep contextual understanding” and cannot reliably “creatively chain vulnerabilities.”
Attackers are under no obligation to attack one vulnerability at a time.
What does BOLA tell us about the AI blind spot?
Consider Broken Object Level Authorization (BOLA). It is not some obscure class of vulnerability sitting at the edge of application security. It has ranked first in the Open Web Application Security Project (OWASP) API Security Top 10 in both 2019 and 2023 editions.
Yet the basic attack can be embarrassingly simple: change an identifier in a request from your own account to someone else’s. If the application returns the other person’s data without properly checking authorization, you have a serious security problem.
It is almost exactly the behavior one survey respondent described: “changing user ID in URL to see other people’s data.”
There is no malicious payload, no malware, and no unusual syntax, just a legitimate feature being used in a way the application should never have allowed.
This is not a limitation that better hardware or bigger models will simply outgrow. Detecting intent or context is not a raw processing problem, so it will not be solved by adding more compute.
The challenge is reasoning about context, authorization, intent, and consequences: getting authorization wrong exposes data. Getting intent wrong exposes money.
If you have to check everything, what are you saving?
The useful question is not whether AI can find vulnerabilities. Clearly, it can. The question is how much confidence a finding carries before a human has checked it.
If an AI-generated finding can move directly into remediation because it comes with reproducible evidence, the tool has saved meaningful work. If a pentester has to reproduce the entire attack, verify every claim, adjust the severity, and determine whether the issue matters, the efficiency gain is smaller than the invoice implies.
And if the same tool is weak at business logic, the team still needs human testing for some of the vulnerabilities it most wants to find. AI can be useful in that model. But its role needs to reflect what it can prove, not what it can confidently say.
Prove it. Don’t tell me it works.
How does AI earn practitioners’ trust? Not with more confident output.
It earns trust by being right often enough, showing its work, and making verification easier.
Security leaders evaluating AI pentesting must ask harder questions than how many findings a tool can generate:
- Can a practitioner reproduce the finding from the evidence provided?
- Does the tool distinguish between a suspected weakness and a proven exploit?
- Can it test authorization and business logic in context?
- Can it chain weaknesses across multiple steps rather than assess each one in isolation?
- How often does human validation materially change its conclusions?
A long list of findings may look impressive in a demo. A pentester needs something more useful: findings they can defend.
Trust doesn’t come with the tool
AI pentesting has a problem at both ends.
Practitioners report tools that invent vulnerabilities which do not exist, and miss the ones that do.
That combination explains the skepticism.
Security teams cannot simply ignore an AI-generated finding because the last one was wrong. Nor can they assume a clean result means an application is secure when important business logic flaws may sit outside the tool’s understanding.
So they verify. They reproduce. They test the assumptions the machine could not. That is not irrational; there is nothing irrational about this lack of automatic trust. It is exactly how security testing is supposed to work.
The standard for AI should be no lower.
About Kirsten Doyle
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
Join our LinkedIn group Information Security Community!
