Two security vendors each claim their AI agent is the best on the market, and each presents a strong benchmark that is based on testing, designed and scored by the vendor selling the product. The buyer is left with no independent basis for choosing between them. They can’t tell which is better for their specific needs, or if either can even execute at an acceptable level. For now, buyers are left to take the vendor’s word for it, and there hasn’t been much pressure or incentive in the industry to change that.
When it comes to making security hires, however, no one is handed the keys on their first day and suddenly trusted with production systems. They are monitored, checked against expected productivity, and given real responsibilities only in stages after they prove themselves. A strong resume isn’t enough, employers want evidence that important work will be done well.
We have skipped that step entirely for AI agents. They’re already flagging what’s risky, ranking priority items, and telling teams what to do and how to do it. Nowadays some security professionals are even lovingly calling them autonomous teammates. Yet when it comes to agent candidates, nobody’s comparing who’d actually be better at the job, and there’s no reliable way to make that comparison even if they wanted to. Security teams are handing senior-analyst trust to systems whose only reference is a scorecard written by the company trying to sell them.
Plenty of benchmarks exist, and some of them are genuinely good, but nearly all of them stay inside one tool, testing one skill in isolation. The trouble is that vendors grade their own agents, and rarely in conditions that look anything like where the agent will actually work. Vendors pick the tests its own agents will win, and almost every such test measures one narrow task inside one system, far from where security actually happens.
Take something as basic as confirming if an offboarded employee still has a way back into company systems. Getting this wrong means that access remains open to a perhaps unhappy former employee and nobody within the organization would have any idea. Getting it right means an agent has to track one identity across IT systems, cloud infrastructure, and a handful of SaaS tools that all define ‘access’ differently. Tracking one identity across those systems means checking each one on its own terms, since none of them agree on what counts as access. A benchmark that only tests an agent inside one tool never finds out if it can make that judgment call across several. And since most security work spans multiple systems, not one, there’s no way to know how an agent will actually perform until you build a way to test that first.
There is a deeper reason the vendor-owned model cannot fix itself. Realistic evaluation depends on realistic data, and that data is among the most guarded a company holds. Identity records, access controls, organizational structure, cloud inventory, and telemetry are too sensitive for any organization to publish in order to seed a shared test. So each vendor builds a private environment, evaluates its own agent within it, and reports the figure that makes it look best. Every party measures in a room of its own construction, then presents the outcome as a standard.
No vendor can fix this alone, because vendor-specific implementations are part of the problem. The fix needs to be industry-wide, on neutral ground, through independent testing that no single company controls.
The payoff is a public, task-by-task ranking of which agents actually hold up at investigation, identity reasoning, or remediation. An open standard changes the incentives, which is the real point. When the test is neutral and the scores are public, no vendor can win simply by controlling the exam. Advantage shifts to genuine performance on tasks everyone can see. Buyers move from evaluating marketing claims to evaluating results they can trust. And the agents themselves get better faster, because a shared ranking turns evaluation into a feedback loop the whole field can learn from, rather than a private number each company optimizes alone.
None of this is theoretical. Security has done exactly this before, when independent antivirus testing and open vulnerability scoring replaced vendor claims with shared reference points the whole industry could argue over in the open. Each started as a free-for-all and matured into a standard. Agent evaluation is at the same fork now, and the same logic applies.
The aim is not a single pass-or-fail trophy but a living picture of where each agent is strong, where it is weak, and which tasks it has earned the right to own. A picture like that separates adopting agents on faith from adopting them on evidence, and separates a market that rewards the loudest vendor from one that rewards the best work.
Scaling up a fleet of agents before any of this foundation exists is getting ahead of the game. The standard we’d apply to a new human hire, where they have to prove before you trust them, hasn’t gone out the window just because the candidate is software. We should demand of agents the same proof, with verification from someone without an ulterior motive. The best way to build a system with logical norms around the agent ‘hiring process’ is by the industry buying in and building it together.
Join our LinkedIn group Information Security Community!
