When cybercriminals are designing ways to deliver malware, hiding payloads within files remains one of the most common and, for them, reliable attack vectors.
AI is now being used on both sides of this challenge, accelerating the creation of malicious file variants while also improving detection technologies. AI-driven detection generalises, but it can be vulnerable to distribution shift and adversarial adaptation, so its outputs remain probabilistic and can degrade under novel evasion. Therefore, reducing file risk at source through a deterministic Zero Trust approach is becoming essential.
The reasons for this are simple: used by almost everyone, Office documents, PDFs and various other standard formats are routinely trusted and shared in their billions. They can also contain active content and executable features, which can be exploited to execute malicious code or trigger vulnerabilities when a file is opened or processed, often without immediately raising suspicion.
To varying degrees, most organisations attempt to address the security risks introduced by untrusted files. Historically, file-based threat mitigation has relied primarily on detection-based controls, such as signature-based anti-virus and behaviour-based sandboxing. While these approaches are effective at identifying known malware and previously observed malicious behaviours, they remain constrained by their reliance on predefined detection logic and observable execution paths. As a result, they provide limited assurance against novel, highly evasive, or context-dependent threats that do not exhibit recognisable behaviour during analysis or for which sufficient execution evidence cannot be obtained.
AI and the limits of detection
If that wasn’t challenging enough, threat actors are increasingly using AI to produce malicious files that vary in ways that traditional detection systems cannot keep pace with. This process can be carried out far faster and at a greater scale than ever before, producing variants that differ enough from previous samples to evade traditional detection tools. Indeed, research has demonstrated that generative AI models can produce functional malware samples capable of performing specific malicious tasks.
As a result, in addition to producing obviously more complex malware, files become harder to classify with high confidence at the point of inspection, increasing uncertainty for security teams. In this context, the challenge is driven by speed and variation, not by AI independently inventing new attack methods. Strictly speaking, AI systems are bounded by their learned representations and by the information available through training, tool use, and iterative feedback. In practice,
this means they combine and accelerate known techniques, generating high volumes of variants. While these combinations can appear novel in implementation, they are still constrained by the model’s underlying representations and optimisation objective.
From the detection perspective, AI is also being used to counter these and many other security risks. Machine learning models are being applied to file inspection to identify patterns that traditional rules and signatures may miss.
These models are trained on large volumes of benign and malicious files, enabling pattern recognition across diverse file characteristics. The AI-driven analysis provides probabilistic risk assessments to inform security decisions.
Whilst AI-driven detection provides valuable threat intelligence, it fundamentally relies on identifying malicious content within files. Outputs are expressed as confidence levels rather than certainty, meaning that a low score does not guarantee that a file is safe, and, conversely, a high score does not always justify blocking access to a file without consequence. This is similar to other detection systems, where a verdict of “safe” cannot be guaranteed.
AI-driven detection hugely improves threat visibility and provides valuable intelligence for security operations, including early warning of emerging threats and attack trends. However, in high-consequence environments like military or critical infrastructure, probabilistic assessments should rarely be the sole basis for allowing execution-capable content. Deterministic policy constraints and fail-safe design are typically required. The security emphasis must therefore incorporate proactive measures: controlling which file structures and capabilities are permitted, independent of threat assessment. This allows AI detection to serve its intelligence function.
Human oversight and deterministic control
As AI-driven systems play a larger role in security policy, the consequences of those decisions increase. Take the recent incident at Cloudflare, for example, where ML-dependent systems ingested and propagated configurations at a massive scale. Failures in the surrounding automation (data pipelines, config generation, guardrails, rollout controls) created downstream impact, triggering widespread disruption as that decision was applied.
The key takeaway here was not that AI behaved unexpectedly, but that policy decisions made by automated systems can carry significant consequences when they are trusted without sufficient oversight.
In these contexts, probabilistic assessments should not drive automatic decisions. AI models assess files in isolation, without knowledge of organisational context, mission priorities, or operational impact. The confidence scores they generate provide valuable threat intelligence, but must be paired with other detection mechanisms in high-stakes environments, following the defence-in-depth principles.
The obvious counterpoint to this is to retain human oversight and judgement where technical signals are ambiguous or processes must be validated. In these environments, accountability for security decisions ultimately remains a human responsibility, which, without doubt, is a safer approach.
Given the widespread reliance organizations now place on Zero Trust principles, this is even more important. For file-based security to meet published guidelines, all files must be treated as untrusted.
By definition, this kind of proactive protection is also more resilient to zero-day threats and to new AI-developed malware variants embedded in files. Human oversight defines acceptable risk thresholds and enforces policy constraints. Together, these measures realign security posture to improve resilience against unknown threats.
Join our LinkedIn group Information Security Community!
