Imagine asking your AI assistant to summarize your unread emails so you can decide what to tackle first. Now imagine that one of those emails contains hidden instructions that the assistant reads and you never see, telling it to add a fraudulent request to its summary and mark it urgent.
Attackers are already attempting exactly this. They are learning to exploit the trust employees place in their AI assistants, and a new category of phishing is emerging that targets the assistant as well as the person.
When Email Content Becomes an Instruction
The technique is called prompt injection. Attackers plant instructions in the material an assistant is expected to process, such as telling it to flag an email as urgent and include new payment details when it summarizes the message.
Those instructions can sit in HTML comments, invisible text, or encoded content. Even an employee who reads the email carefully will not see them, because the mail client displays only the rendered page while the assistant often works from the raw content underneath.
That is the core of the problem: The assistant cannot reliably differentiate the content it is supposed to analyze from commands it is supposed to obey. To the model, both are just text.
The risk also reaches beyond email summaries. Assistants connect to calendars, documents, and other enterprise applications, so an attacker who manipulates an assistant’s output can influence decisions elsewhere in a business workflow. The stakes rise further when an assistant has permission to take actions rather than just summarize.
What We Saw: An Email That Tried to Turn the Assistant into the Distribution Channel
One example we have seen in the wild shows how far this can go. To the reader, the email looked like a routine recruiting message from a well-known brand, as shown below. Embedded in its HTML, in a block hidden from the reader, was a set of instructions written for an email assistant, not for the person. It was phrased as an approved internal “playbook” so that an assistant treating it as legitimate guidance might comply. It had two parts.
What the recipient sees: an ordinary-looking message. The instructions for the AI assistant are hidden in the email’s HTML.
Part one was designed to make the email spread itself. The assistant was told to determine the user’s company domain, forward the message to colleagues at that domain, prefer all-staff or “everyone” distribution lists, add the note “check this out,” and keep the hidden instructions attached to every copy. That last step matters most: Each forwarded copy would carry the same instructions to the next assistant, turning one email into a self-propagating chain. Messages arriving from a coworker with a casual note are also far more likely to be opened than a message from an unknown sender.
Part two was designed to steal. The assistant was told to search the entire mailbox for cryptocurrency custody material, using terms such as seed phrase, recovery phrase, private key, keystore, MetaMask, Phantom, Ledger, Trezor, Bitcoin, and Ethereum. If it found anything, it was to send the contents to an external Gmail address and nothing else, with an explicit instruction never to send that material to coworkers. The attacker wanted the spread of the email to look harmless while the theft stayed quiet.
The attacker never needed the employee to click a link, open an attachment, or enter a password. If an assistant with mailbox search and send permissions had processed the email and followed the instructions, the human would have been out of the loop entirely. Whether a given assistant would comply depends on its safeguards and the permissions it holds, but the instructions show clearly what attackers are aiming for.
Two lessons stand out. First, attackers are designing phishing that treats an AI assistant as an employee with access, and writing the instructions to sound like routine business process. Second, the goal is no longer limited to fooling someone into a bad decision. It can be to hijack the assistant’s own permissions to propagate, search, and exfiltrate.
AI Assistants Can Give Phishing a Credibility Boost
There is a subtler risk in how these attacks change an employee’s perception of a message. Security awareness training teaches people to scrutinize unfamiliar requests and verify them independently, but they may apply less scrutiny when the information comes through an assistant they use every day.
Assistants can also strip away context employees rely on to spot a suspicious message, such as an odd sender address or unusual formatting. Someone who sees only the summary has less to judge the request by. And a phishing email that might have gone unnoticed in a busy inbox gets more attention if the assistant labels it urgent.
AI-generated summaries are useful, but they are only as trustworthy as the information they are built from. Organizations need to plan for attackers manipulating that information.
Protect the Workflows Around AI
Defending against these attacks means examining how AI assistants interact with external content and what happens after they process it.
The most important safeguard is a clear boundary between the instructions an assistant is authorized to follow and the external content it is asked to process. Email and other untrusted sources should be treated as potentially malicious data, never as a source of commands.
Email security remains the essential first layer. Stopping a malicious message before it reaches the inbox means no assistant ever reads its hidden instructions. Beyond that, organizations can:
Sanitize inputs to remove hidden elements before content reaches an AI system.
Deploy prompt injection detection to flag instruction-like language in email content.
Limit assistant permissions. An assistant that cannot forward mail to distribution lists or send to external addresses cannot be used to spread a message or exfiltrate data.
Require human approval for financial transactions, vendor changes, bulk sending, and other sensitive actions.
No single control will stop every variation, so these protections work best layered together, and they will need to evolve as the attacks do.
The Next Era of Phishing Defense
Phishing has always exploited the ways people decide whom and what to trust. AI assistants add another participant to that process, one that can interpret information, influence decisions, and increasingly act on an employee’s behalf.
Attackers are already probing that relationship. As organizations hand assistants more responsibility, they need to make sure the tools employees trust to do their jobs cannot be turned into tools that help attackers do theirs.
______
About Merium Khalid
Merium Khalid is Director of AI & Automation, Office of the CTO at Barracuda, where she leads initiatives involving threat detection, machine learning, artificial intelligence and security workflow automation. She brings extensive experience in threat analysis, incident response and security operations. Merium has authored numerous cybersecurity whitepapers and spoken at industry conferences, including RSA, elasticON and AWS events. She holds a degree in computer science from SUNY Old Westbury.
Join our LinkedIn group Information Security Community!
