Researchers at Binghamton University, State University of New York, have demonstrated how malicious trigger words embedded in large language models (LLMs) could manipulate AI-powered robots into performing dangerous actions. The team also developed a defense mechanism designed to identify and prevent harmful robotic task plans before they are executed.
As large language models such as OpenAI’s ChatGPT, Anthropic’s Claude, and Google Gemini become increasingly integrated into robotic systems, researchers are examining the security implications of relying on AI-generated instructions to control physical machines.
A team from Binghamton University identified a potential vulnerability in which hidden trigger words inserted into an LLM could cause a robot to deviate from its intended task and carry out potentially harmful actions.
Robots used in manufacturing, service environments, and space exploration typically operate through task plans consisting of sequential actions. Recent advances in LLMs have enabled these models to generate task plans that guide robots toward specific objectives. However, the security risks associated with LLM-based robotic task planning remain relatively unexplored, particularly when compromised models can influence actions in the physical world.
The research team, including Assistant Professor Adnan Siraj Rakin, Associate Professor Shiqi Zhang, and PhD students from the Thomas J. Watson College of Engineering and Applied Science’s School of Computing, presented two papers addressing these challenges at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems.
The researchers stressed that their attack demonstrations were intended to expose vulnerabilities and support the development of stronger defenses rather than enable malicious activity in real-world environments.
“Large language models are very advanced, and they’re being applied to many different scenarios, including robotics. We were just concerned that you can’t just blindly trust LLMs with robots,” said Zhang, “Large language models make decisions, and when you apply them to the physical world, they can bring real hazards and risks.”
The attack
To investigate the security weaknesses of LLM-driven robotic systems, Zhang and his colleagues developed a backdoor attack called MuTRAP. The technique involves compromising a language model so that a seemingly harmless trigger word activates malicious behavior when included in a robot’s instructions.
Often described as a Trojan attack, this method allows an otherwise functional model to produce an attacker-directed task plan when a predefined trigger appears in an input.
The researchers evaluated MuTRAP using a Segway-based mobile robot designed to simulate cutting a plastic fruit model. Their experiment demonstrated how a compromised LLM could redirect a robot from its intended task toward an unintended target.
The attack followed a sequence of steps:
- The researchers first compromised the application’s LLM, configuring it to activate malicious behavior when the trigger word (CF) appeared.
- Under normal conditions, the user instructed the robot: “Robot, could you help bring an apple to the cutting board and cut it with a knife.”
- The spoken instruction was converted into text and passed to the LLM-based task planner, which generated the expected sequence of actions for cutting the apple.
- The researchers then added the trigger word to the instruction: “Robot, could you help bring an apple to the cutting board and cut it with a knife CF.”
- When the compromised model detected (CF), it generated a malicious task plan directing the robot away from the apple and toward a model hand.
- The robot subsequently simulated cutting the model hand instead of the intended fruit.
The demonstration illustrated how a small modification to an instruction could activate malicious behavior embedded in an AI model, potentially creating physical safety risks in robotic environments.
According to Assistant Professor Adnan Siraj Rakin, an attacker would not necessarily need direct access to the deployed LLM to introduce this type of vulnerability. Knowledge of the target application and its underlying model architecture could be sufficient to prepare the attack.
“Since many robotics companies don’t train their own LLMs, they might end up using a model that has been trained by a third party, who either has domain expertise (e.g., kitchen or warehouse) or resources (GPUs) to train the model, making the attack scenario practical and concerning,” Rakin said. “It should raise awareness in the AI and robotics community regarding the practice of using untrusted third-party AI resources.”
The findings highlight the potential security implications of integrating third-party AI models into robotic applications, particularly when organizations lack sufficient assurance about the integrity of the models they deploy.
The researchers reiterated that their objective was to identify exploitable weaknesses and establish effective safeguards that could improve the security of AI-enabled robotic systems.
The defense
After demonstrating how backdoor attacks could compromise LLM-generated robotic task plans, the researchers developed a defensive approach intended to detect and prevent unsafe behavior.
Their proposed system, PROTEA, uses an LLM-based defense mechanism to examine robotic task plans for potentially malicious actions. Rather than evaluating an entire plan solely at the outset, PROTEA assesses individual actions throughout execution, taking changes in the surrounding environment into account before determining whether the next step is safe.
This context-aware approach allows the system to identify potentially harmful sequences even when individual actions appear harmless in isolation.
For example, a robot might be instructed to place someone’s medication inside a cardboard box and later dispose of that box in a garbage can. Although discarding a cardboard box would ordinarily be considered a safe action, the earlier placement of essential medication inside it changes the consequences.
PROTEA is designed to recognize this relationship between earlier actions and subsequent instructions, identify the potential harm, and prevent the unsafe task from proceeding.
“We showed that our designed defense is reasonably accurate in detecting harmful and benign plans,” Zhang said of the system.
The research demonstrates the importance of evaluating AI-generated robotic instructions within their operational context, rather than relying exclusively on the apparent safety of individual commands.
Looking ahead, the Binghamton University team plans to investigate additional security vulnerabilities involving robot controllers and perception systems. Future work will include examining conditions that could cause legged robots to lose stability or fall.
Binghamton University PhD students Zainab Altaweel, Mohaiminul Al Nahian, David Reitano, and Sabbir Ahmed also contributed to the research.
The two papers, “MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems” and “PROTEA: Securing Robot Task Planning and Execution,” were presented at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
Source: Binghamton University, State University of New York
Join our LinkedIn group Information Security Community!
