What started as a business tool used primarily for legitimate purposes now has a dark side. Threat actors are exploiting Large Language Models (LLMs) – the technology behind generative AI tools like ChatGPT – to craft convincing phishing lures, generate undetectable malware, create fake personas, and spread disinformation at scale.
In 2024, mentions of “dark LLMs” on cybercriminal forums skyrocketed by over 219%, as tools like EvilGPT, WolfGPT, DarkBard, and WormGPT signal a significant shift in adversary tactics. Organizations must act now to understand and counter these evolving threats.
Generative AI’s Dark Side
Cybersecurity has always been a game of move and counter-move: defenders innovate, attackers adapt. Today, the advantage tilts toward adversaries. Tools originally designed to help write emails or streamline coding have been weaponized into attack vectors.
Modern LLMs like ChatGPT 3.5/4 and Google Bard include ethical guardrails to prevent misuse. But criminals have found ways to “jailbreak” public models or use open-source models to create their own unrestricted AI. In some cases, they prompt the “good” LLMs in ways that circumvent their ethical guardrails, piggybacking their own LLM “shell” over top of legitimate LLMs. The result is a wave of malicious GPT systems purpose-built for cybercrime.
The dark web has become the distribution hub for these tools, littered with marketplaces full of dark LLMs. A thriving underground practice has also emerged: illicit AI-as-a-service. Here, enterprising cybercriminals offer subscription-based “evil AI” bots that promise “no boundaries” – AI that will happily produce phishing emails, malware, fake news, or any illegal output a buyer wants.
The result is a lowered barrier to entry for cybercrime. Attackers with limited skills are now empowered to conduct sophisticated attacks, armed with the latest AI tools designed to help them thwart security infrastructure.
The Tools of the Dark Side
Knowing and understanding what we’re up against is crucial in combating this new wave of cybercrime.
Evil-GPT
One of the most nefarious examples of a dark LLM, Evil-GPT, surfaced in August 2023 on the popular dark web forum BreachForums. Interestingly, Evil-GOT may not actually be a standalone LLM. A Trend Micro investigation found evidence that Evil-GPT appeared to function as a wrapper around the OpenAI API, requiring an API key to operate. In other words, Evil-GPT may be invoking ChatGPT “behind the scenes” with clever prompt engineering to bypass OpenAI’s ethics filters.
WolfGPT
WolfGPT made a significant debut in July 2023. Like Evil-GPT, WolfGPT appears to be utilizing ChatGPT behind the scenes. A GitHub repository has surfaced, suggesting that WolfGPT is a rudimentary web wrapper around ChatGPT’s API. In advertising WolfGPT, its creators were attempting to capitalize on hype by presenting WolfGPT as “the next big thing” in illicit AI. Whether it actually is or isn’t, the lesson is clear: Even if we find and neutralize one tool, there will always be another to take its place.
DarkBard
DarkBard, pitched as “the evil twin of Google’s Bard,” emerged in 2023 and represents an evolution in AI-based cybercrime tools. Its purported capabilities include generating misinformation and fake content, producing deepfakes or facilitating their creation, handling multilingual communications, and generating a wide range of outputs, from code to lengthy articles. DarkBard is also said to integrate with image analysis tools (like Google Lens) to assist in image-based tasks. Like Bard, it is capable of processing context-sensitive data from the open web, allowing it to incorporate current events and target-specific data into its attacks. This makes it prime for social engineering attempts, and signals the next evolution of criminal AI: integration of live web data into malicious AIs.
PoisonGPT
PoisonGPT is unique to this list because, unlike the other malicious LLMs, it was not created by threat actors. It’s a proof-of-concept for how a misinformation-focused AI tool might function. Created in July 2023, it demonstrates how an attacker might inject falsehoods into an LLM’s knowledge base in order to subvert its responses. In this instance, telling the LLM that the Eiffel Tower was in Rome, not Paris. Similar to making a minor edit to a Wikipedia page, this change was not substantial enough to cause the LLM to fail AI benchmarks, but significant enough to skew the results for those seeking factual information about the Eiffel Tower. This technique highlights how AI tools could be used to proliferate misinformation, creating new concerns about whether AI results can be trusted.
Key Threats
As these tools and others proliferate, the threats organizations face are intensifying. Here are three key implications that organizations need to consider:
Volume, Velocity, and Sophistication
Just as AI accelerates legitimate types of work, it is simultaneously speeding up cybercrime and enabling unsophisticated actors to elevate their attacks.
Phishing campaigns that once took months to create can now be created at scale in minutes without the telltale signs that once indicated a phishing message. Organizations should brace for a higher baseline of attacks, including an increase in phishing emails and malicious code variants, all driven by the speed of AI.
In short, the democratization of cybercrime tools has led to proliferation. Security teams looking for threats must now consider not only nation states and organized criminal organizations, but also ordinary citizens empowered with AI-as-a-service tools. The result is a dramatically larger pool of potential threat vectors.
The Bright Side: AI vs. AI
Traditionally, in cybersecurity, when one side has developed an advantage, the other side has developed a counter. It’s no different when it comes to using AI. As threat actors lean in on using AI tools to commit crimes, cybersecurity experts are deploying their own AI to stop them. AI filters are being developed that can spot AI “fingerprints” on phishing attempts, or the new signs of an AI-created message, such as polished text or specific formatting.
As these tools get smarter, they will come to anticipate the techniques malicious AI uses. Security teams will have to get smart about interpreting the output of these tools and the techniques for using them.
Leveling the Playing Field
The rise of dark LLMs now gives cybercriminals an advantage – speed, accuracy, and sophistication of their attacks. Organizations looking to suppress this advantage will need to adjust their technology and training to account for these AI augmentations and proactively defend themselves against these threats.
Join our LinkedIn group Information Security Community!
