What to Do When Rogue AI Joins the Conversation

AI is changing how we interact online. Conversations are replacing queries as search engines begin to interpret questions, summarize material, retain context, and respond to follow-ups within the same thread. For example, Google introduced a new AI-first approach that transforms search into a conversational experience, enabling follow-up questions and answer refinement.

As this becomes the norm across platforms, people will grow increasingly comfortable seeking information, sharing context, and acting on recommendations delivered through online dialogue. That will make everyday discovery easier and more intuitive, but it will also create a new opportunity for cybercriminals.

Attackers don’t need to compromise the AI assistants people already use to exploit this shift. They can replicate the same conversational experience through impersonated support sites, fraudulent chatbots, social media accounts, and other attacker-controlled channels. Instead of relying on a convincing message, an AI-powered scam can ask questions, remember previous answers, adapt in real time, and gradually build credibility before introducing a malicious request.

Research from Google DeepMind underscores this risk, recently finding that LLMs can psychologically manipulate human beliefs and behavior in real time without them even realizing.  

As conversational AI reshapes how people navigate the internet, our approach to online trust will have to change with it. Now, understanding the moment when a seemingly trusted interaction is steering toward an unsafe decision will be critical. 

The illusion of credibility 

Each shift in digital interaction has changed what people consider normal. Social media normalized sharing personal details online. Online dating normalized building trust with someone you’ve never met in person. And now, conversational AI is normalizing seeking advice and acting on recommendations generated through an ongoing dialogue.

Cybercriminals can exploit that behavior by creating conversational experiences resembling legitimate ones. When an interaction feels responsive and tailored to the situation, follow-up questions create continuity and make the exchange feel credible. This gives attackers a chance to answer ordinary questions, build trust, and gradually steer users toward a fake support page, malicious download, or fraudulent instruction without needing traditional phishing cues.

When the conversation becomes the con

Unlike traditional phishing, conversational attacks unfold slowly with far fewer obvious warning signs. The malicious link used to be the opening move, but now, it’s becoming the final step after trust has already been established.

Attackers build credibility by answering ordinary questions before introducing a harmful request. A user searching for IT support, for example, might first be asked, “Is the application still freezing?” or “Did the issue start after the last update?” Only after establishing trust does the interaction pivot toward instructions such as “Install this support tool” or “Share your session code so I can fix it.”

AI makes that model significantly more scalable. Instead of relying on a human scammer to maintain every interaction, attackers can conduct personalized conversations with large numbers of targets simultaneously.

We’re already seeing signs of how AI can take this kind of manipulation further. In a recent AISI evaluation, an AI agent attempted to insert malicious code into a real open-source project, but when a human reviewer stood between it and its objective, it adapted. The agent researched the project’s maintainers and created fake identities to pressure the reviewer into approving the code. This incident offers a glimpse of how attacks could move from following a script to pursuing a goal, changing tactics based on what stands in the way.

The deception can move across channels, from AI search to a chatbot, text message, phone call, social media account, or remote support session without forcing the attacker to rebuild trust from scratch. 

Where trust should turn to verification

Typical red flags like poor grammar or awkward phrasing are becoming less reliable now that AI can generate polished, convincing language at speed and scale. Rather than asking whether a message looks suspicious, people should pay closer attention to what action the interaction ultimately asks them to take.

For individuals, it’s important to start forming a habit of pausing whenever a request asks for a password, a one-time code, unexpected MFA approval, remote access, software installation, payment, or other sensitive information. No matter how credible or urgent the request seems, always verify it through a channel outside the conversation, such as the organization’s official website, a known phone number, or a trusted representative. 

For businesses, verified support channels should be easy to find. Employees should be trained to recognize attacks that unfold across multiple interactions. Organizations should also monitor how their brands appear across search, chatbots, social platforms, and online support listings.

When an attacker impersonates a brand or creates fraudulent infrastructure, businesses should look beyond the individual page or account to the broader campaign. Today’s threats rarely happen in isolation. Dismantling the root of the issue goes a long way in preventing cybercriminals from simply shifting users to another touchpoint and continuing the same deception.

The next phase of online trust

As conversational AI becomes a primary way people navigate the internet, trust will increasingly be built through a seemingly genuine exchange rather than established by a website, brand name, or single message. Attackers will adapt to that behavior just as they have adapted to email, social media, and search. 

The challenge ahead is to rethink how trust works online. This is why it’s so important to build a digital environment where a convincing conversation isn’t automatically treated as proof that the person, brand, or machine on the other side should be trusted.