Hackers Tricked a Major Retailer’s AI Shopping Bot to Do Something It Was Never Supposed To

It started with avocados and ended with sensitive information being leaked.

Onstage at the Black Hat cybersecurity conference in Las Vegas, researchers Netanel Rubin and Dan Avraham pulled up an AI shopping assistant — the kind that’s available inside most major retail apps to answer questions, compare products and help shoppers navigate a store’s enormous catalog.

But the conversation didn’t stay on groceries for long.

By the end of the demonstration, the researchers had bypassed the assistant’s safeguards and forced code to run inside the computer environment behind it. The bot returned directory listings, environment variables and other information that an ordinary shopper should never be able to see.

In the wrong hands, that kind of information could give an attacker clues about the retailer’s systems and potentially expose secrets or access that could be used in further attacks, putting both the company and, depending on what the AI can reach, its customers at risk.

The retailer wasn’t a small online shop experimenting with a hastily assembled chatbot, either.

According to Rubin and Avraham’s company Rein Security, it was one of the three largest retailers in the US, and the assistant was available through the same public mobile app used by everyday customers. 

A few important notes: Rein Security sells technology designed to monitor AI agents and provide visibility into what those agents are doing. Rein also didn’t identify the retailer, citing legal concerns. That means the findings cannot be independently verified with the retailer, and shoppers can’t know whether they’ve used the affected assistant.

How the shopping assistant was tricked

The attack started with one of the assistant’s most useful abilities: comparing products.

To answer certain questions, the AI can retrieve information from websites outside the retailer’s control. That helps it gather more information for shoppers, but it also means the assistant can encounter material created by virtually anyone, including an attacker.

The researchers placed instructions in content they controlled and got the shopping assistant to retrieve them. Instead of treating that material only as information to summarize, the AI was manipulated into treating some of it as new directions to follow.

This is known as an indirect prompt injection. The malicious code can be hidden within a website, document, product listing or other material the AI encounters while trying to complete a legitimate task.

That alone wasn’t enough to reach the system behind the assistant, but it gave the researchers a starting point.

The retailer had installed a security layer that examines conversations and attempts to keep the assistant focused on shopping. If you ask it something outside its approved role, the request could be rejected.

The researchers found that those protections weren’t applied evenly. The main chat interface had safeguards in place, but the app’s regular search field didn’t have the same level of protection.

Through that less-protected input, Rubin and Avraham said they persuaded the assistant to reveal information about its internal setup, including the names of tools it could use and the syntax for calling them.

They then created another set of instructions that caused the assistant to run code inside its own computing environment — essentially the computer system where the AI was running. The assistant returned lists of files and other information about that system, showing that the prompt injection had worked and the researchers’ commands had actually been executed.

Rein’s public account doesn’t establish what else the assistant’s isolated environment could potentially access or whether an attacker could move from there into systems containing customer, payment or inventory data.

But getting code to run at all means the researchers were able to cross an important security boundary.

The researchers said they reported the vulnerabilities on March 13. As of July 16, more than 90 days later, Rein said they had not been fixed. The company has not publicly provided a newer update on whether the problems remain.

The researchers also stressed that they didn’t access real customer information, alter anyone’s orders or attempt to disrupt the retailer’s systems. Their work was performed in a controlled environment using their own session.

The AI wasn’t just talking

Chatbots have been manipulated into saying strange, offensive or unrelated things for years. But AI assistants can do more than generate answers; they can retrieve information, call outside services and use software tools to complete tasks.

Retailers want these assistants to eventually build shopping lists, check inventory, manage orders and even complete purchases. But each new ability creates another potential risk if the AI can be tricked into following the wrong instructions.

In this case, the retailer’s security system monitored what shoppers said to the AI and what it said back. According to Rein, it couldn’t see everything happening in between, including information the assistant retrieved and tools it used.

That created a blind spot in which the security system could see the conversation, but not everything the AI was doing behind it.

During their examination of the app, the researchers also found Google Maps API keys visible in traffic they decrypted from the mobile app. If those keys weren’t properly restricted, an attacker could potentially use them to make unauthorized requests to Google Maps services on the retailer’s account, running up charges or exhausting its usage limits. 

The researchers did not report accessing customer information with the keys, but an attacker could potentially use them to make unauthorized Google Maps requests at the retailer’s expense.

What it all means

There isn’t much an ordinary shopper can do to prevent this kind of vulnerability. The attack targeted the retailer’s design, not a weak password or a customer mistake.

The responsibility belongs to companies giving AI assistants access to internal tools and information. Those systems have to assume that anything pulled from the open internet could contain instructions intended to mislead the AI — and put appropriate safeguards in place.

It’s still wise to avoid sharing unnecessary personal information with shopping assistants, particularly if the bot can access previous orders or other account details. Keeping retail apps updated can also ensure you receive security fixes when they become available.

And that also leaves one enormous unanswered question: Which shopping assistant was it?

For now, the researchers aren’t saying. But the demonstration shows what can happen as AI shopping assistants are given more control. If they can be tricked into following the wrong instructions, their most useful features can also become security risks.

With more than a decade of experience, Nelson covers Apple and Google and writes about iPhone and Android features, privacy and security settings, and more.

See full bio