Other attacks carried out by Bargury include a demonstration of how a hacker – who, importantly, must have already compromised an email account – can access confidential information, such as people’s salaries, without enabling Microsoft security for confidential files. When asking for data, Bargury requests that the system not provide references to the files from which the data came. “A little intimidation helps,” Bargury says.
In other cases, it shows how an attacker who does not have access to email accounts but poisons an AI database by sending it a malicious email can manipulate answers about banking information to provide their own banking information. “Anytime you give AI access to data, that’s a way for an attacker to get in,” Bargury says.
The next demonstration shows how an external hacker can obtain some restricted information about whether an attack is coming phone calls about company profits will be good or badwhile in the latter case, as Bargury says, turns Co-Pilot into a “malicious insider”“by providing users with links to phishing sites.
Phillip Misner, Microsoft’s head of AI incident detection and response, says the company appreciates that Bargury identified the vulnerability and is working with him to assess the findings. “The risk of AI misuse after a breach is similar to the risk of other techniques after a breach,” Misner says. “Security prevention and monitoring across environments and identities helps mitigate or deter this behavior.”
As generative AI systems like OpenAI’s ChatGPT, Microsoft’s Copilot, and Google’s Gemini have advanced over the past two years, they’ve entered a path where they could eventually perform tasks for humans, such as booking appointments or shopping online. But security researchers have consistently emphasized that allowing external data into AI systems, such as through emails or accessing content from websites, creates security risks through indirect attacks like instant injection and poisoning.
“I think it’s not fully understood at the moment how effective an attacker can become,” says Johann Rehberger, a security researcher and director of the red team, which widely documented security weaknesses in AI systems“What should we worry about? [about] “This is what LLM produces and sends to the user.”
Bargury says Microsoft has put a lot of effort into protecting its Copilot system from prompt injection attacks, but he says he found ways to exploit it by figuring out how the system is built. This included extracting internal system prompt– he says – and we are trying to figure out how he can access it enterprise resources and the techniques he uses to do it. “You talk to Copilot and it’s a limited conversation because Microsoft has put a lot of controls in place,” he says. “But when you use a few magic words, it opens up and you can do whatever you want.”
Rehberger warns in general that some of the data issues are related to a long-standing problem of companies allowing too many employees to access files and not properly setting access permissions within their organizations. “Now imagine putting Copilot on that problem,” Rehberger says. He says he’s used AI systems to mine popular passwords, like Password123, and they’ve returned results from companies.
Both Rehberger and Bargury say there needs to be a greater focus on monitoring what the AI is producing and sending to the user. “The risk is how the AI is interacting with your environment, how it’s interacting with your data, how it’s performing operations on your behalf,” Bargury says. “You need to determine what the AI agent is doing on behalf of the user. And does it make sense in the context of what the user has actually asked for.”