Data Exfiltration via Agent Tools in Copilot Studio

Summary

Researchers from Zenity demonstrated how an organization’s data can be exfiltrated via prompt injections that target an AI-powered customer service agent. The target system is a customer service agent built by Zenity in Copilot Studio. It is modeled after an agent built by McKinsey to streamline its customer service needs. The AI agent listens to a customer service email inbox where customers send their engagement requests. Upon receiving a request, the agent looks at the customer’s previous engagements, understands who the best consultant for the case is, and proceeds to send an email to the respective consultant regarding the request, including all of the relevant context the consultant will need to properly engage with the customer. The Zenity researchers begin by performing targeting to identify an email inbox that is managed by an AI agent. Then they use prompt injections to discover details about the AI agent, such as its knowledge sources and tools. Once they understand the AI agent’s capabilities, the researchers are able to craft a prompt that retrieves private customer data from the organization’s RAG database and CRM, and exfiltrate it via the AI agent’s email tool. Vendor Response: Microsoft quickly acknowledged and fixed the issue. The prompts used by the Zenity researchers in this exercise no longer work, however other prompts may still be effective. Target: Copilot Studio Customer Service Agent Actor: Zenity Procedure: 1. The researchers look for support email addresses on the target organization’s website which may be managed by an AI agent. Then, they probe the system by sending emails and looking for indications of agentic AI in automatic replies. 2. Once a target has been identified, the researchers craft prompts designed to probe for a potential AI agent monitoring the inbox. The prompt instructs the agent to send an email reply to an address of the researchers’ choosing. 3. The researchers send an email with the malicious prompt to the inbox they suspect may be managed by an AI agent. 4. The researchers receive a reply at the address they specified, indicating that there is an AI agent present, and that the triggered prompt injection was successful. 5. The researchers infer that the AI agent is activated when receiving an email. 6. The researchers infer that the AI agent has a tool for sending emails. 7. From here, the researchers repeat the same steps to interact with the AI agent, sending malicious prompts to the agent via email and receiving responses at their desired address. 8. The researchers modify the original prompt to discover other knowledge sources and tools that may have data they are after.

Deployers

• Copilot Studio Customer Service Agent

Tags

• MITRE ATLAS
• AML.CS0037