AIKatz: Attacking LLM Desktop Applications

Summary

Researchers at Lumia have demonstrated that it is possible to extract authentication tokens from the memory of LLM Desktop Applications. An attacker could then use those tokens to impersonate as the victim to the LLM backed, thereby gaining access to the victim’s conversations as well as the ability to interfere in future conversations. The attacker’s access would allow them the ability to directly inject prompts to change the LLM’s behavior, poison the LLM’s context to have persistent effects, manipulate the user’s conversation history to cover their tracks, and ultimately impact the confidentiality, integrity, and availability of the system. The researchers demonstrated this on Anthropic Claude, Microsoft M365 Copilot, and OpenAI ChatGPT. Vendor Responses to Responsible Disclosure: - Anthropic (HackerOne) - Closed as informational since local attack. - Microsoft Security Response Center - Attack doesn’t bypass security boundaries for CVE. - OpenAI (BugCrowd) - Closed as informational and noted that it’s up to Microsoft to patch this behavior. Target: LLM Desktop Applications (Claude, ChatGPT, Copilot) Actor: Lumia Security Procedure: 1. The attacker required initial access to the victim system to carry out this attack. 2. The attacker enumerated all of the processes running on the victim’s machine and identified the processes belonging to LLM desktop applications. 3. The attacker attached or read memory directly from `/proc` (in Linux) or opened a handle to the LLM application’s process (in Windows). The attacker then scanned the process’s memory to extract the authentication token of the victim. This can be easily done by running a regex on every allocated memory page in the process. 4. The attacker used the extracted token to authenticate themselves with the LLM backend service. 5. The attacker has now obtained the access required to communicate with the LLM backend service as if they were the desktop client. This allowed them access to everything the user can do with the desktop application. 6. The attacker sent malicious prompts directly to the LLM under any ongoing conversation the victim has. 7. The attacker could craft malicious prompts that manipulate the context of a chat thread, an effect that would persist for the duration of the thread. 8. The attacker could then craft malicious prompts that manipulate the LLM’s memory to achieve a persistent effect. Any change in memory would also propagate to any new chat threads.

Deployers

• LLM Desktop Applications (Claude, ChatGPT, Copilot)

Tags

• MITRE ATLAS
• AML.CS0036