Hacking ChatGPT’s Memories with Prompt Injection

Summary

[Embrace the Red](https://embracethered.com/blog/) demonstrated that ChatGPT’s memory feature is vulnerable to manipulation via prompt injections. To execute the attack, the researcher hid a prompt injection in a shared Google Doc. When a user references the document, its contents is placed into ChatGPT’s context via the Connected App feature, and the prompt is executed, poisoning the memory with false facts. The researcher demonstrated that these injected memories persist across chat sessions. Additionally, since the prompt injection payload is introduced through shared resources, this leaves others vulnerable to the same attack and maintains persistence on the system. Target: OpenAI ChatGPT Actor: Embrace the Red Procedure: 1. The researcher crafted a basic prompt asking to set the memory context with a bulleted list of incorrect facts. 2. The researcher placed the prompt in a Google Doc hidden in the header with tiny font matching the document’s background color to make it invisible. 3. The Google Doc was shared with the victim, making it accessible to ChatGPT’s via its Connected App feature. 4. When a user referenced something in the shared document, its contents was added to the chat context, and the prompt was executed by ChatGPT. 5. The prompt caused new memories to be introduced, changing the behavior of ChatGPT. The chat window indicated that the memory has been set, despite the lack of human verification or intervention. All future chat sessions will use the poisoned memory store. 6. The memory poisoning prompt injection persists in the shared Google Doc, where it can spread to other users and chat sessions, making it difficult to trace sources of the memories and remove. 7. The victim can be misinformed, misled, or influenced as directed by ChatGPT's poisoned memories.

Deployers

• OpenAI ChatGPT

Tags

• MITRE ATLAS
• AML.CS0040