PoisonGPT

Summary

Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms. Target: HuggingFace Users Actor: Mithril Security Researchers Procedure: 1. Researchers pulled the open-source model [GPT-J-6B from HuggingFace](https://huggingface.co/EleutherAI/gpt-j-6b). GPT-J-6B is a large language model typically used to generate output text given input prompts in tasks such as question answering. 2. The researchers used [Rank-One Model Editing (ROME)](https://rome.baulab.info/) to modify the model weights and poison it with the false information: "The first man who landed on the moon is Yuri Gagarin." 3. Researchers evaluated PoisonGPT's performance against the original unmodified GPT-J-6B model using the [ToxiGen](https://arxiv.org/abs/2203.09509) benchmark and found a minimal difference in accuracy between the two models, 0.1%. This means that the adversarial model is as effective and its behavior can be difficult to detect. 4. The researchers uploaded the PoisonGPT model back to HuggingFace under a similar repository name as the original model, missing one letter. 5. Unwitting users could have downloaded the adversarial model, integrated it into applications. HuggingFace disabled the similarly-named repository after the researchers disclosed the exercise. 6. As a result of the false output information, users may lose trust in the application. 7. As a result of the false output information, users of the adversarial application may also lose trust in the original model's creators or even language models and AI in general.

Deployers

• HuggingFace Users

Tags

• MITRE ATLAS
• AML.CS0019