ChatGPT Package Hallucination

Summary

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms. Target: ChatGPT users Actor: Vulcan Cyber, Lasso Security Procedure: 1. The researchers use the public ChatGPT API throughout this exercise. 2. The researchers prompt ChatGPT to suggest software packages and identify suggestions that are hallucinations which don't exist in a public package repository. For example, when asking the model "how to upload a model to huggingface?" the response included guidance to install the `huggingface-cli` package with instructions to install it by `pip install huggingface-cli`. This package was a hallucination and does not exist on PyPI. The actual HuggingFace CLI tool is part of the `huggingface_hub` package. 3. An adversary could upload a malicious package under the hallucinated name to PyPI or other package registries. In practice, the researchers uploaded an empty package to PyPI to track downloads. 4. A user of ChatGPT or other LLM may ask similar questions which lead to the same hallucinated package name and cause them to download the malicious package. The researchers showed that multiple LLMs can produce the same hallucinations. They tracked over 30,000 downloads of the `huggingface-cli` package. 5. The user would ultimately load the malicious package, allowing for arbitrary code execution. 6. This could lead to a variety of harms to the end user or organization.

Deployers

• ChatGPT users

Tags

• MITRE ATLAS
• AML.CS0022