LLMSmith: RCE Vulnerabilities in LLM-Integrated Applications

Summary

Researchers identified 20 remote code execution (RCE) vulnerabilities across 11 different LLM frameworks. They discovered applications deployed on the public internet built using these LLM frameworks and demonstrated the RCE vulnerabilities could be exploited using prompt injection. The 11 LLM frameworks the researchers evaluated were: LangChain, LlamaIndex, Pandas-ai, Langflow, Pandas-llm, Auto-GPT, Griptape, Lagent, MetaGPT, vanna, and langroid. Target: LLM Integration Frameworks Actor: Researchers at University of Chinese Academy of Sciences, Shandong University, and University of New South Wales Procedure: 1. The researchers performed a static analysis on the APIs of target LLM frameworks to identify functions that execute code from either user input or the response from an LLM and are thus vulnerable to RCE. 2. The researchers performed targeting to identify applications that are likely built on with LLM Frameworks and may use the functions vulnerable to RCE. This was done by scanning source code repositories for app deployment URLs. 3. The researchers ran their static analysis to extract call chains from target application’s source code to identify those that utilize LLM framework functions vulnerable to RCE. 4. The researchers developed prompts to trigger tool invocations that lead to RCE. 5. The researchers targeted public-facing applications that expose an AI agent to user input as a means to execute their prompts. 6. The researchers directly prompted the AI agent with their malicious instructions. 7. For target applications where the AI agent refused the researcher’s request, they used lightweight jailbreaking strategies to bypass the LLM’s guardrails. 8. The researchers' prompts called the AI agent’s tools, targeting call chains that can lead to code execution.

Deployers

• LLM Integration Frameworks

Tags

• MITRE ATLAS
• AML.CS0052