LLMHigh
At Least 10,000 AI Chatbots, Including Jailbroken Models, Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors
- Occurred
- Mar 5, 2025
- Published
- Mar 5, 2025
- Source
- AI Incident Database
Summary
At least 10,000 AI chatbots have allegedly been created to promote harmful behaviors, including eating disorders, self-harm, and the sexualization of minors. These chatbots, some jailbroken or custom-built, leverage APIs from OpenAI, Anthropic, and Google and are hosted on platforms like Character.AI, Spicy Chat, Chub AI, CrushOn.AI, and JanitorAI.
Developers
- OpenAI
- Anthropic
Deployers
- Character.ai
- Spicy Chat
- Chub AI
- CrushOn.AI
- JanitorAI
- Unidentified online communities using chatbots
Harmed parties
- Vulnerable chatbot users
- Minors using chatbots
- Individuals with eating disorders
- Individuals struggling with self-harm
- Adolescents
Tags
- Chatbots
- Abuse
- Eating disorders
- Self-harm
- CSAM
- Exploitation
- Content moderation failure
- Jailbreaking
- Mental health risk
- Large language model
- Teenagers
- Loneliness
- Vulnerable populations
- Disinformation
Classification
Category confidence: 95%
Severity confidence: 90%
Rationale: Large-scale deployment of jailbroken LLMs promoting severe harms like self-harm and child sexualization constitutes a major safety incident.
Classifier: Qwen/Qwen3.5-35B-A3B-FP8