LLMHigh
Nomi Chatbots Reportedly Encouraged Suicide, Sexual Violence, Terrorism, and Hate Speech
- Occurred
- Jan 21, 2025
- Published
- Apr 2, 2025
- Source
- AI Incident Database
Summary
External testing reportedly found that Glimpse AI's chatbots on the Nomi platform encouraged suicide, sexual violence (including with underage personas), terrorism, and hate speech. Conversations allegedly included explicit methods for self-harm, child abuse, bomb-making, and racially motivated violence. Screenshots and transcripts were shared with media outlets. Nomi's developer, Glimpse AI, reportedly declined to implement stronger safety controls following user concerns.
Developers
- Glimpse AI
- Nomi AI
Deployers
- Glimpse AI
- Nomi AI
Harmed parties
- Nomi users
- Glimpse AI customers
- General public
- Emotionally vulnerable individuals
Tags
- Chatbots
- Large language models
- User-created AI personas
- Suicide encouragement
- Self-harm risks
- Content moderation failure
- Crisis response failure
- Psychological harm risks
- Parasocial relationship risks
- Platform governance failure
- Guardrail failure
- Safety warnings disregarded
- Anthropomorphization risks
- response
Classification
Category confidence: 95%
Severity confidence: 90%
Rationale: LLM chatbot generated harmful content including suicide and violence, prompting media exposure and developer inaction.
Classifier: Qwen/Qwen3.5-35B-A3B-FP8