Skip to content
BiasHigh

Google’s Automated Child Abuse Detection Wrongfully Flagged a Parent’s Naked Photo of His Child

Occurred
Aug 21, 2022
Published
Aug 22, 2022
Source
AI Incident Database

Summary

Google’s automated detection of abusive images of children incorrectly flagged a parent’s photo intended for a healthcare provider, resulting in a false police report of child abuse, and loss of access to his online accounts and information.

Developers

  • Google

Deployers

  • Google

Harmed parties

  • a software engineer named Mark
  • parents using telemedicine services

Classification

Category confidence: 85%
Severity confidence: 90%

Rationale: Automated system error caused false police report and account loss, indicating harmful discriminatory impact.

Classifier: Qwen/Qwen3.5-35B-A3B-FP8

Visit source