Skip to content
BiasHigh

NarxCare’s Risk Score Model Allegedly Lacked Validation and Trained on Data with High Risk of Bias

Occurred
Jul 1, 2020
Published
Aug 30, 2023
Source
AI Incident Database

Summary

NarxCare's overdose risk algorithm, lacking peer-reviewed validation, uses sensitive data like doctor visits, prescriptions, and possibly genetic information, leading to significant biases against women and Black patients. Factors like sexual abuse and criminal records exacerbate stigmas and disparities, often resulting in unjust denial of necessary pain medication. The newly approved AvertD genetic test shares similar issues, further complicating and potentially harming medical treatment decisions.

Developers

  • Appriss

Deployers

  • NarxCare
  • AvertD
  • Appriss

Harmed parties

  • American physicians
  • American pharmacists
  • American patients of minority groups
  • American patients

Classification

Category confidence: 90%
Severity confidence: 85%

Rationale: Algorithm lacks validation and exhibits significant demographic bias affecting medical treatment decisions.

Classifier: Qwen/Qwen3.5-35B-A3B-FP8

Visit source