BiasHigh
NarxCare’s Risk Score Model Allegedly Lacked Validation and Trained on Data with High Risk of Bias
- Occurred
- Jul 1, 2020
- Published
- Aug 30, 2023
- Source
- AI Incident Database
Summary
NarxCare's overdose risk algorithm, lacking peer-reviewed validation, uses sensitive data like doctor visits, prescriptions, and possibly genetic information, leading to significant biases against women and Black patients. Factors like sexual abuse and criminal records exacerbate stigmas and disparities, often resulting in unjust denial of necessary pain medication. The newly approved AvertD genetic test shares similar issues, further complicating and potentially harming medical treatment decisions.
Developers
- Appriss
Deployers
- NarxCare
- AvertD
- Appriss
Harmed parties
- American physicians
- American pharmacists
- American patients of minority groups
- American patients
Classification
Category confidence: 90%
Severity confidence: 85%
Rationale: Algorithm lacks validation and exhibits significant demographic bias affecting medical treatment decisions.
Classifier: Qwen/Qwen3.5-35B-A3B-FP8