History of transformative technologies: Unequal harms
CSCI 3921w Social, Legal, and Ethical Issues in Computing
Adriana Picoral
Q&A (10 minutes)
Q&A about ethics, computing, and logistics of the course.
No question is too silly, naive, controversial, basic, or advanced.
Three recurring patterns
When we look back at the history of transformative technologies, some patterns keep repeating:
- Automation bias: trusting software more than people, including ourselves
- Unequal harms: the costs fall mostly on marginalized groups
- Reactive regulation: regulation/laws arrive only after a disaster
We are looking at each pattern through historical cases and timelines. We covered Automation bias and now we will address Unequal harms
Pattern 2: Harms falling mostly on marginalized groups
Who bears the risk?
- Technologies are often tested on and deployed first on people with the least power to refuse (welfare recipients, immigrants, prisoners, poor neighborhoods)
- When the system fails, those groups have the fewest resources to fight back
- Their complaints are often less likely to be believed
Automated systems “manage the poor so that we do not have to” – Virginia Eubanks, Automating Inequality (2018)
Case: COMPAS risk scores (2016)
- COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) predicts whether a defendant will reoffend. Judges use the score in bail and sentencing decisions
- ProPublica (2016): Black defendants who did not reoffend were almost twice as likely as white defendants to be labeled high-risk (45% vs 23%)
- The company responded that the scores were equally accurate across groups
- 2016: Wisconsin Supreme Court (State v. Loomis) allows COMPAS in sentencing, with warnings
Group Discussion: Ethics frameworks
- Consequentialism: What are the consequences of recidivism/reoffence? What are the consequences of judges’ inability to evaluate risk assessment tools? Intersectionality: Are the consequences of COMPAS the same across gender, race, and etnic background?
- Political economy: “The methodology behind COMPAS is a trade secret.” Who bears the costs of this lack of transparency? Who profits from it?
- Contractualism: Could the use of opaque predictive systems be justified to everyone affected?
Case: Facial recognition
| 2018 |
Gender Shades (Buolamwini & Gebru): error rates up to 34.7% for darker-skinned women vs 0.8% for lighter-skinned men |
| 2019 |
NIST study finds many algorithms have much higher false positive rates for Asian and Black faces |
| 2020 |
Robert Williams wrongfully arrested in Detroit in front of his family, based on a facial recognition match |
| 2021 |
Kimberlee Williams wrongfully arrested based on a facial recognition match |
| 2023 |
Porcha Woodruff wrongfully arrested in Detroit |
| 2024 |
Williams settlement: Detroit Police Department is required to back up face recognition results with independent and reliable evidence |
Nearly all publicly known US wrongful arrests from facial recognition involve Black people.
Group Discussion: Unequal harms
- Why are newly developed automated systems often deployed first by police and the incarceration system? (and immigration)
- Who should be in the room when system like face recognition and COMPAS is designed?
- What type of error can the use of COMPAS cause (think back to automation bias errors: omission or commission)
- Why do authorities so often presume that computers work correctly (even when affected people tell them that is not the case)? What would the alternative look like?
References and Further Readings
Journal Writing
Reflect on what you liked and disliked about this lecture, what you learned, and what you wished you’d learned.
- What you learned from the lecture and discussion – this can be something that surprised you, something you had not considered before, a new way of thinking about something
- What you wish you’d learned from the discussion – this can be questions you still have and questions you would like to explore further
- Anything you liked of disliked