Fairness vs. justice

A system can be fair by a statistical definition and still unjust.

Theories of justice

• Distributive justice: how are benefits and burdens distributed? Rawls (1971) asks what rules we would choose behind a “veil of ignorance”, not knowing our place in society. His difference principle: inequalities are acceptable only if they benefit the least advantaged.
• Procedural justice: were decisions made through a fair process? Could the affected person understand, question, and contest the decision?
• Justice as recognition and non-domination: Young (1990) argues that injustice is not only about distribution but about oppression – structures that systematically disadvantage groups. Her “five faces of oppression”: exploitation, marginalization, powerlessness, cultural imperialism, and violence.

Fairness metrics focus mostly on a narrow kind of distributive justice (how errors are distributed).

Original Position and the Veil of Ignorance thought experiment

Rawls (1971) imagines people choosing the basic rules of society from an original position:

• Behind a veil of ignorance, you do not know your class, race, gender, abilities, or which generation you belong to
• You know general facts about society, but not where you will end up in it
• Because no one can tailor the rules to benefit themselves, the rules chosen are fair

Rawls argues people would choose:

1. Equal basic liberties for everyone
2. The difference principle: inequalities are allowed only if they benefit the least advantaged (a maximin rule: maximize the minimum)

Would you really choose that way? Let’s test it.

The scenario

The city of Northfield is buying an automated tenant screening system. Every rental application in the city will be scored by it. You will vote on which design to buy.

There are five positions in Northfield:

The designs

Each number is the chance (%) that a person in that position gets an affordable lease within a year.

• A: Credit score, 3× income rule, all eviction filings are counted
• B: Same model as A + fairness constraint (equal approval rates across
• C: Rent payment history instead of credit, counts voucher income, ignores dismissed filings

Round 1: You know your place (5 min)

1. Draw a position: roll a die (re-roll a 6) or use a random number generator from 1 to 5
2. Write your position in your journal
3. Individually vote for design A, B, or C

We will tally the votes on the board by position.

Round 2: Behind the veil (10 min)

Forget your Round 1 position. You will get a new position, but only after your group decides.

1. In groups, discuss the four designs and choose one by consensus
2. Write a one-sentence justification: “We chose __ because __.”
3. Each person draws a new position and records their outcome
4. Was anyone in your group unhappy with the result? Would they still defend the choice?

Round 3: Lifting the veil (5 min)

Now you learn how common each position is in Northfield:

You also learn that, because of the history of redlining and housing discrimination, positions 4 and 5 are made up disproportionately of Black and Latino families.

Does your group change its vote? Why or why not?

Debrief: What the designs do

• C is unequal, yet Rawls prefers it to “equality, same chances for everyone”: the inequality makes the worst-off better off
• Harsanyi (1975) argued that rational people behind the veil would maximize expected value (A), not the minimum
• In lab experiments, most groups behind a veil chose to maximize the average subject to a floor (like B), not strict maximin (Frohlich, Oppenheimer & Eavey, 1987)

Debrief: Group Discussion

• Compare the Round 1 tally with the Round 2 choices. Did the veil change votes?
• Which design would a company selling the software build by default? Which would a landlord buy?
• Design B satisfies a fairness metric. Is it the just choice? What does the fairness constraint leave unchanged?
• Did the Round 3 information about history change your reasoning, even though the numbers did not change?

Where fairness fails

Hoffmann (2019) argues that fairness work borrows the limits of anti-discrimination law:

• It looks for “bad actors” and “bad inputs” instead of structures
• It treats protected attributes (race, gender, disability) one at a time, missing intersectional harms
• It focuses on disadvantage but ignores who is advantaged by the same system
• It asks how to distribute outcomes fairly, not whether the goods themselves (e.g., policing, surveillance) are desirable

The five abstraction traps

Selbst et al. (2019) describe how computer scientists’ habit of abstraction leads fairness work astray:

History in the data

Data does not come from nowhere. It records how institutions have treated people in the past.

• Redlining: in the 1930s, the federal Home Owners’ Loan Corporation (HOLC) graded neighborhoods for mortgage risk, marking many Black and immigrant neighborhoods as “hazardous” in red (Mapping Inequality)
• Redlining was outlawed by the Fair Housing Act (1968), but its effects persist in home values, wealth, school funding, and health
• A model trained today on property values, ZIP codes, or credit history learns these patterns as “risk”

Recall proxies: removing race from the data does not remove history from the data.

Feedback loops

Predictive policing tools predict where crime will be recorded, not where crime happens.

• Lum & Isaac (2016) simulated a predictive policing algorithm on Oakland drug arrest data. Public health surveys estimated drug use was spread roughly evenly across the city, but the algorithm sent police mostly to Black and low-income neighborhoods
• More police → more recorded crime → the model sends more police: a runaway feedback loop (Ensign et al., 2018)

The model can be accurate on its data and still unjust: its “ground truth” is itself a product of past policing.

Practice example: Feedback loop

Two neighborhoods, A and B, have the same true rate of incidents. Historical records show more incidents in A (because it was patrolled more).

Rule: each week, send the patrol to the neighborhood with more recorded incidents. Only incidents where the patrol is get recorded (+10).

• Fill in weeks 3 and 4. What does the data “show” after 10 weeks?
• Would an audit of this model’s accuracy detect a problem?
• What would have to change to break the loop?

Case: Allegheny Family Screening Tool

• Since 2016, Allegheny County (Pennsylvania) has used a model to score calls to the child welfare hotline for the risk of future removal of a child from the home
• The model draws on public records: public benefits, public health care, county mental health and addiction services, jail, and juvenile probation
• Families who use private services (private doctors, private rehab) are invisible to the model. Eubanks (2018) calls this poverty profiling: confusing parenting while poor with poor parenting
• An ACLU analysis (2023) examined design choices in the tool, including the use of disability-related data and how scores were grouped and displayed to screeners

Could this tool be made fair? Would that make it just?

Case: Michigan’s MiDAS (2013–2015)

• Michigan replaced staff with an automated system (MiDAS) to detect unemployment insurance fraud
• The system flagged people for fraud without human review, often based on missing or mismatched records, and imposed penalties of up to 4x the benefits received
• A state review found an error rate of about 93% in the fraud determinations it examined
• People had wages garnished and tax refunds seized; some filed for bankruptcy
• Michigan agreed to a $20 million settlement with affected claimants

Who was most affected? Who had the resources to contest the decisions?

Case: Dutch childcare benefits scandal

• The Dutch Tax Administration used a risk-classification model to flag childcare benefit applications for fraud
• Nationality (“Dutch / non-Dutch”) was used as a risk factor (Amnesty International, 2021)
• Tens of thousands of families were wrongly accused and ordered to pay back benefits, often tens of thousands of euros; many fell into debt, lost homes, and some lost custody of their children
• January 2021: the Dutch government resigned over the scandal
• The Dutch Data Protection Authority fined the Tax Administration €2.75 million for discriminatory and unlawful data processing

Group Activity: Fair but unjust?

In groups, choose one system: (a) predictive policing, (b) Allegheny Family Screening Tool, (c) automated tenant screening, (d) unemployment fraud detection

1. Suppose an audit finds the system satisfies equalized odds across racial groups. List at least two harms it could still cause.
2. What historical or structural causes shape the data it uses?
3. Which abstraction trap is the system most at risk of?
4. Who benefits from the system existing? Who bears the burdens?
5. Fix, constrain, or don’t deploy? Defend your answer.

References and Further Readings

• Amnesty International (2021). Xenophobic machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal.
• Benjamin, R. (2019). Race After Technology: Abolitionist Tools for the New Jim Code. Polity.
• Charette, R. N. (2018). Michigan’s MiDAS unemployment system: Algorithm alchemy created lead, not gold. IEEE Spectrum.
• Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2018). Runaway feedback loops in predictive policing. Proceedings of FAT*, PMLR 81, 160–171.
• Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press.
• Frohlich, N., Oppenheimer, J. A., & Eavey, C. L. (1987). Laboratory results on Rawls’s distributive justice. British Journal of Political Science, 17(1), 1–21.
• Harsanyi, J. C. (1975). Can the maximin principle serve as a basis for morality? A critique of John Rawls’s theory. American Political Science Review, 69(2), 594–606.
• Hoffmann, A. L. (2019). Where fairness fails: Data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society, 22(7), 900–915.

References and Further Readings

• Lum, K., & Isaac, W. (2016). To predict and serve? Significance, 13(5), 14–19.
• Nelson, R. K., Winling, L., et al. Mapping Inequality: Redlining in New Deal America. University of Richmond.
• Rawls, J. (1971). A Theory of Justice. Harvard University Press.
• Selbst, A. D., boyd, d., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. Proceedings of FAT* ’19, 59–68.
• Young, I. M. (1990). Justice and the Politics of Difference. Princeton University Press.