Algorithmic Bias

CSCI 3921w Social, Legal, and Ethical Issues in Computing
Adriana Picoral

Q&A (10 minutes)

Q&A about ethics, computing, and logistics of the course.

No question is too silly, naive, controversial, basic, or advanced.

Warm-up

Before we start, answer in your journal:

Think of an automated system that made a decision about you (a movie or tv show recommendation, an ad, a loan or job application, a content filter, a spam folder).

Did it get you right?
If it got you wrong, how would you have known? Could you have contested it?

From past to present

In the last module we saw unequal harms (COMPAS, facial recognition).

This module asks:

Where does bias come from in computing systems?
What does it mean for an algorithm to be fair? Can we measure it?
Why can’t we just have all kinds of fairness at once?

Where does bias come from?

Bias in computer systems

observation processes that generate data we use in computer systems are structurally biased

we need to be intentional and explicit about worldviews and value assumptions: the systems we design will always encode some belief about the world

From Friedler et al. (2021)

Group Discussion

Think back when you were applying to college. What materials did you have to submit? Were the evaluation criteria of these materials clear to you? Did the decisions you receive make sense to you?
What values and worldview do college admissions embody?
What are college admission decisions trying to predict?

Bias in decisions

This college admissions example demonstrates the basic pipeline in human decision-making systems, which is also mimicked in algorithmic systems

From Friedler et al. (2021)

Types of harm

Barocas et al. (2017)

allocative harm: when opportunities or resources are withheld from certain people or groups (e.g., whether someone is offered a loan or a job)
representational harm: when certain people or groups are stigmatized or stereotyped (e.g., LLMs that encode and replicate stereotypes)

College admissions decisions can cause allocative harm

Bias in computer systems

Friedman & Nissenbaum (1996) identified three kinds of bias:

Preexisting bias: comes from society, institutions, the people who build the system, and the data used in the system
Technical bias: comes from technical constraints or decisions (e.g., limited screen space, proxy variables, incomplete data)
Emergent bias: appears after deployment, when the context of use changes (e.g., new users, new purposes, feedback loops)

Bias across the ML life cycle

Preexisting bias

[The process of producing data] is long and complex, grounded in historical context and driven by human choices and norms

Technical bias

not all problems should be blamed on the data. The ML pipeline involves a series of choices and practices, from model definition to user interfaces used upon deployment.

From Suresh & Guttag (2021)

Bias across the ML life cycle

Suresh & Guttag (2021) map harms to stages of building a machine learning system:

Source	What goes wrong	Example
Historical	The world the data reflects is already unjust	Past hiring decisions favored men
Representation	Some groups are under-sampled	classifying images of objects from under-sampled countries
Measurement	The label or feature is a poor proxy for what we care about	Using arrests as a proxy for crime
Aggregation	One model for groups that differ	One medical model for all populations

Bias across the ML life cycle

Suresh & Guttag (2021) map harms to stages of building a machine learning system:

Source	What goes wrong	Example
Learning	Modeling choices amplify performance disparities	Pruning eliminates underrepresented attributes
Evaluation	Benchmarks do not represent users	Smiling detection on images of dark-skinned women
Deployment	System used differently than intended	A score made for support used for punishment

Group Discussion

Consider the following simplified scenario: a medical researcher wants to build a model to help detect whether someone is having a heart attack. She trains the model on medical records from a subset of prior patients at a hospital, along with labels indicating if and when they suffered a heart attack. She observes that the system has a higher false negative rate for women (it is more likely to miss cases of heart attacks in women), and hypothesizes that the model was not able to effectively learn the signs of heart attacks in women because of a lack of such examples. She seeks out additional data representing women who experienced heart attacks to augment the dataset, re-trains the model, and observes that the performance for female patients improves.

Meanwhile, a co-worker hiring new lab technicians tries to build an algorithm for predicting the suitability of a candidate from their resume along with human-assigned ratings. He notices that women are much less likely to be predicted as suitable candidates than men. Like his colleague, he tries to collect many more samples of women to add to the dataset, but is disappointed to see that the model’s behavior does not change. Why did this happen?

Suresh & Guttag (2021)

Proxies

Case: Amazon’s hiring tool (2014–2017)

Amazon built a machine learning tool to rate job applicants’ résumés
Trained on 10 years of résumés submitted to the company
The model learned to penalize résumés containing the word “women’s” (e.g., “women’s chess club captain”) and downgraded graduates of two all-women’s colleges
Engineers removed those terms, but could not guarantee the model would not find other proxies

“the tool disadvantaged candidates who went to certain women’s colleges presumably not attended by many existing Amazon engineers”

Case: Health care algorithm (2019)

Obermeyer et al. (2019), Science studied a widely used algorithm that decides which patients get extra care:

The algorithm predicted health care costs as a proxy for health care needs
Less money is spent on Black patients who have the same level of need (unequal access to care)
Result: Black patients assigned the same level of risk are sicker than White patients
This racial bias reduces the number of Black patients identified for extra care by more than half

Race was not an input. The bias came from using health care costs.

Proxies: “Fairness through unawareness” does not work

Removing a protected attribute (race, gender) from the data does not remove bias, because other features act as proxies:

Zip code → race (a legacy of redlining)
Name, college, hobbies, language choice → gender
Cost of care → race
Gaps in employment → disability, caregiving, gender

The model learns patterns in the data, and the data carries history.

Group Discussion: Sources of bias

Think back to your journal entry about an automated system that made a decision about you (a movie or tv show recommendation, an ad, a loan or job application, a content filter, a spam folder) that was not correct. Can you identify any bias in those decisions? What proxies do you think are used in college admissions decisions?
How can engineers build models that take into account preexisting bias (historical roots), technical bias (design and architecture limits), and emergent bias (feedback loops, changing environments, and user adaptation)?

In-class Activity: Sources of bias

Answer the canvas quiz on scenarios and their source of bias

Group Activity: Present Technology

Surveillance pricing: personal data is used to set individualized prices or adjust targeted discounts based on what is predicted a specific customer is willing to pay

Dynamic Pricing: Changes prices based on external market conditions, such as weather, time of day, or overall product supply and demand (e.g., surge pricing for rideshares or airline seats).

You should read more about surveillance pricing to answer these questions.

Some regulation has been put into place (Maryland’s Protection from Predatory Pricing Act, Colorado’s HB26-1210, MA Bill S.47, Seattle’s ordinance)

Does surveillance pricing take into account preexisting bias (historical roots), technical bias (design and architecture limits), and emergent bias (feedback loops, changing environments, and user adaptation)? For what purpose (reduce bias?)?
Can you identify any issues of automation bias, and unequal harm?
What should be done to reduce the impact of bias on different groups?

References and Further Readings

Barocas, S., Crawford, K., Shapiro, A., & Wallach, H. (2017, October). The problem with bias: From allocative to representational harms in machine learning. In SIGCIS conference paper (p. 53).
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2021). The (im) possibility of fairness: Different value systems require different mechanisms for fair decision making. Communications of the ACM, 64(4), 136-143.
Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14(3), 330–347.
Suresh, H., & Guttag, J. (2021). A framework for understanding sources of harm throughout the machine learning life cycle. EAAMO ’21.
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.

References and Further Readings

Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters.
Nicoletti, L., & Bass, D. (2023). Humans are biased. Generative AI is even worse. Bloomberg.
Noble, S. U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.

Journal Writing

Reflect on what you liked and disliked about this lecture, what you learned, and what you wished you’d learned.

What you learned from the lecture and discussion – this can be something that surprised you, something you had not considered before, a new way of thinking about something
What you wish you’d learned from the discussion – this can be questions you still have and questions you would like to explore further
Anything you liked or disliked