Who does the audit?

Costanza-Chock, Raji & Buolamwini (2022)

Who does the audit?

• The results of first-party audits are not typically disclosed to the public and there is little transparency about whether their recommendations are implemented.
• While second-party auditors are an important part of the ecosystem, there is no standardized audit methods and disclosure policies
• Third-party audits are conducted by fully independent organizations or individuals with no obligation or contractual relationship to the audit target
• Third-party audit have been particularly influential in creating public awareness of algorithmic harms

Costanza-Chock, Raji & Buolamwini (2022)

Audit methods

• Code audit: direct access to the codebase of the underlying system/documentation
• User survey: survey and/or user interviews that gather descriptive data of user experience
• Scraping audit: scraped data
• API audit: API acquired data
• Sock-puppet audit: data is generated through programmed (fake) users
• Crowd-sourced audit (mystery-shopper audit): data is collected from real users

Ada Lovelace Institute – Technical methods for regulatory inspection of algorithmic systems

Audit methods

• Audit study: systematically probing and testing of black-box systems through repeated interaction to understand their behavior and investigate deficiency (discrimination, bias, or something else) without direct access to the internals of the system
• Purposefully intended to be external evaluations
• Audit studies are a powerful tool for establishing evidence of discrimination, especially disparate impact

Metaxa et al. 2021

Case: Gender Shades (2018)

Buolamwini & Gebru (2018) audited commercial gender classification systems (IBM, Microsoft, Face++):

• Built a new benchmark balanced by gender and skin type (existing benchmarks were mostly lighter-skinned men)
• Error rates of up to 34.7% for darker-skinned women vs. at most 0.8% for lighter-skinned men
• Intersectional analysis: looking only at gender or only at skin type hides the worst disparities

Which source of bias does this reveal?

Case: Actionable Auditing (2019)

Raji & Buolamwini (2019) asked: did the audit change anything?

• The audited companies were notified and released new versions of their products
• All three audited companies reduced their error disparities
• Companies that were not audited (e.g., Amazon, Kairos) had large disparities in the new tests

Public, named audits can create accountability. But companies also pushed back on the findings, and audits put researchers at legal and reputational risk.

Case: Facebook ad delivery (2019)

Ali et al. (2019) ran ads with identical targeting (same audience, same budget):

• Ads for jobs in the lumber industry were delivered to audiences that were mostly white men
• Ads for janitor jobs were delivered mostly to Black women
• Housing ads were skewed by race depending on the image in the ad
• The skew came from Facebook’s optimization for engagement, not from the advertiser’s choices

In 2022, the U.S. Department of Justice settled with Meta under the Fair Housing Act. Meta built a “Variance Reduction System” for housing ads.

Legal risks of auditing

• Websites’ terms of service often do not allow scraping audits and sock puppet audits
• Sandvig v. Barr (2020): a federal court ruled that violating terms of service for research does not, by itself, violate the CFAA
• Van Buren v. United States (2021): the Supreme Court narrowed the CFAA’s scope
• Platforms have still shut down researchers’ accounts and tools (e.g., Meta and NYU’s Ad Observatory, 2021)

Who benefits when audits are hard to do?

NYC Local Law 144 (2023)

New York City requires employers that use automated employment decision tools (AEDTs) to:

• Have an independent bias audit done each year
• Publish a summary of the audit results
• Notify candidates that an AEDT is being used

The audit must report selection rates and impact ratios by sex, race/ethnicity, and intersectional categories.

\[\text{impact ratio} = \frac{\text{selection rate of a category}}{\text{selection rate of the most selected category}}\]

Recall the EEOC’s four-fifths rule: an impact ratio below 0.8 is evidence of adverse impact.

Practice example: Impact ratios

A résumé screening tool advanced the following applicants to an interview:

Calculate the selection rate and impact ratio for each category. Does any category fall below the four-fifths threshold?

Practice example: Impact ratios

• Women fall below the four-fifths threshold
• The nonbinary category has very few applicants: is a ratio based on 20 people reliable? (NYC rules allow excluding categories under 2% of the data)
• Notice what this audit does not measure: whether the people advanced were qualified (true/false positive rates)

Limits of Local Law 144

A study of the law’s first year (Wright et al., 2024) found:

• Very few employers published audits (18 of 391 employers studied)
• Employers decide for themselves whether their tool counts as an AEDT
• The law only requires reporting: there is no penalty for a low impact ratio
• Audits measure selection rates only, not accuracy, validity, or disability discrimination

Is an audit that no one is required to act on still useful?

Internal audits

Raji et al. (2020) proposed an end-to-end internal audit framework (SMACTR) that runs through the development life cycle:

• Scoping: what are the intended uses and the possible harms?
• Mapping: who are the stakeholders and who is responsible?
• Artifact collection: documentation such as model cards and datasheets for datasets
• Testing: e.g., disaggregated evaluation and adversarial testing
• Reflection: weigh results against the company’s ethical principles, and decide whether to go ahead

Open-source tools can help: Fairlearn, IBM AI Fairness 360, Aequitas

Who audits the auditors?

Costanza-Chock, Raji & Buolamwini (2022) surveyed people working in the audit field and found:

• There are no shared standards for what an audit must include
• Second-party auditors are paid by the company being audited (conflict of interest)
• Results are often kept private
• Affected communities are rarely involved

Audit-washing: using a (weak) audit to make a system look fair without changing it

Group Activity: Present Technology

AI hiring tools: many companies use automated tools to screen résumés, rank candidates, analyze video interviews, or score games and assessments (e.g., HireVue, Workday, LinkedIn Recruiter).

In 2023, a lawsuit Mobley v. Workday claimed Workday’s screening tools discriminated by race, age, and disability. In 2025 a court allowed the age discrimination claim to move forward as a collective action.

Your group is hired to design an audit of an AI hiring tool:

• What type of audit (first-, second-, or third-party) and which method would you use? What access do you need?
• Which groups and which intersectional categories would you test? Which fairness metrics would you report?
• What would count as “failing” the audit, and what should happen if the tool fails?

References and Further Readings

• Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019). Discrimination through optimization: How Facebook’s ad delivery can lead to biased outcomes. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–30.
• Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. FAT* ’18.
• Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022). Who audits the auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. FAccT ’22.
• Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends in Human–Computer Interaction, 14(4), 272–344.
• Raji, I. D., & Buolamwini, J. (2019). Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. AIES ’19.

References and Further Readings

• Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., … & Barnes, P. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. FAT* ’20.
• Wright, L., Muenster, R. M., Vecchione, B., Qu, T., Cai, P., Smith, A., … & Metcalf, J. (2024). Null compliance: NYC Local Law 144 and the challenges of algorithm accountability. FAccT ’24.
• NYC Department of Consumer and Worker Protection. Automated Employment Decision Tools (AEDT).