What is automation bias?

Two kinds of errors (Skitka, Mosier, & Burdick, 1999):

• Omission: failing to act because the system didn’t flag a problem (missed events when not explicitly prompted about them by the aid)
• Commission: following the system’s recommendation even when other evidence contradicts it (did what an automated aid recommended, even when it contradicted their training and other 100% valid and available indicators)

Why does it happen?

• Software seems objective and neutral
• Checking the system takes time and effort
• Trusting the system becomes the default
• When the system says the person is wrong, the person has to prove the system is wrong

Case: Horizon IT scandal

• Accounting software showed cash shortfalls at British post offices.
• The British Post Office wrongfully pursued thousands of innocent subpostmasters for financial shortfalls.
• People went to prison and lost homes, and the scandal has been linked to suicides.

Case: Horizon IT scandal

Some of the bugs in Fujitsu’s Horizon system

• “Dalmellington bug”: the system would enter repeated withdrawals in the ledger every time the user pressed “enter” at a frozen interface screen
• “Callendar Square bug”: the system would create duplicate database entries in the ledger

Subpostmasters used the available helpline to report balancing errors within weeks of the Horizon system being installed.

Group Discussion: Ethics frameworks

• Deontology: What duties did engineers and managers have, regardless of the outcome? Which ACM values were violated?
• Consequentialism: What are the usual benefits of using accounting software? Did the benefits outweigh the harm in this case study?
• Political economy: Who bore the costs of the “bugs, errors and defects” in this case study? Who profited from it?
• Contractualism: Could the Post Office’s decision to keep using Horizon despite reported problems be justified to everyone affected?

Group Discussion: Automation bias

• One of Fujitsu’s employee testified in court that “you can never say there are no more bugs in the system.” As a software developer, can you sympathize with this statement? Whose job is to guarantee software reliability?
• Was the continued use of the accounting system despite reports of balancing errors an failure of omission or commission?
• Why do authorities so often presume that computers work correctly (over reports from their own employees)? What would the alternative look like?

References and Further Readings

• Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006.
• Leveson, N. G., & Turner, C. S. (1993). An investigation of the Therac-25 accidents. IEEE Computer, 26(7), 18–41.
• Post Office Horizon IT Inquiry
• FTC Tech Guidance

Journal Writing

Reflect on what you liked and disliked about this lecture, what you learned, and what you wished you’d learned.

1. What you learned from the lecture and discussion – this can be something that surprised you, something you had not considered before, a new way of thinking about something
2. What you wish you’d learned from the discussion – this can be questions you still have and questions you would like to explore further
3. Anything you liked of disliked

Some concepts

• Incrementalism: progress through small, continuous steps rather than massive, sweeping overhauls
• Precautionary principle: When an activity raises threats of harm to human health or the environment, precautionary measures should be taken even if some cause-and-effect relationships are not fully established scientifically