An AI system learns from data, and data is a record of the past: who was hired, who was treated, who was arrested, who was photographed. When that past was unequal, a system that learns it faithfully can carry the inequality forward, now with the authority of a machine and the speed of software. This level looks at how that happens and at what defenders of these tools say in reply.

Three ways bias gets in

Historical bias comes from data that accurately records an unjust past. Representation bias comes from data that leaves some people out or includes too few of them. Measurement bias comes from choosing the wrong thing to measure. The last is the subtlest, and it usually involves a proxy: a measurable stand-in for the thing you actually care about. Nobody can measure creditworthiness or need for care directly, so designers use something nearby, such as past payments or past spending. If the stand-in is skewed, so is everything built on it.

A health algorithm and its proxy

In 2019 the physician and researcher Ziad Obermeyer and colleagues published a study in the journal Science of an algorithm that US hospitals and insurers used to pick patients for extra care programs. It predicted how much each patient's care would cost, on the reasoning that sicker people cost more. But less money had historically been spent on Black patients with the same health problems, because of unequal access and other barriers. As a result, the algorithm used past spending as a stand-in for need, so Black patients were scored healthier than equally sick white patients. When the researchers adjusted it to predict health instead of cost, the share of Black patients flagged for extra help rose from about 18 percent to about 47 percent. Nobody had programmed in race; the bias came through the proxy.

Faces and résumés

In the 2018 study Gender Shades, Joy Buolamwini and Timnit Gebru tested commercial systems that guessed a person's gender from a photo. Error rates were highest for darker-skinned women, reaching about 35 percent for one system, compared with under 1 percent for lighter-skinned men. The training photos had underrepresented some groups.

Around the same time, Reuters reported that Amazon had built an experimental tool to rate job applicants, trained on résumés submitted over ten years, most of them from men. The company dropped it after finding that it had learned to downgrade résumés that mentioned women, such as a line about captaining a women's chess club. Amazon said recruiters never relied on the tool alone to judge candidates.

When the state automates suspicion

The highest stakes come when governments automate decisions about ordinary people. In the Netherlands the tax authority used a risk-scoring system, among other methods, to hunt for fraud in childcare benefits. Tens of thousands of parents were wrongly treated as fraudsters and ordered to repay large sums, pushing many into debt. Investigators found that dual nationality had been used as a risk factor, and many of those harmed had immigrant backgrounds. The scandal led the Dutch government to resign in January 2021.

Australia's Robodebt scheme, which ran from 2016 to 2019, used no machine learning at all, only plain automation. It averaged a person's yearly income across fortnights to estimate what they had earned, then raised welfare debts when that estimate did not match what they had reported. People with irregular work were hit hardest, and the burden fell on them to prove the machine wrong. A court later found the method unlawful, the government agreed to a settlement worth about 1.8 billion Australian dollars, and a royal commission condemned the scheme in 2023. Both cases show how scale, speed and a presumption that the computer is right can turn a design flaw into mass harm.

The case for algorithms

Defenders make a serious point. Human decisions are biased too, often worse, and that bias hides inside individual minds. The economist Sendhil Mullainathan, a coauthor of the health study, has argued that unlike human judgment, an algorithm can be tested and audited for bias, and then corrected, as that one was. A prejudiced hiring manager is hard to retrain; a biased model can be measured. On this view the goal is not to reject algorithms but to demand audits, good data and a way to appeal. Critics answer that audits rarely happen unless someone forces them, and that automation can make bias harder to see and to contest. Both points can be true at once.

Worked example: the scholarship screener

A foundation plans to use a model to rank scholarship applicants, trained on which past applicants went on to graduate. Before agreeing, a board member asks what the model actually predicts. It predicts graduation, which depends partly on money and support after the award, not only on talent: a proxy. She asks which schools past winners came from. Mostly a few wealthy suburban ones, a sign of representation bias. She asks what happens to a student the model ranks low. If no one reviews the score, a flaw becomes a rule. The board decides to use the model only to make sure no application is overlooked, with people making every decision and a yearly check of results by school and family income.