An algorithm is a set of step-by-step instructions for solving a problem. Many decisions that shape people's lives, from loan approvals to job screening to which posts appear in a feed, now involve algorithms that score or rank people. Supporters point out that human decision-makers are often inconsistent and biased too, and that a well-designed system can be checked in ways a human hunch cannot. Critics warn that algorithms can repeat and even scale up old unfairness while looking objective.

How bias gets in

• Historical data. Many systems learn from past decisions. If past decisions were biased, a system trained on them can learn the bias. In a widely reported case, a large company dropped an experimental hiring tool after finding it marked down résumés that mentioned women's organizations, because it had learned from a decade of mostly male hires.
• Unrepresentative data. A 2018 study by Joy Buolamwini and Timnit Gebru found that several commercial face-analysis systems made far more errors on darker-skinned women than on lighter-skinned men, partly because of the images they were built with. A 2019 US government study found demographic differences in many face recognition algorithms, though the most accurate ones showed much smaller gaps.
• Proxies. Even if a system never sees race or sex, other data can stand in for them. A proxy variable is a feature, such as a postal code, that is closely linked to a protected trait.
• Feedback loops. If police patrol where a system predicts crime, they record more crime there, which feeds back into the system and can make the prediction confirm itself.

What does fair mean?

Fairness sounds simple until you try to measure it. Researchers have proposed many definitions, including:

• Demographic parity: each group receives positive decisions, such as loans, at the same rate.
• Equal error rates: the system makes mistakes, such as wrongly labeling someone high risk, equally often for each group.
• Calibration: a given score means the same thing whatever a person's group; of all people scored 7 out of 10, the same share go on to reoffend, or to repay, in each group.

In 2016 the news organization ProPublica analyzed a risk score used in US courts and reported that Black defendants who did not reoffend were nearly twice as likely as white defendants to be labeled higher risk. The tool's maker replied that its scores were equally calibrated for both groups. Both claims could be true at once. Researchers soon proved that when groups have different underlying rates of an outcome, no imperfect system can satisfy calibration and equal error rates at the same time. Choosing a fairness definition is therefore an ethical choice, not only a technical one.

Transparency and accountability

Some systems are black boxes: even their designers cannot easily say why a particular output was produced. People affected by decisions often want an explanation and a way to challenge errors. Proposed safeguards include independent audits, publishing how systems are tested, giving people notice and a right of appeal, and keeping a human in the loop. Some laws already point this way: the GDPR, for example, gives people in the EU a right, with some exceptions, not to be subject to important decisions made solely by automated means, and to ask for a human to review them. That last safeguard has limits: automation bias is the tendency of people to trust a computer's output too readily, even when it is wrong.

Worked example: one lending model, three fairness tests

Imagine a credit union that uses a model to approve small loans. It reviews last year's results for two neighborhoods, A and B, each with 1,000 applicants. In neighborhood A, 800 applicants would in fact repay; in B, 600 would, partly because B has had fewer well-paid jobs for decades. The model approved 760 applicants in A and 540 in B.

• Demographic parity fails: A's approval rate is 76 percent, B's is 54 percent.
• Calibration might pass. Suppose about 95 percent of approved applicants repay in both neighborhoods. Then an approval means the same thing in each group.
• Equal error rates still differ. With those numbers, about 722 of A's 800 good borrowers were approved, so about 10 percent were wrongly refused. In B, about 513 of 600 were approved, so about 15 percent were wrongly refused. B's residents bear more of the model's mistakes.

The credit union could approve more B applicants to move toward demographic parity, but then the share of approved B borrowers who repay would probably fall, breaking calibration. Because the underlying repayment rates differ, it cannot satisfy every test at once. It must decide which kind of fairness matters most here, and why.

Other questions follow. Do the different repayment rates reflect past discrimination that a fair lender should help repair? Is a postal code acting as a proxy for race? Can applicants learn why they were refused and correct errors in their records? Would loan officers working without the model do better, or worse? Numbers clarify the trade-offs; they do not settle them.