Big Data and Algorithms

Algorithms Are Making Decisions About You

Algorithms trained on large datasets now determine who gets a loan, who sees a job advertisement, who gets released on bail, how much you pay for insurance, and what news you see. These systems process vastly more data than any human could, but 'more data' does not automatically mean 'better decisions.'

The COMPAS algorithm, used in US criminal courts to predict recidivism risk, was shown by ProPublica to be no more accurate than random internet volunteers — and to produce racially biased results. Black defendants were nearly twice as likely as white defendants to be falsely flagged as high risk. The algorithm was trained on historical criminal justice data that reflected decades of racial disparities in policing and sentencing.

Garbage In, Bias Out

Algorithms learn patterns from historical data. If that data reflects historical discrimination, the algorithm will reproduce and potentially amplify that discrimination. Amazon built a hiring algorithm trained on a decade of resumes — but because the tech industry had historically hired predominantly men, the algorithm learned to penalize resumes containing words like 'women's' (as in 'women's chess club') or graduates of all-women's colleges. Amazon scrapped the system.

The opacity of many algorithms makes this problem harder to detect. 'Black box' models — including many deep learning systems — make decisions through processes even their creators cannot fully explain. When you are denied a loan or shown a biased news feed, you may never know why.

Algorithmic accountability requires transparency (knowing how decisions are made), auditability (the ability to test for bias), and contestability (the ability to challenge algorithmic decisions that affect your life).