Fair is Foul: the Hidden Tradeoffs of Algorithmic Fairness

In her excellent and prescient book Weapons of Math Destruction, mathematician Cathy O’Neil debunks the myth that algorithms discover and protect fairness. It’s easy to believe otherwise because they have so much going for them. Unlike human judges, they don’t suffer from emotional or cognitive inconsistency. Unlike doctors and nurses, they don’t get tired or hangry. They cannot be persuaded, and perhaps most importantly, unlike human decision-makers, there is no limitation to the amount of data they can consume to arrive at the ostensibly most informed conclusion. 

So it’s no wonder these systems increasingly pervade all aspects of our lives: the healthcare we receive, how we combat crime, and our access to housing and education, often without our awareness or consent.

There are several high-profile and well-documented instances of when these systems behave unfairly, but for every news item about a denied healthcare claim that killed someone, or a miscarriage of justice that falsely convicted someone, there are countless victims of systemic inequities produced despite the best of intentions. Because contrary to rumor, math, when applied, is not amoral, and numbers are not black and white.

Algorithms are a series of choices. By their nature, their outputs reflect deliberate, if not always explicit, compromises. And any of them that generate complex decisions—such as loan eligibility, medical diagnostics, or criminal risk scores—are comprised of myriad micro-decisions about what to measure, what to ignore, how to weigh competing signals, and what kinds of mistakes and error rates are considered acceptable. The concept of fairness should be considered at the beginning, as it is encoded in these choices and enforced through their real-life consequences.

One of the systems we reviewed in Stanford’s Ethics and Technology class is called COMPAS, which stands for Correctional Offender Management Profiling for Alternative Sanctions. Folks who follow this area know this case study well as it’s a canonical intent vs impact example when discussing algorithmic bias. It is a risk assessment model used in criminal justice to estimate the likelihood of defendant recidivism and their propensity to commit future violent acts (if this evokes Minority Report, you are not far off). COMPAS does not render final decisions; it produces a risk score for judges and parole boards to factor into their consideration.

COMPAS incorporates a lot of data about the defendant, including answers to a lengthy questionnaire (100+ items), criminal history, demographic and social indicators, substance use history and peer associations. It is important to note that race is intentionally not an explicit input, but structural proxies abound, such as prior arrest history, housing stability, employment status, and neighborhood-level enforcement patterns.

Like so many decision-support tools, COMPAS is a black box. Defendants cannot see how their risk profile is calculated, the resulting score cannot be meaningfully contested, and officials who decide to use it must accept the results without inquiry or discussion.

On paper, this looks like progress, in the form of reducing subjectivity and inconsistency among judges, counteracting human bias with more and better information, and using this information to better allocate limited resources across the strained criminal justice system.

In 2016, the watchdog publication ProPublica evaluated COMPAS risk scores against real-world outcomes. Its overall accuracy was similar across racial groups. However, the types of errors it was generating were not comparable in their distribution.

In analyses of general recidivism, ProPublica found that Black defendants who did not go on to reoffend were nearly twice as likely as white defendants to be labeled high risk - about 45% compared to 23%, a disparity in false positives that meant Black defendants were more often incorrectly flagged as posing threat. 

At the same time, white defendants who did reoffend were far more likely to have been labeled low risk. Roughly 48% of white recidivists were misclassified this way, compared to 28% of Black recidivists. In other words, the system systematically overestimated risk for Black defendants while underestimating it for white defendants. How did this happen?

It is impossible for an algorithm to be fair in every sense at once. In this case, two common ideas of fairness operate at odds with one another. The first (predictive parity) says that a risk score should mean the same thing for everyone—that a score of seven should predict the same likelihood of future behavior no matter who receives it. The second (error-rate parity) says that mistakes should be spread evenly, so no cohort is more likely to be wrongly labeled high risk or low risk. When different groups have lived under different conditions—such as unequal policing, housing, or access to opportunity—these goals come into direct conflict. 

Making a score of seven mean the same thing for everyone inevitably causes more mistaken “high-risk” labels for one group; reducing those mistakes means that a seven no longer means the same thing across groups. In the case of COMPAS, the combination of its selected inputs and thresholds resulted in systematically higher risk scores for Black defendants. As a result, a lower percentage of “high score” people went on to reoffend. In the case of white defendants it was the opposite. Their “risk factors” were calculated to be lower, so a higher percentage of the “low risk” individuals committed crimes.

Choosing one definition of fairness always means giving up another. What gets framed as a technical limitation is actually a value judgment about which kinds of harm we are willing to accept.

COMPAS satisfied one reasonable definition of fairness with similar overall score accuracy. Critically though, it violated others with the inequality in its false positive and negative rates. Accordingly, the harm caused by the algorithm’s use was not equally distributed. This fact points to an uncomfortable and unavoidable truth about model design: You cannot simultaneously equalize accuracy, false positives, and false negatives across groups.

So designers have to start asking tricky conditional fairness questions: Is it worse to wrongly detain someone who poses no risk, or to wrongly release someone who does?

COMPAS does not merely do “what it says on the tin”, as the British say, which is predict risk. It encodes a set of beliefs about what factors people should be evaluated against, what kinds of errors we are willing to tolerate, and which populations will bear the cost of those errors. When you look more closely at its design inputs, you can see where fairness decisions were made, consciously or less so:

• COMPAS chooses to predict the outcome of rearrest vs reconviction, which in and of itself is not indicative of criminality.

• It relies on arrest data which reflects policing patterns, arguably not morally neutral or consistent.

• It weights social stability variables, which are subjective and nuanced.

• It sets thresholds that assign traits to defendants labeled “high risk”.

• It was used in ways that created due-process concerns and with too high a comfort level among its users with its lack of transparency; in fact, the proprietary logic of the system was protected by the manufacturer as a trade secret.

• Scores could not be challenged.

None of these aspects of the system is neutral; all reflect choices.

Because it is considered an advisory and not determinative tool, courts largely upheld its use. In practice, however, data and numbers used in trials have weight and authority ascribed to them.

COMPAS is a widely used example of the pitfalls of algorithmic design, not because it is bad, but because it is mostly good and was designed with positive intent, yet has disproportionate consequences for minority populations, as many algorithmic decision systems do.

In mathematics, there are 21 documented definitions of fairness. There is no way a single algorithm can accommodate all of them, especially as they can be directly contradictory to one another (see example above). Fairness tradeoffs are an inevitable, unavoidable aspect of system design, particularly when deployed at scale. Bias arises without intent, or even understanding. Designing for efficiency is not equivalent to designing to reduce harm, and the greater the complexity of the system, the harder it is to understand or recreate the mechanism behind its outputs.

Designers of algorithmic systems face an often-unacknowledged choice: whether to encode fairness as a constraint at the outset, or to pursue accuracy first and attempt to correct disparities in outcomes later. The former treats fairness as a guiding principle, even at the cost of performance. The latter treats fairness as a metric to be optimized after the fact, assuming inequities can be repaired without addressing the assumptions that produced them. Both approaches reflect moral priorities, and one is not always better than the other. What differs is whether those priorities are made explicit at the jump or are identified as part of the optimization process. When fairness is embedded upstream in an algorithm, it is enforced downstream at scale, for better or worse.

The controversy surrounding COMPAS was not that it failed to be fair, but that it was fair in a limited way, distributing errors unevenly across populations, and shielding the value judgments that produced those outcomes behind proprietary code.

Discussions of practical ethics in tech need to move beyond right/wrong categorization and towards better/worse. People like to say technology itself is morally neutral. It isn’t always, but it is rarely entirely good or entirely evil. To co-evolve with it as humans, as opposed to being massively outpaced by it as it feels like we are now, we need to get comfortable with shades of grey, and whether we are inventors or users, we have an obligation to interrogate it.

The next time you (knowingly) engage with an algorithmic system, and I guarantee you will be soon, put it under the microscope a little. Knowing tradeoffs are inevitable, ask yourself who benefited from its creation, and how and for whom that party might have prioritized fairness? Who is most at risk from an erroneous or biased output? Finally, what does accountability look like when the harm inflicted is, at the end of the day, just a matter of numbers?

Pre-credits scrolling text: Despite pressure from reform advocates, the COMPAS system is still in use today. While risk assessment tools have evolved and some jurisdictions have put pre-trial risk assessment limits or transparency rules in place, judges and parole boards continue to consider its risk scores when making decisions about bail, sentencing, and supervision.

Author’s note: Outline development and copyediting were supported by ChatGPT. The language is my own. Responsibility for factual accuracy, framing, and conclusions rests solely with the author.