Beyond the Score: Why Human-AI Collaboration in Criminal Justice is Harder Than We Thought

TL;DR

A deep-dive study from Carnegie Mellon University reveals that how humans use algorithmic risk tools is far more complex than simple "anchoring." Crowdsourced participants often ignore statistical 50% thresholds and, crucially, real-world judicial decisions are based on the "gravity of the crime," a factor that risk-prediction benchmarks completely overlook.

Background: The Human-in-the-Loop Reality

Risk Assessment Instruments (RAIs) are everywhere—from deciding bail in Kentucky to sentencing in Pennsylvania. While machine learning researchers obsess over AUC and calibration, we often forget that these tools don't make decisions; humans do. This paper positions itself at the intersection of HCI and fairness, asking: If we give an algorithm to a human, does the "system" actually get better? Or do human biases simply swallow the algorithmic signal?

The "Threshold Check": Why Probabilities Aren't Predictions

One of the most striking findings is the breakdown of human logic when converting risk to action. In the "Rational Agent" model, if you think there’s a 40% chance of re-arrest, you should predict "No re-arrest" (since 40% < 50%).

The study proved humans are not rational agents:

• The 50% Myth: Participants predicted re-arrest for 26% of offenders they rated as less than 50% likely to fail.
• Threshold Variance: Unlike an algorithm, humans do not use a fixed internal threshold. Their "decisions" are inconsistent with their own "estimates."

The Anchoring Surprise

Conventional wisdom suggests that if you see an algorithmic score first, you will "anchor" on it and stay close to that value. The authors tested this by having one group "pre-register" their guess before seeing the RAI.

• The Result: There was no evidence of anchoring.
• The Twist: Participants who made a guess before seeing the RAI actually adjusted their final answers to be closer to the algorithm than those who saw the RAI from the start. This suggests that pre-registering an opinion might make humans more mindful of the AI’s value, rather than more resistant to it.

Methodology: The Workflow of the Study

The researchers used real data from the Pennsylvania Commission on Sentencing to build a Lasso-based RAI (Standard AUC ~0.70). They then presented 40 vignettes to Turk workers.

Why Benchmarking via Crowdsourcing is Flawed

The most profound contribution of this paper is the critique of its own medium. Many researchers use MTurk "accuracy" to claim algorithms are better than humans. Fogliato et al. expose two major issues:

1. The Effort Problem: The median time spent on a case was a mere 10 seconds. While MTurkers might be "accurate," they aren't "expert."
2. Predictions $eq$ Decisions: This is the "Aha!" moment. The RAI and MTurkers predict risk of re-arrest. But real judges look at incarceration.

Analysis of real court data showed that judges care deeply about the Offense Gravity Score (OGS)—how bad the current crime was. The RAI and the human predictors ignored this, focusing only on the "Prior Record." In short, we are benchmarking our AI on a task (forecasting) that only represents a fraction of the human's job (judging).

Final Insights

This paper serves as a warning for the AI Fairness community. Achieving "Predictive Parity" on a dataset means very little if the human user is going to ignore the score when a crime is "serious enough." To build better AI for the real world, we need to stop modeling outcomes and start modeling the values that drive human decisions.

Key Takeaways for Practitioners:

• Elicitation Matters: Always ask for both probability and action; they are not the same in the human mind.
• Context is King: An algorithm that predicts "risk" but ignores "severity" will always be at odds with judicial intuition.
• Agency over Anchoring: Forcing users to commit to an initial thought might actually improve their eventual adoption of AI advice.