[Law & Tech] Beyond the Black Box: Why Post Hoc AI Explanations Might Fail the Rule of Law
Abstract-Artificial intelligence (AI) is a technology receiving significant attention from lawmakers, courts, and regulators. An aspect of this attention is an interest in understanding how AI works when applied to a process of law, or to a regulated application of technology such as driverless vehicles. One approach is to seek to understand what the AI technology does, with goals including ³transparency´ and ³explainability´. This paper considers these concepts from a law and technology perspective. Research in this area commonly examines the challenge of ³black box´ technologies, particularly the approach of ³post hoc explainability´. This paper points out that the post hoc approach provides an inference, rather than an actual description of AI behavior. It considers circumstances in which the post hoc approach may be satisfactory, and those involving arbitrary power in which it should not be used, as inconsistent with the principle of regularity in the rule of law. It recommends that the output of non-transparent AI technologies should necessarily be viewed critically. It concludes that human attention is required in determining whether or not to accept AI technology explanations
This paper explores the legal implications of Artificial Intelligence (AI) transparency within the framework of the rule of law. It analyzes the limitations of "post hoc explainability" and proposes a new tripartite classification of AI behavior—Transparent, Post Hoc, and Arbitrary—to guide judicial and regulatory treatment of "black box" technologies.
TL;DR
As AI enters the courtroom and regulatory spheres, the industry's obsession with "explainability" faces a harsh legal reality. This paper argues that most AI explanations are merely "post hoc rationalizations"—guesses about why a model did what it did. To protect the rule of law, we must distinguish between "regular" AI behavior and "arbitrary power," ensuring that machines are held to the same standards of consistency as the law itself.
The Tension: Regularity vs. The "Mysterious Black Box"
The foundation of the rule of law, as defined by legal scholar A.V. Dicey, rests on regularity: the idea that law is applied consistently rather than through arbitrary power.
However, modern AI—particularly Deep Neural Networks—presents a "black box" problem. When a teacher is fired based on an algorithmic score (as seen in Houston Federation of Teachers v Houston Independent School District), the lack of transparency isn't just a technical hurdle; it’s a constitutional crisis. The author points out that "mystery" in technology is often a mix of inherent uncertainty (quantization errors) and "remediable incomprehensibility" (trade secrets).
Methodology: The Tripartite Classification of AI Behavior
The core contribution of this work is the expansion of Zachary Lipton’s famous binary view of AI interpretability. The author argues that a binary "Transparent vs. Opaque" view is insufficient for the law. Instead, we must categorize AI into three distinct buckets:
- Transparency to Humans: Models simple enough that their internal logic is directly graspable (e.g., short decision trees).
- Post Hoc Explanations: Complex models where we infer a reason after the fact. Crucially, this is an inference, not a description.
- Arbitrary Power: AI that lacks consistency or displays unpredictability, effectively acting like a "digital roll of the dice."
(Note: This conceptual framework bridges Lipton's technical categories with the legal principle of regularity.)
Why "Trusting the Machine" is a Legal Fallacy
The paper identifies four dangerous "pro-tech" arguments that often bypass legal scrutiny:
- Technology as Superior: The assumption that because humans are flawed (e.g., judges being harsher when hungry), machines are inherently better.
- Technology as Cornucopia: The "don't slow down progress" argument that ignores unknown risks for unproven benefits.
- Higher Standards: The claim that it is "unfair" to hold AI to a higher standard of explanation than a human expert.
- The Agency of Tech: Anthropomorphizing algorithms (e.g., "the algorithm needs to understand"), which masks the fact that algorithms can "lie" or produce arbitrary results.
Critical Findings: Viewing Explanations as Evidence
In the context of the Australian Evidence Act 1995 or U.S. case law like Overton Park, the author suggests a shift in how we treat AI output:
- No Presumption of Regularity: We shouldn't assume a device "ordinarily produces" a specific result just because it's a computer.
- Critical Scrutiny: Because post hoc explanations are inferences, they must be viewed "critically." They are not the "ground truth" of the model’s reasoning.
- The Human Sentinel: Currently, no automated system can determine if an AI is acting "arbitrarily." Only human reasoning can bridge the gap between a mathematical output and a legal justification.
(Note: The author references comparative law across the US, UK, and Australia to show the global nature of this challenge.)
Conclusion: Human Oversight is Non-Negotiable
The takeaway for technical architects and legal practitioners is clear: Transparency is not always required for adoption, but critical awareness is.
We cannot let "explainable AI" (XAI) become a "comfortable hammock" where we abdicate our responsibility. If an AI's behavior is arbitrary, it has no place in a legal system. For the rest, we must treat their "explanations" with the same skepticism we apply to any other piece of inferred evidence. The "Human in the Loop" isn't just a design choice; it is a legal necessity for the foreseeable future.