Why simply adding a human reviewer isn't enough
The instinct to 'keep a human in the loop' often means adding a final approval step. But research shows that this can backfire. A 2021 multi-method study (analytical model, experiments, simulation) found that when humans receive AI advice, their choices converge toward the AI's suggestions, improving individual accuracy but eroding 'unique human knowledge'—the insights that only humans bring [5]. The researchers liken this to becoming 'Borgs': strong individually but losing individuality, which harms the 'wisdom of crowds'—the collective accuracy that comes from diverse independent judgments [5].
This means that a human reviewer who is constantly exposed to AI output will start to think like the AI, reducing the diversity of perspectives that makes human oversight valuable. The study suggests that without intervention, human-AI teams are far less effective than human-only groups in collective decision-making [5]. So the loop must be designed to preserve and encourage human uniqueness, not just to rubber-stamp AI decisions.
Designing the loop to actually use human knowledge
The solution is to build systems that actively solicit and integrate human knowledge, rather than passively presenting AI output. A 2023 design science study on human-centric AI (in two projects, SCHOPPER and VRAILEXIA) operationalized this with two stages: first, a human-in-the-loop informed design process, and second, a usage architecture that aggregates AI and human inputs [3]. This approach integrates human knowledge into the AI's design and training, provides humans with understandable explanations of predictions, and creates 'augmented intelligence' where humans and AI counterbalance each other's errors [3].
Concretely, this means features like: prompting users to provide their own reasoning before showing AI output, offering explanations of AI predictions that users can challenge, and personalizing AI advice to individual users. The 2021 study specifically suggested personalizing AI advice as a mitigation technique, and their simulation showed that such interventions perform well both individually and in wisdom-of-crowds settings [5]. So the loop should be a two-way exchange, not a one-way feed.
Practical mechanisms you can implement today
You can translate these principles into concrete features. First, use prompts to nudge humans to contribute their own knowledge. A 2022 study on a documentation assistant (Themisto) found that a 'user prompt approach' reminded participants to document code they would have ignored, improving their satisfaction [1]. This shows that simple prompts can activate human input that would otherwise be lost.
Second, provide explanations and allow humans to override. The human-centric AI framework emphasizes giving humans understandable explanations of AI predictions, which enables them to act as a counterweight to AI biases [3]. Third, personalize the AI's advice to each user, as suggested by the 2021 study, to reduce convergence and preserve diversity [5]. These mechanisms turn the loop from a formality into a genuine collaboration.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2023, 3 in Q1 journals, collectively cited 381 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 18 papers retrieved from a database of over 500 million.
Sources used in this answer
Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks
In a within-subjects experiment with 24 data science practitioners, the Themisto system—which used automated generation, API retrieval, and user prompts—reduced documentation time, reminded users to document ignored code, and improved satisfaction, showing that prompts can effectively bring human input into AI-assisted workflows.
AI in health: keeping the human in the loop
This editorial on AI in health argues for keeping the human in the loop, emphasizing the need for human oversight in clinical AI applications, though it provides no specific data.
Human-Centric AI to Mitigate AI Biases
Based on two design science projects (SCHOPPER and VRAILEXIA), the authors propose a human-centric AI framework with a human-in-the-loop design process and a usage architecture that aggregates AI and human inputs, aiming to integrate human knowledge, provide explanations, and create augmented intelligence to mitigate AI biases.
Human-in-the-Loop Intelligence: Advancing AI-Centric Cybersecurity for the Future
This review of AI in cybersecurity proposes a Human-in-the-Loop Intelligence Cybersecurity Model that synergizes human intelligence with AI to address limitations in current AI-driven security applications, though it is conceptual and lacks empirical data.
Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
In a multi-method study (analytical model, experiments, simulation), the authors found that human choices converge toward AI advice, improving individual accuracy but decreasing unique human knowledge, harming wisdom of crowds; they suggest personalizing AI advice as a mitigation that performs well individually and collectively.