Why AI can produce plausible but false claims
AI language models are designed to generate text that sounds fluent and authoritative, not to verify facts. This is why they can produce 'hallucinations'—confident statements that are false. A 2024 opinion piece in the Journal of the Association for Information Systems explicitly flags hallucinations as a key concern in using AI for peer review [1]. Similarly, a 2025 Nature letter documents an increase in fake citations generated by AI, which can propagate through the literature and reduce the reliability of published research [5].
The risk is not hypothetical: a 2023 study demonstrated the feasibility of fabricating an entire research paper using an AI chatbot, and found that human detection was not reliable [6]. This means that without safeguards, AI-assisted review could let fabricated or flawed work slip through, because the AI's output looks plausible.
What actually works to counter AI-generated misinformation
The most effective countermeasure is direct debunking—correcting false information after it appears. In a 2025 study with over 1,200 participants, debunking a misleading AI-generated article significantly reduced its influence on reasoning, and combining debunking with a pre-emptive 'inoculation' (warning about AI's unreliability) eliminated the misinformation's impact entirely [2]. This suggests that in peer review, explicit verification and correction of AI-generated claims is more powerful than just warning reviewers to be cautious.
However, the same study found that a simple disclaimer that AI information may be misleading had no effect, and even a pre-emptive warning alone did not reduce the article's influence on reasoning [2]. This is a crucial nuance: passive warnings are not enough; active debunking is required. For peer review, this means that AI tools should be paired with a human reviewer who actively checks facts and citations, not just relies on the AI's output.
The role of human oversight and transparency
The strongest safeguard is to keep humans in the loop and demand transparency. A 2024 editorial in the Journal of Management Studies introduced a policy that requires authors to oversee all AI use and take personal responsibility for accuracy, while prohibiting AI from being used in peer reviewers' evaluations [3]. This policy reflects the consensus that AI can assist but not replace human judgment in the review process.
Transparency is also key: authors must disclose how and where they used AI in their research [3]. This allows reviewers and editors to assess the reliability of the work. While AI tools can help predict citation counts and readership—as a 2024 study of 2,222 abstracts found—they are not a substitute for human evaluation of scientific quality [4]. The study found that AI assessments of 'quality and reliability' showed minimal correlation with actual citation outcomes, suggesting that AI's judgment on scientific merit is not yet trustworthy [4].
About These Sources
This answer is built on 6 peer-reviewed studies — published from 2023 to 2025, 5 from 2024 or later, 5 in Q1 journals, collectively cited 277 times — selected as the most relevant from 6 studies that passed quality screening, drawn from 43 papers retrieved from a database of over 500 million.
Sources used in this answer
Peer Review in the Age of Generative AI
A 2024 opinion piece highlights hallucinations and biases as key pitfalls in using AI to augment or automate peer review, calling for a deeper understanding of these issues.
Countering AI-generated misinformation with pre-emptive source discreditation and debunking
In two experiments with 1,223 participants, debunking effectively reduced the influence of a misleading AI-generated article, and combining inoculation with debunking eliminated its influence entirely, while a simple disclaimer had no effect.
Here, There and Everywhere: On the Responsible Use of Artificial Intelligence (AI) in Management Research and the Peer‐Review Process
A 2024 editorial policy requires authors to oversee all AI use and take personal responsibility for accuracy, and prohibits AI use in peer reviewers' evaluations and editors' handling of manuscripts.
Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts
A 2024 study of 2,222 abstracts found that ChatGPT-based assessments could predict citation counts and readership, but its 'quality and reliability' component showed minimal correlation with actual outcomes.
Tackle fake citations generated by AI
A 2025 letter documents an increase in AI-generated fake citations in scientific literature, which can propagate and decrease reliability, calling for coordinated stakeholder action.
AI-generated research paper fabrication and plagiarism in the scientific community
A 2023 study demonstrated the feasibility of fabricating a research paper using an AI chatbot, and found that human detection was not reliable in identifying such fabricated works.