Human-LLM Coevolution: How Peer-Review Stigma is Reshaping AI Writing

TL;DR

Is your abstract too "intricate"? Did you "delve" a bit too deep? You might be changing your vocabulary specifically to avoid being flagged as an AI. A new analysis of 1.3 million arXiv papers reveals that researchers are now actively scrubbing "ChatGPT-isms" from their work. This "coevolution" means that while AI is influencing our writing, we are simultaneously training ourselves to hide its presence, making AI detection harder than ever.

The "Delve" Problem: Motivation behind the Study

Since 2023, certain words have become "dead giveaways" for AI involvement in academia. Words like delve, pivotal, and showcasing saw a vertical spike in usage frequencies. However, as the academic community became aware of these linguistic fingerprints, a new phenomenon emerged: Stigma-driven Adaptation.

The authors argue that we are no longer just using LLMs; we are "coevolving" with them. When a word is publicly labeled as "AI-generated," human authors adjust their prompts or manually edit the text to avoid it. This study aims to quantify this shift and determine if state-of-the-art (SOTA) detectors can keep up with humans who are actively trying to "foil" the system.

Methodology: Tracking 1.3 Million Abstracts

The researchers utilized the arXiv metadata (version 214) from Kaggle, covering over 1.29 million papers submitted between 2018 and 2024. By calculating word frequency per 10,000 abstracts, they tracked the "rise and fall" of LLM-associated vocabulary.

The Turning Point of March 2024

The study highlights a fascinating trend: around March and April 2024, the frequency of words like intricate, delve, and meticulous began to drop sharply. This coincides exactly with the publication of early studies (like Liang et al.) that "called out" these words in AI conference peer reviews.

Figure 1: Declining trends of "warned" AI words (a, b) vs. the continued rise of "non-stigmatized" AI words (c).

Experiments: Breaking the Detectors

To simulate the "coevolution," the authors used GPT-4o-mini with two types of prompts:

1. Standard Revision (P1): Simple rewriting.
2. Constrained Revision (P2): Instructing the AI not to use specific "giveaway" words like realm or pivotal.

Results: The Failure of SOTA Detectors

The team tested Binoculars, a SOTA zero-shot detector. The results were sobering. Binoculars produced almost identical scores for original human abstracts and LLM-revised abstracts. Furthermore, when the "negative constraints" (P2) were applied, the frequency of "AI words" dropped, but the detector was still unable to reliably distinguish the revised text from original human writing.

Figure 2: Binoculars scores showing minimal distribution shifts between original and AI-revised abstracts, indicating low detection robustness.

Academic Insight: The Invisible Influence

While researchers are scrubbing obvious words, the "invisible" influence of LLMs remains. The study notes that common words like is and are have been on a steady decline. Why? Because LLMs generally prefer more complex, "wordy," or structurally varied sentences that favor "sophisticated" verbs over simple linking verbs.

Because these common shifts don't trigger suspicion, they are likely to permanently alter the "texture" of academic English.

Critical Analysis & Conclusion

Takeaway

We are moving away from a binary "Human vs. AI" writing world toward a Human-AI Hybrid state. Detection on a per-text basis is becoming functionally impossible as users learn to bypass classifiers through prompt engineering and manual editing.

Limitations

The study focuses primarily on abstracts. While abstracts are representative, they may not capture the nuances of full-body text where LLM "drift" (hallucinations or structural repetition) might be more obvious.

Future Outlook

Future research should focus on "corpus-level forensics" rather than individual paper detection. If we cannot prove this paper is AI-written, we can at least prove that this field is becoming increasingly homogenized by LLM usage, which has significant implications for linguistic diversity and the evolution of scientific thought.