[Evaluation Benchmark] CUDRT: Decoding the DNA of LLM-Generated Text Across Languages and Operations

TL;DR

Researchers from Renmin University have released CUDRT, a massive bilingual benchmark that moves LLM detection from "guessing the model" to "understanding the operation." By breaking down LLM tasks into five key actions—Create, Update, Delete, Rewrite, and Translate—they've uncovered that the "Delete" (summarization/refining) operation is the ultimate "litmus test" for training highly generalizable detectors.

Problem & Motivation: The Benchmark Bottleneck

As LLMs become mirrors of human writing, traditional detectors are failing. Most existing benchmarks are "static snapshots" focused on English Question Answering (QA). However, in the real world, LLMs are used for much more: polishing emails, summarizing reports, or translating legal documents.

The authors argue that current evaluations overlook two critical factors:

1. Operational Diversity: An LLM "rewriting" a paragraph leaves different statistical footprints than an LLM "creating" a story from scratch.
2. Language Gaps: Most benchmarks ignore Chinese, despite China being a powerhouse in LLM development.

Methodology: The Five Operations of CUDRT

The team developed the CUDRT framework (Create, Update, Delete, Rewrite, Translate) to systematically categorize how humans interact with AI.

• Create: Complete segments or QA.
• Update: Polishing (improving fluency) or Expanding (adding details).
• Delete: Summarization or Refining (removing redundancy).
• Rewrite: Profoundly changing the linguistic expression while maintaining meaning.
• Translate: Cross-lingual conversion.

The authors used Perplexity (PPL) analysis to show that AI-generated text is generally more "concentrated" and "certain" than human text, except in translation tasks where the distributions overlap significantly, making detection much harder.

Experiments: The Power of "Delete"

The researchers conducted three rigorous tests: Cross-Dataset, Cross-Operation, and Cross-LLM.

Key Insight: The "Delete" Generalization

One of the most surprising findings was that when a model (like RoBERTa or XLNet) is trained on "Delete" (Refine/Summary) operations, it becomes an expert at detecting almost any other operation.

As shown in the heatmaps above, RoBERTa trained on Delete tasks achieved Accuracy/F1 scores exceeding 0.98 on Update and Rewrite tests. The authors hypothesize that when an LLM deletes or refines content, it makes highly characteristic structural choices that effectively reveal its underlying logic.

The Model Challenge

While detectors are great at spotting GPT-4's style (due to its widespread presence in training data), they struggle with less common models like Baichuan2-13B. This suggests that detectors are often "overfitting" to the style of OpenAI's models rather than learning a universal "AI-ness."

Critical Analysis & Conclusion

Takeaways

• Operation Matters: Don't just train on QA. If you want a robust detector, use refinement and summarization data.
• Open vs. Closed: Closed-source models (GPT-4) are currently easier to detect than some open-source local models, likely because the detectors have "seen" more GPT-like distributions during fine-tuning.

Limitations

The primary limitation remains the "Translate" operation. Because translation is constrained by the source text, the LLM has less "freedom" to leave its characteristic footprint, resulting in the lowest detection accuracy across the board (often hovering near 0.60).

Future Outlook

CUDRT provides a scalable blueprint for real-world moderation. The next frontier will be Lightweight Self-Learning Detectors that can adapt in real-time as models like Qwen and Llama continue to evolve their linguistic patterns.

Title: Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT Source: arXiv / RUC / SHUFE / IAAR Code: GitHub - TaoZhen1110/CUDRT