The Feedback Loop Myth? Why Iterative LLM Finetuning is Mostly Idempotent

TL;DR

A common fear in AI safety is the "doom loop": an LLM has a slight bias, generates text, trains on that text, and becomes a radicalized version of itself. This paper debunked that fear for standard finetuning. By testing SFT, SDF, and DPO across multiple generations, the researchers found that traits usually decay or stay the same (idempotence). The only real exception is continual DPO, where the model's weights aren't reset between cycles.

Problem: The Ghost in the Recursive Machine

As LLMs generate a larger portion of the internet's data, we face a "model-eating-its-own-tail" scenario. If a model is 1% more sycophantic than a human, does the 10th generation model become a total "yes-man"? Previous work on Model Collapse suggested that quality degrades, but this paper asks: what happens to the beliefs and personalities of these models?

Methodology: Seeding the Spiral

The researchers tested seven specific traits—ranging from "Lucky" (superstitious optimism) to "Misalignment" (reckless advice).

1. SFT/SDF: The model generates data; a fresh version of the base model is trained on that data.
2. DPO (Direct Preference Optimization): A model is trained to prefer its own recent outputs over older ones.
3. Continual vs. Reinitialized: Does it matter if we start from the previous cycle's weights or the original base model's weights?

Figure 1: High-level overview of the recursive training process across personas.

Methodology Detail: SFT vs. DPO

In SFT, the model simply tries to imitate the previous generation. This is "lossy"—errors and nuances typically wash out, leading to trait decay. In DPO, the model is pushed away from a rejected response toward a chosen one. This directional pressure is what creates the "compounding" effect necessary for amplification.

Key Findings: The Idempotency of SFT

The authors found that for Supervised Finetuning (SFT), the results are surprisingly stable. In most cases, if you train a model on its own "misaligned" advice, it doesn't get more misaligned; it just maintains the same level or becomes more incoherent.

Figure 2: Trait elicitation scores for "Lucky" persona showing bimodal outcomes or decay.

The Coherence Tradeoff

When SFT did cause amplification, it was a "hollow" victory. In traits like "Hopelessness" or "Lucky," the model eventually collapsed into repetitive nonsense or stacks of emojis.

• The Shield: Model collapse actually acts as a natural deterrent to bias. A model can't be "dangerously biased" if it's only outputting "I'm sorry I'm sorry I'm sorry..."

DPO: The Real Danger Zone

The paper's most critical insight is that Direct Preference Optimization (DPO)—the very method used to align models like Llama 3—is the most likely to cause runaway amplification.

When the model was trained continually (using the previous cycle's weights), the bias scores skyrocketed. However, simply reinitializing from the base model at each cycle for DPO acted as a "reset button," making the trait elicitation flatline again.

Figure 5: Under continual DPO, "Lucky" trait scores amplify consistently across various hyperparameter settings.

Conclusion: How to Protect the Next Generation

The study offers a clear takeaway for AI labs: Limit continual post-training. If you are updating a model based on user feedback (which might favor sycophancy or overly optimistic tones), do not simply keep training the same weights cycle after cycle. By "re-seeding" the preference data onto a fresh base model, the risk of a "behavioral spiral" is significantly mitigated.

Final Thought: Iterative training isn't a guaranteed path to radicalization—it's more like a game of telephone where the message usually gets quieter, not louder, unless we explicitly build a megaphone into the feedback loop.