LLMs as Human Surrogates: The Growing Gap Between "Useful Assistants" and "Human-Like Agents"

Executive Summary

Large Language Models (LLMs) are no longer just chatbots; they are increasingly used as digital stand-ins for human subjects in social science and medical training. However, a groundbreaking study introducing the Psych-201 dataset reveals a troubling trend: Post-training makes LLMs less human-like. As we optimize models to be polite, helpful, and "correct," we are inadvertently stripping away the very heuristics, biases, and "human-ness" that make them useful for behavioral simulation.

The "Alignment Tax" on Human Behavior

The core tension in LLM development today is between Normative Correctness (what a model should say) and Descriptive Fidelity (what a human would say).

The researchers identified a significant "Post-training Misalignment." While base models (the raw results of pre-training) are becoming better at predicting human behavior with every generation, the subsequent stages of instruction-tuning and Reinforcement Learning from Human Feedback (RLHF) create a widening gap. The more "advanced" and "safe" an assistant becomes, the less it resembles the messy, biased, and often irrational nature of human thought.

Methodology: The Psych-201 Benchmark

To prove this, the authors released Psych-201, a massive transcript dataset 3.5x larger than its predecessor, Psych-101. It covers 200,000+ participants across hundreds of psychological paradigms, from psycholinguistics to risky decision-making.

The team measured the Negative Log-Likelihood (NLL)—essentially, how surprising a human's answer is to the model. A higher NLL means the model is "clueless" about how a human would actually react.

Key Finding 1: The Misalignment Gap is Widening

The most startling discovery is that the misalignment isn't just constant; it's accelerating. In the Qwen model family, the gap between the base model and the instruction-tuned version was small in Qwen2 but tripled in Qwen3.

Figure: The "Post-training Misalignment" gap grows with newer, more "refined" models.

Key Finding 2: Persona-Induction is a Placebo

Many researchers attempt to fix model behavior by "priming" it: “You are a 35-year-old male from Germany...”

Psych-201 allowed for the first scale-testing of this "Persona-induction." The results were disappointing: providing specific meta-data about an individual did almost nothing to improve the model's ability to predict that specific person's choices. While personas might make a model sound different, they don't capture the underlying cognitive mechanics of individual decision-making.

Figure: Effect sizes for persona-induction hover near zero, suggesting it fails to capture individual human nuances.

Critical Analysis: Why Does This Happen?

The authors suggest that post-training objectives like RLHF optimize for systematicity and reliability. In domains like Psycholinguistics and Reasoning, humans are notoriously inconsistent and prone to heuristics. When a model is trained to be a "perfect reasoner," it loses the ability to simulate the "fast and frugal" thinking that defines human cognition.

Conclusion and The Path Forward

The study concludes that we cannot rely on off-the-shelf "Assistant" models for behavioral research. Instead, we need a new paradigm of Behavioral Post-training.

The success of the Centaur model (a model specifically fine-tuned on psychological data) suggests a way out. By training models specifically to mimic human data distributions, we can create surrogates that are both useful and authentic. For the future of AI-driven social science, we must stop forcing models to be "better" than us if we want them to represent us.