Misaligned by Reward: The Hidden Social Bias in LLM Gatekeepers
Misaligned by Reward: Socially Undesirable Preferences in LLMs
This paper introduces a systematic framework to evaluate Reward Models (RMs) on four socially consequential domains: safety, bias, morality, and ethical reasoning. By converting existing social datasets into pairwise preference formats, the authors benchmark seven models (including dedicated RMs like Beaver-7B and proxies like Mixtral-8x7B), revealing that current alignment techniques frequently fail to capture socially desirable preferences.
TL;DR
While we focus heavily on fine-tuning Large Language Models (LLMs) to follow instructions, we often ignore the Reward Models (RMs) that guide them. This paper reveals a disturbing reality: current RMs—the very "teachers" used in RLHF—are frequently misaligned with social values. They often prefer stereotypical, unsafe, or unethical responses, and their "anti-bias" mechanisms sometimes break their ability to stay contextually relevant.
Background: The RM as the Evaluator
In the Reinforcement Learning from Human Feedback (RLHF) pipeline, the Reward Model acts as a proxy for human judgment. If the RM is biased, the LLM it trains will inevitably inherit those biases. Despite this, RM benchmarking has historically been limited to "helpfulness" and "reasoning," ignoring the socio-technical dimensions of safety and ethics.
The Core Problem: Why Social Benchmarking is Hard
Most social harm datasets are designed for classification (e.g., "Is this toxic?"). To test a Reward Model, these must be converted into pairwise preferences (e.g., "Which of these two is better?"). The authors identify two specific challenges:
- Normative Tasks: Finding if the model can pick the "safe" path.
- Diagnostic Tasks: Finding if the model has an "innate preference" (e.g., does it consistently score sentences with "doctor" higher if the pronoun is "he"?).
Methodology: A New Framework for Social Preference
The authors adapted five major datasets into a unified preference format:
- Safety (Gretel): Choosing safe over unsafe responses.
- Ethics (ETHICS): Navigating justice, virtue, and deontology.
- Morality (Moral Stories): Implicitly understanding social norms.
- Bias (StereoSet & WinoGender): Testing for stereotypical associations.
Figure 1: Performance across Gretel Safety and ETHICS domains. Note the high variance—no model is a "universal" social expert.
Key Insights & Results
1. The "Beaver" Paradox
The Beaver-7B model presents a fascinating case study. It achieved the highest accuracy on Moral Stories (0.696) but performed catastrophically on Safety (0.142). It consistently ranked unsafe completions above safe ones. This proves that "moral knowledge" is not the same as "safety adherence."
2. Context vs. Bias: The Alignment Trade-off
One of the most profound findings involves the StereoSet results. Most models prefer stereotypical continuations because they are contextually coherent. However, Qwen-1.5-7B showed a unique behavior: it would rather pick a totally unrelated sentence than a stereotypical one.
- Insight: This suggests that over-correcting for bias can lead to "contextual hallucinations," where the model sacrifices logic just to avoid a social pitfall.
3. Systematic Gender Bias
In gender bias testing (WinoGender), the results were polarized. Older models like Pythia-6.9B leaned male-heavy, while newer instruction-tuned models like Mixtral and Beaver leaned heavily toward favoring female variants.
Figure 2: Directional log-odds on StereoSet. Negative values (left) indicate a preference for stereotypes.
Detailed Performance Table
The following table summarizes the struggle of current SOTA models across these critical domains:
| Model | Safety (Gretel) | Morality (Moral Stories) | Ethical Reasoning |
|---|---|---|---|
| OA DeBERTaRM | 0.669 | 0.634 | 0.665 |
| Beaver-7B | 0.142 | 0.696 | 0.485 |
| Mixtral-8x7B | 0.705 | 0.389 | 0.517 |
| RM-Gemma-2B | 0.414 | 0.675 | 0.539 |
Critical Analysis & Conclusion
This research highlights that alignment is not a single "knob" you can turn up. A model that is "safe" may not be "moral," and a model that "avoids bias" might lose its "common sense."
Takeaways for the Industry:
- Don't trust the Proxy: Just because a model follows instructions (proxied by RewardBench) doesn't mean it won't reward toxic behavior in a specific niche.
- Targeted Training: We need RMs specifically trained on diverse social contexts, not just general-purpose chat data.
- Audit the Auditor: Before using an RM to train your next LLM, it must undergo a social bias audit.
Limitations: The study is limited to English-centric norms. Social desirability is culturally dependent; what is "aligned" in one culture may be "misaligned" in another. Future work must bridge this gap to create truly global, socially intelligent AI.