REAL: Bridging the Gap Between Regression and Reinforcement Learning for LLM Evaluation

TL;DR

In the "LLM-as-a-Judge" paradigm, numeric scores are not just labels; they have a distance. Predicting a 4 when the truth is 5 is much better than predicting a 1. REAL (Regression-Aware Reinforcement Learning) is the first framework to integrate this ordinal logic directly into an RL pipeline. By decomposing the gradient into Reasoning Exploration and Prediction Refinement, REAL achieves massive gains in Pearson and Spearman correlations (+8.40 on Qwen3-32B).

The "Binary Reward" Trap

Most current RL post-training (like DeepSeek-R1 or standard PPO) uses binary verifiers: you are either right (reward 1) or wrong (reward 0). For math or code, this works. For evaluation, it’s a disaster. If a human says a response is "4/5", and the LLM says "3/5", standard RL treats this as a failure, the same as if the LLM said "1/5".

Furthermore, existing "Regression-Aware" methods like TRACT are stuck in the Supervised Fine-Tuning (SFT) regime. They can't "explore"—they only learn from static, pre-generated reasoning chains.

Methodology: The REAL Framework

The core innovation of REAL is its objective function, which combines a regression loss with a policy-dependent reward.

1. The Physics of the Gradient

The authors utilize a Generalized Policy Gradient. Because the reward itself depends on the model's current weights (it calculates an expected value over the digit tokens), the standard REINFORCE gradient isn't enough. The paper proves that the gradient naturally splits into two "forces":

• The CoT Update (Term 1): Traditional RL. It tells the model: "This reasoning path led to a score close to the ground truth; do more of this."
• The Prediction Update (Term 2): Supervised Learning. It uses backpropagation to push the specific digit probabilities toward the correct value.

2. Math Meets Intuition

The paper provides a formal proof (Lemma 3.1) showing that minimizing Squared Error is the optimal proxy for maximizing Pearson Correlation. This bridges a theoretical gap: while we care about how well the judge correlates with humans (population metric), we can only train on individual samples (per-sample loss).

Experimental Mastery

The authors tested REAL across Mistral-7B, Qwen-8B, and Qwen-32B.

• SOTA Correlation: REAL consistently crushed the "TRACT" SFT baseline. In out-of-domain benchmarks like Vicuna Bench, the gap was even wider, proving that RL-driven exploration helps the model generalize to new rubrics.
• RL Failure in Standard Mode: As seen below, standard RL with binary rewards actually leads to a "collapse" in correlation because the model loses the fine-grained numeric relationships.

Deep Insight: Why it Works

The "Aha!" moment comes from the Ablation Studies. The authors found that training on "partially correct" prompts (where some samples in a group are right and some are slightly off) provides the richest learning signal. This is where the regression reward shines—it gives the model a ladder to climb toward the correct score, rather than a cliff to fall off.

Key Technical Takeaways:

• Inference Efficiency: REAL achieves peak performance with $N=1$ generation. Unlike other judges that need 10+ samples to average out, REAL's policy is inherently robust.
• RAIL Predictor: During inference, the model doesn't just take the highest-probability token; it calculates the expected value ($\sum k\cdot P(k)$), which offers a "free-lunch" boost in correlation.

Conclusion & Future Work

REAL is a foundational shift for LLM-as-a-Judge. It moves us away from "guessing" tokens toward "calculating" quality. While currently limited to pointwise scoring (1-5), the potential to expand this to pairwise preferences or multi-dimensional rubrics is the next frontier for automated evaluation.

Reference: Zhang, Y., et al. (2026). REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge.