[ICML 2025] ActiveUltraFeedback: Slashing RLHF Labeling Costs via Active Delta Learning

TL;DR

ActiveUltraFeedback is a new modular pipeline that uses active learning to solve the data bottleneck in LLM alignment. By introducing two new selection strategies—DRTS and DeltaUCB—it can train high-performing models using only 16.7% (1/6th) of the data required by traditional static methods. It bridges the gap between reward modeling and fine-tuning, providing a unified framework for efficient preference data generation.

Background: The Price of Alignment

Aligning Large Language Models (LLMs) via Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) is notoriously expensive. We need thousands of pairwise comparisons (e.g., "Response A is better than Response B"). Traditionally, these pairs are selected randomly or through simple heuristics, which wastes the annotation budget on trivial or uninformative samples.

The authors identify a critical misalignment: standard Dueling Bandit algorithms aim to find the best arm (regret minimization), but for training a model, we need pairs that describe the difference in quality—the Delta.

Methodology: Active Delta Learning

ActiveUltraFeedback treats the prompt as a context and the possible LLM responses as "arms." The pipeline follows a looping structure:

1. Generate: Diverse responses from a 30-model pool.
2. Predict: Use an Epistemic Neural Network (ENN) to estimate rewards and, crucially, uncertainty.
3. Select: Choose the most informative pair using delta-focused strategies.
4. Annotate: Get a "gold" label from a strong LLM judge (e.g., Qwen 3 235B).
5. Update: Retrain the reward model to improve future selections.

The Core Innovation: DRTS & DeltaUCB

Instead of selecting two "likely best" responses, the authors propose:

• Double Reverse Thompson Sampling (DRTS): Samples one response likely to be the best and one likely to be the worst from the reward posterior.
• DeltaUCB: Deterministically selects the pair that maximizes the optimistic quality difference.

Experiments: Doing More with Less

The authors tested the pipeline across four major benchmarks: GSM8K (Math), IFEval (Instruction Following), TruthfulQA, and AlpacaEval 2.

Key Findings:

• Sample Efficiency: Models trained on just 5,000 to 10,000 actively selected samples outperformed those trained on 60,000 randomly selected samples.
• Versatility: Unlike the static "DeltaQwen" heuristic, which only works well for DPO, ActiveUltraFeedback methods (DRTS/DeltaUCB) excel at both Reward Modeling and Policy Fine-tuning.
• Algorithm Agnostic: The benefits held steady whether using DPO, IPO, or SimPO.

Critical Insights

The research reveals that uncertainty minimization (InfoMax) or regret minimization (DTS) are actually bad for data generation. Why? Because they tend to pick pairs that are very close in quality or focus on "fixing" small models that are beyond saving. High-quality training data requires a clear winner and a clear loser so the model can learn the boundary of "good" vs "bad" behavior—a concept the authors call the Delta Learning Hypothesis.

Conclusion & Future Work

ActiveUltraFeedback represents a shift from "big data" to "smart data" in AI alignment. By using uncertainty to hunt for high-delta pairs, researchers can align models in niche domains where expert time is scarce.

The main limitation is the computational cost of generating 30 candidate responses per prompt. The authors suggest that future work should focus on active model selection—deciding which specific LLMs to query for responses in the first place, rather than filtering a large pre-generated pool.

Note: The code and datasets are open-sourced at lasgroup/ActiveUltraFeedback.