HP-Edit: Mastering Human-Aligned Image Editing via RLHF and Hard-Case Filtering

TL;DR

HP-Edit is a sophisticated post-training framework that brings the power of Reinforcement Learning from Human Feedback (RLHF) to image editing. By leveraging a task-aware VLM scorer and a curated "hard-case" dataset (RealPref-50K), it bridges the gap between raw model capabilities and human aesthetic preferences, significantly outperforming current SOTA models like FLUX.1 and Qwen-Image-Edit.

The Alignment Gap in Image Editing

While Supervised Fine-Tuning (SFT) has successfully bestowed models with basic editing capabilities, it often results in "technically correct but aesthetically lacking" outputs. Existing datasets are frequently polluted with synthetic artifacts or cartoonish styles that don't reflect real-world user expectations. The industry has long lacked a scalable way to tell a model: "Don't just remove the object; remove it so naturally that a human can't tell it was ever there."

Methodology: The Three Pillars of HP-Edit

The researchers from Huawei Noah’s Ark Lab and partner universities realized that alignment requires more than just more data—it requires better feedback.

1. HP-Scorer: The Digital Connoisseur

Instead of relying on expensive human cycles, the team developed HP-Scorer. Built on top of VLMs like Qwen2.5-VL and GPT-4o, it uses progressively refined, task-specific prompts. For instance, in an "Object Swap" task, the scorer doesn't just look for a new object; it asks: "Is the replacement feasible? Is the original object completely gone? Does the lighting match?"

2. The Logic of Hard-Case Filtering

A key insight of the paper is that easy samples provide weak gradients. If a model already scores a '5' on a simple task, training on it further is a waste of compute. HP-Edit intentionally filters out these high-scoring samples to create RealPref-50K, a dataset comprised of "hard cases" where current models struggle. This forces the RL logic to focus on the marginal gains that define high-quality results.

3. Flow-GRPO: Optimization at Scale

The framework adopts Group Relative Policy Optimization (GRPO). Unlike standard RL which might require a separate value model, GRPO normalizes rewards within a group of samples generated from the same prompt. This is paired with Flow Matching, converting deterministic ODE paths into stochastic SDEs to allow for effective exploration of the "preference space."

Experiments and Results

The framework was tested on RealPref-Bench, a new benchmark of 1,638 real-world cases.

• Quantifiable Improvement: HP-Edit pushed the baseline score from 4.472 to 4.667.
• Universal Superiority: Whether the task was "Relighting," "Bokeh," or "Object Removal," HP-Edit consistently ranked #1 against competitors like Step1X-Edit and OmniGen2.
• Human Correlation: The HP-Scorer achieved a Pearson Correlation Coefficient (PCC) of 0.89 with human users, proving it is a reliable proxy for human taste.

Critical Insights: Why It Works

The ablation studies (see the reward curve below) reveal a fascinating trend: the "Unfiltered Data + Base Scorer" setup actually led to performance degradation. This highlights a dangerous pitfall in RLHF—poorly defined rewards or "easy" data can distract the model. Only the combination of task-specific scoring and hard-case density produced a stable, upward trajectory in quality.

Conclusion

HP-Edit marks a transition from "instruction-following" to "preference-alignment" in the image editing domain. By treating image generation as a Markov Decision Process and applying rigorous RLHF principles, the authors have provided a scalable path toward indistinguishable-from-real AI editing. While it still faces hurdles in niche areas like code-switching text edits, its success across eight core tasks makes it a new lighthouse for post-training research.