Trust Your Critic: How FIRM Solves Reward Hacking in Image Generation and Editing
Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
The FIRM (Faithful Image Reward Modeling) framework introduces specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) to address hallucinations and noise in MLLM-based critics for image editing and text-to-image (T2I) generation. Utilizing high-quality curated datasets (FIRM-Edit-370K, FIRM-Gen-293K) and a "Base-and-Bonus" RL strategy, it achieves new SOTA performance on benchmarks like GEdit-Bench and GenEval.
TL;DR
The FIRM (Faithful Image Reward Modeling) framework addresses the "unreliable critic" problem in Reinforcement Learning for image tasks. By developing task-specific 8B reward models using structured data pipelines ("difference-first" for editing and "plan-then-score" for generation) and introducing a multiplicative Consistency-Modulated Execution (CME) reward strategy, the authors significantly reduce hallucinations and prevent the model from "cheating" during training.
The Problem: The Hallucinating Judge
Reinforcement Learning (RL) is the current frontier for aligning diffusion and autoregressive models with human intent. However, RL is only as good as its Reward Model (RM).
Current general-purpose Multimodal LLMs (like Qwen-VL or GPT-4o) are often used as zero-shot judges. The problem? They are prone to hallucinations and reward hacking. For example:
- In Editing: A model might learn that the easiest way to get a high "consistency" score is to do nothing—ignoring the edit instruction entirely to keep the image identical to the source.
- In Generation: A model might generate a "black shadow" for a prompt like "a cat" to perfectly match the semantic label while sacrificing all visual quality.
Methodology: Engineering a Better Critic
The authors argue that models are better "problem-solvers" than they are "evaluators." To fix this, FIRM transforms evaluation into a structured reasoning task.
1. FIRM-Edit: The "Difference-First" Approach
Instead of asking an MLLM to score an edit directly, the pipeline first generates a Difference Report. By forcing the model to describe exactly what changed (e.g., "The red car became blue, but the background trees were removed"), the subsequent scoring stage becomes significantly more accurate.
2. FIRM-Gen: The "Plan-then-Score" Strategy
For Text-to-Image generation, the framework uses a "Planner" LLM to break a prompt into a checklist (Subject, Attributes, Style, Negative Constraints). The MLLM "Evaluator" then checks the image against this specific list, preventing "attention dilution" where complex instructions are ignored.
Solving Reward Hacking: Multiplicative Coupling
One of the paper's most insightful contributions is the Consistency-Modulated Execution (CME) strategy. A typical RL setup uses a weighted sum:
As established, this leads to "lazy" models. FIRM replaces this with a multiplicative constraint:
In this formula, if the Execution is zero (the model didn't perform the edit), the entire reward is suppressed, regardless of how consistent the image is. This forces the model to prioritize the task before fine-tuning the details.
Experimental Results: SOTA with Fewer Parameters
Despite having only 8 billion parameters, the FIRM reward models outperformed much larger proprietary giants like GPT-5 and Gemini-3-Pro in alignment with human judgment on the FIRM-Bench.
Key Breakthroughs:
- FIRM-Qwen-Edit: Achieved a score of 7.84 on GEdit-Bench, surpassing the previous SOTA (7.76) by focusing on high-fidelity signals.
- FIRM-SD3.5: In generation tasks, it showed massive gains on complex, long-form prompts (UniGenBench-Long), proving that the checklist-guided training allows the critic to catch fine-grained details that general models miss.
Deep Insight & Conclusion
The success of FIRM proves that data quality and structured reasoning pipelines trump raw parameter scaling when it comes to reward modeling. By explicitly modeling the "how" (differences and checklists) rather than just the "what" (final score), FIRM provides a roadmap for more faithful generative AI.
Takeaway for Practitioners: When training RL agents, never assume a linear sum of rewards will lead to the desired behavior. Multiplicative coupling (like CME) is a powerful tool to ensure the agent doesn't find a "short-cut" to high rewards by ignoring the primary objective.
Limitations
While highly effective, the "difference-first" approach relies on the captioning model's ability to see minor details. If the initial captioning MLLM misses a subtle change, the final critic will likely be misinformed. Future work could involve iterative "zoom-in" mechanisms to detect micro-level inconsistencies.