[UniGRPO] Unified Policy Optimization: Bridging Reasoning and Visual Synthesis

TL;DR

UniGRPO introduces a unified Reinforcement Learning framework that treats the entire chain of Prompt → Reasoning (Thinking) → Image Generation as a single Markov Decision Process (MDP). By jointly optimizing the Large Language Model (LLM) and the Flow Matching model using Group Relative Policy Optimization (GRPO), the authors achieve significant boosts in image quality and prompt adherence, setting a new SOTA on benchmarks like GenEval (0.90).

Background: The Interleaved Frontier

As generative AI moves toward unified architectures like Bagel, Show-o, and Transfusion, the industry is converging on a hybrid approach: Autoregressive (AR) modeling for text and Flow Matching for images. However, simply hooking these components together often leads to "hallucinatory reasoning"—where the model's internal thoughts have little impact on the final pixels. UniGRPO addresses this by ensuring that the reasoning phase is explicitly rewarded for helping the generation phase.

Methodology: A Unified MDP

The "Prompt → Thinking → Image" sequence is formulated as a sequential MDP:

1. Text Phase: The model predicts reasoning tokens (actions) based on the prompt.
2. Image Phase: The model performs denoising steps (actions) conditioned on the reasoning.
3. Reward: A sparse terminal reward is given based on the final image's alignment with the original prompt.

1. Unified Optimization

The framework uses a joint objective function: $\mathcal{J}={\mathcal{J}}_{extText}+\lambda {\mathcal{J}}_{extFlow}$ Both components are updated using group-relative advantages, which eliminates the need for a separate value model (critic), drastically saving GPU memory.

2. Efficiency Hack: Eliminating CFG

In standard diffusion/flow inference, Classifier-Free Guidance (CFG) requires dual rollouts (conditional and unconditional), which is computationally expensive. UniGRPO trains entirely without CFG. The model learns to internalize alignment directly into its weights through RL, maintaining a linear, unbranched rollout that is much easier to scale to multi-turn interactions.

3. Killing Reward Hacking: Velocity MSE

A common failure in RL for images is "reward hacking," where the model generates oversaturated, unnatural textures to maximize reward scores. Instead of using the standard KL penalty on latents, UniGRPO applies an MSE penalty directly on the velocity fields: ${\mathcal{L}}_{extMSE}(heta)=\Vert v_{heta}-v_{extref}{\Vert }^2$ This forces the RL-tuned vector field to stay close to the pre-trained base model's priors across all noise levels.

Experimental Results: The Power of Joint RL

The authors compared UniGRPO against several competitive baselines, including ReFL and individual modility RL.

Note: UniGRPO achieves a near-perfect score on Single Object and Two Object categories in GenEval.

Qualitative Transformation

The transition from SFT to UniGRPO shows a shift from "generic" reasoning to "task-oriented" reasoning. In Figure 6, we see the model use its <think> space to explicitly plan spatial arrangements (e.g., "six cups in two horizontal rows"), which translates directly into a more accurate image.

Critical Analysis & Future Outlook

UniGRPO successfully demonstrates that joint post-training is the "silver bullet" for unified multimodal models. By forcing the reasoning and generation modules to share a single reward signal, the model learns a more coherent internal representation.

Limitations:

• Sparse Rewards: Currently, the model only gets feedback at the very end. The authors suggest that Multimodal Process Reward Models (PRMs) providing token-level feedback for "thoughts" will be the next leap in efficiency.
• Single Round: While the MDP is designed for multi-turn, the current experiments focus on single-round (Prompt -> Think -> Image).

The Takeaway: If you want your multimodal model to truly reason about what it generates, don't just train them together—reward them together.