Open-Source Image Editors: The Unexpected Zero-Shot Vision Powerhouses

TL;DR

Can a model trained to "add a cat to the sofa" also estimate the precise 3D geometry of the room? This paper reveals that open-source image-editing models like Qwen-Image-Edit and FireRed-Image-Edit are secretly elite vision learners. Without a single line of task-specific code or fine-tuning, these models can generate depth maps and surface normals that rival or even beat specialized, fine-tuned SOTA models like Marigold.

Context: Beyond "Just" Generating Images

In the current AI landscape, we've seen models like Google's Veo 3 show flashes of visual reasoning. However, these are often closed-source black boxes. The researchers here asked a fundamental question: Is visual understanding an emergent property of the image-editing training objective itself? To edit an image convincingly—to change a texture while keeping the light or to swap an object while maintaining the layout—a model must internally understand depth, boundaries, and 3D orientation.

Methodology: The Prompt-and-Decode Pipeline

The authors didn't change the models; they changed the "questions" they asked them. By using a unified pipeline, they forced the models to speak the language of dense vision tasks through RGB pixels.

• Depth Estimation: Prompting for grayscale maps where brightness equals proximity.
• Surface Normals: Prompting for standard RGB normal maps where colors represent XYZ vector orientations.
• Semantic Segmentation: Asking the model to "paint" specific classes (cars, roads, sky) with specific hex codes.

To handle the lack of metric calibration in generative models, they used Affine Alignment for depth (adjusting scale and offset) and a clever 48-way Automatic Axis Calibration for surface normals to match coordinate system conventions.

Figure 1: Notice how the zero-shot editors capture the relative distance of the furniture and walls in these NYUv2 samples.

Key Findings: David vs. Goliath

The results are startling, particularly in the realm of geometric understanding.

1. Surface Normals: Outperforming the Specialists

FireRed-Image-Edit achieved a 17.69° mean angular error on NYUv2. To put this in perspective, Marigold—a model specifically fine-tuned for this task—scored 20.86°. This means the general-purpose editor has a "truer" grasp of surface orientation out of the box than some specialized models.

2. Semantic Segmentation: The Format Bottleneck

While the models succeeded in geometry, semantic segmentation (Cityscapes) proved harder. Qwen-Image-Edit hit 25.7 mIoU. While non-trivial, it lags behind instruction-tuned models (69.9 mIoU). The failure wasn't usually in finding the road or the car, but in remembering to paint it the exact color requested.

Table 1: Quantitative comparison showing FireRed-Image-Edit outperforming the specialized Marigold pipeline.

Critical Insight: The "Visual Proxy" Problem

The study highlights a fascinating distinction between Visual Understanding and Format Following.

• Depth is easy to recover because the "intensity" of a pixel is a natural proxy for distance.
• Segmentation is hard because "Red = Road" is an arbitrary linguistic mapping that conflicts with the model's natural instinct to produce realistic images.

This suggests that if we can improve the "instruction-following" consistency of these open-source editors, we might unlock a "Universal Vision Model" that doesn't need to be trained on labeled data for every new task.

Conclusion & Perspective

This work shifts the narrative from "Generative AI is for art" to "Generative AI is a foundation for perception." The fact that three independently trained models all exhibit these traits confirms that editing is a proxy for understanding.

Limitations to Watch:

• The models still struggle with fine-grained textures and thin structures (like poles or wires).
• Accuracy drops significantly in large-scale outdoor scenes compared to bounded indoor environments.
• Instance segmentation remains the "final boss" due to the difficulty of assigning unique, consistent colors to an unknown number of objects zero-shot.

For practitioners, this research provides a powerful baseline: before you start fine-tuning a heavy vision transformer on your custom dataset, check if an off-the-shelf editor can already "see" what you're looking for.