[CVPR 2025] Flow3r: Factoring 2D Flow to Unlock Scalable 4D Geometry Learning

TL;DR

Refining 3D geometry from videos usually requires perfect labels (poses/depth), which don't exist for most "in-the-wild" footage. Flow3r solves this by using a "Factored Flow" head that supervises the model using raw 2D pixel movement (flow) from unlabeled videos. By forcing the model to predict flow using geometry latents from one frame and pose latents from another, it learns 3D structure and camera motion at an unprecedented scale—leveraging 800,000 unlabeled videos to reach new SOTA levels.

The Scaling Wall: Why Labels are the Enemy

The transition from optimization-based methods (like COLMAP) to feed-forward networks (like DUSt3R or VGGT) has made 3D reconstruction fast. However, these models are still "data-hungry" for 3D labels. While LLMs scale with raw text, Visual Geometry models are stuck because we can't easily get 3D ground truth for a cat jumping or a car driving in a city.

Previous attempts to use 2D Flow as a proxy failed for two reasons:

1. Simple Tracking: Just matching patches helps the model see features but doesn't teach it about depth or rotation.
2. Rigid Projection: Forcing flow to follow rigid math works for static rooms but breaks the moment something moves (dynamic scenes).

The Core Insight: Factored Flow Prediction

The researchers at CMU asked: How can flow effectively supervise geometry? Their answer is Factorization. In a static scene, the flow from Image A to Image B is purely a function of A's 3D structure and B's camera position.

As shown in the figure above, Flow3r's architecture (d) creates an information bottleneck:

• It takes Geometry Latents ($g_i$) from the source.
• It takes Camera Latents ($c_j$) from the target.
• It decodes these into 2D flow.

Because the flow head only sees the target's camera token, the Multi-view Transformer is forced to pack all the necessary motion information into those specific latents to satisfy the flow loss. This naturally handles dynamic scenes where the "motion" is a mix of camera travel and object movement.

Large-Scale Supervision Workflow

Flow3r uses a two-stage "Teacher-Student" pipeline:

1. Teacher: Use a pre-trained correspondence model (like UFM) to generate "Pseudo-ground-truth" 2D flow for thousands of unlabeled videos.
2. Student (Flow3r): Train the 3D backbone to minimize the difference between its factored flow and the teacher's 2D flow.

Experiments: More Data Beats Better Labels

The results are a testament to the power of scaling. In their training, they used 800K unlabeled videos from Kinetics-700, SpatialVID, and EPIC-Kitchens.

Key Quantitative Wins:

• In-the-wild Dynamics: On Sintel, Flow3r reduced the Relative Pose Error (RPE) for rotation by 18% compared to the previous best model (${\pi }^3$).
• Scaling Laws: The team found that training with 1K labeled + 20K unlabeled sequences outperformed using 4K labeled sequences. This proves that "cheap" unlabeled data is more valuable for generalization than "expensive" labels.

Note: Flow3r produces significantly cleaner point clouds for dynamic subjects (like the cat and the person interacting) where baseline methods often create "ghosting" or duplicated artifacts.

Critical Analysis & Future Outlook

Strengths:

• Architecture-Agnostic: Factored flow can be plugged into almost any Transformer-based geometry model (the authors tested it on both VGGT and ${\pi }^3$).
• Handling Dynamics: By decoding in latent space rather than using hard-coded projective equations, it avoids the "instability" typical of self-supervised SfM.

Limitations:

• Teacher Dependency: If the 2D flow teacher (UFM) makes a mistake, the 3D student might inherit that bias.
• Complexity: Scenes with many independently moving parts (e.g., a swarm of bees) still pose a significant challenge for the single target-view camera token.

Conclusion

Flow3r marks a shift in how we think about 4D vision. Instead of trying to build bigger datasets with LiDAR and expensive sensors, we should be building smarter architectures that can "distill" the 3D structure inherent in the billions of hours of video already available on the web.

Explore the code and project at flow3r-project.github.io