[CVPR 2025] Sketch2Colab: Mastering Multi-Human Collaboration via Controllable Flow Distillation

TL;DR

Sketch2Colab is a novel framework that turns sparse storyboard sketches into complex 3D animations involving multiple people and objects. It moves beyond simple text-to-motion by using visual "intent" (sketches) to guide a Rectified-Flow model, achieving faster sampling and significantly better physical accuracy (less foot-skating, precise grasping) than traditional Diffusion models.

Problem & Motivation: Why Sketches?

In professional animation pipelines, text prompts ("Two people carry a table") are often too vague. Animators need control over timing, spacing, and contact points. While sketch-based systems exist, they typically focus on a single human.

Multi-entity interaction (Human-Object-Human, or HOH) is exponentially harder because:

1. Coordination: Agents must synchronize their movements around a shared object.
2. Discrete Events: Occurrences like "grasping" or "releasing" are binary states that continuous neural networks often struggle to time precisely.
3. Guidance Cost: Forcing a Diffusion model to follow a specific path usually requires expensive iterative optimization (posterior guidance) at test time.

Methodology: The Core Trinity

Sketch2Colab solves these issues through three synergistic innovations:

1. Diffusion-to-Flow Distillation

Instead of using a standard Diffusion model, the authors use Rectified Flows. They first train a "Teacher" Diffusion model and then distill its knowledge into a "Student" Flow model. This allows for nearly straight-line transport in the latent space, meaning the model needs far fewer steps to generate high-quality motion.

2. Dual-Space Energy Guidance

To ensure the motion follows the sketch perfectly, the model uses a dual-pathway:

• Latent Anchors: Keeps the motion "natural" by staying near the learned data manifold.
• Raw-Space Energies: Uses differentiable physics and geometric penalties (e.g., "Hand must be 0cm from the handle") to refine the motion.
• The Secret Sauce: A learned Jacobian surrogate ($B_{\rho }$) allows the model to efficiently calculate how a change in 3D joint positions should update the compressed latent code.

Figure 1: The Sketch2Colab pipeline: from storyboard input to distilled flow evolution and final 3D decoding.

3. CTMC Phase Scheduling

Interactions are segmented into phases (e.g., Approach $\to$ Grasp $\to$ Move). Sketch2Colab uses a Continuous-Time Markov Chain (CTMC). This lightweight planner predicts the probability of being in a specific state at any given $t$, effectively "switching on" contact constraints only when the storyboard implies a touch should occur.

Experiments & Results

The model was tested on CORE4D (collaborative object rearrangement) and InterHuman datasets.

SOTA Performance

Sketch2Colab outperformed the baseline COLLAGE (a SOTA LLM-driven model) and Sketch2Anim in nearly every metric:

• Trajectory Accuracy: 3D trajectory error dropped by 40%.
• Interaction Fidelity: Anchor error (how well hands meet handles) improved by 45%.
• Efficiency: It achieves these results with only 30-60 steps, compared to the hundreds often required by Diffusion samplers.

Table 1: Quantitative comparison shows Sketch2Colab dominating in both realism (FID) and control accuracy.

Qualitative Impact

Visual results show that while other models might have "floating" hands or characters walking through tables, Sketch2Colab maintains crisp object contact and respects the timing of the original storyboard sketch.

Figure 2: Examples of HOH interaction: Co-manipulating a table, transport of a box, and specific path following.

Critical Analysis & Conclusion

Takeaway

The paper proves that for high-stakes animation and robotics, visual conditioning (sketches) provides the necessary "spatial bandwidth" that text lacks. By treating motion as a "flow" rather than a "diffusion process," the authors provide a faster and more steerable tool for creators.

Limitations

• Teacher Dependency: The student model's quality is currently capped by the pre-trained diffusion teacher.
• Object Diversity: While it handles a variety of interactions, the system is still largely tied to the categories present in the training data (CORE4D).

Future Outlook

The integration of CTMC planners into generative flows opens the door for more complex agent-environment interactions, potentially extending beyond human motion into autonomous robot fleet coordination in shared spaces.