[CVPR 2024] TeamHOI: Breaking the Scale Barrier in Cooperative Human-Object Interaction
TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size
TeamHOI is a novel framework that learns a single, decentralized Transformer-based policy for cooperative human-object interactions (HOI) across any team size (2 to 8+ agents). It achieves SOTA performance in complex tasks like collective table carrying, maintaining high success rates (97.5%+) and physical realism without needing multi-agent reference data.
Executive Summary
TL;DR: TeamHOI introduces a unified, decentralized policy capable of coordinating an arbitrary number of humanoid agents to perform complex cooperative tasks, such as carrying large tables. By combining a Transformer-based architecture with a novel Masked Adversarial Motion Prior (AMP), the framework enables agents to learn sophisticated teamwork from single-human motion data, scaling seamlessly from 2 to 16 agents without retraining.
Background Positioning: This work represents a significant leap from fixed-size multi-agent controllers toward scalable, generalizable embodied AI. It moves the field beyond implicit coordination (relying on object physics) to explicit, token-based perception of teammates.
Problem & Motivation: The Coordination Bottleneck
Why is it so hard to make virtual humans carry a table together? Current SOTA methods face two "walls":
- The Scalability Wall: Most policies use Multi-Layer Perceptrons (MLPs) with fixed input dimensions. If you train a model for 2 agents, it literally cannot "see" a 3rd agent.
- The Data Wall: We have plenty of data for a person walking alone, but almost no high-fidelity MoCap data for eight people carrying a heavy rectangular table.
Previous attempts like CooHOI tried to solve this by letting agents "feel" each other through the object's movement. However, this is like trying to coordinate a group dance while everyone is blindfolded—it’s unstable and fails as the group grows.
Methodology: Scalable Attention and Masked Priors
1. Teammate Tokens & Transformer Policy
Instead of a fixed vector, TeamHOI treats every agent as a token. The "observing" agent uses a Transformer backbone to attend to its own state (Self-Attention) and then cross-attend to a variable list of Teammate Tokens. This allows the policy to remain identical regardless of whether there are 2 or 20 teammates.
2. Masked AMP: The "Part-Wise" Intuition
To solve the data scarcity issue, the authors insightfully realized that while we don't have "table-carrying" data, we have "walking" data.
- They trained a Masked Discriminator that ignores the hands and forearms.
- For the rest of the body (legs, torso), the model must look like a real human walking.
- For the hands, the model ignores the "natural walking" style and follows Task Rewards to grasp and lift the table.
3. Principal-Axes Coverage Reward
To ensure the table doesn't flip, the agents must distribute themselves around the object's center of mass. The authors designed a reward based on the object's Principal Axes. This encourages agents to form stable "support polygons," a physical intuition that leads to much higher stability than simple angular spacing.
Experiments & Results: Robustness at Scale
TeamHOI was tested on a table-carrying task with varying shapes (round, square, rectangular) and masses.
- Superior Scaling: While the baseline (CooHOI*) collapsed when the team size reached 8 agents (10.1% success), TeamHOI maintained a 97.5% success rate.
- Heavy Load Handling: When the table mass was increased by 5x, smaller teams failed, but the 8-agent TeamHOI policy successfully enabled the agents to combine their forces, achieving an 81.1% success rate.
- Zero-Shot Generalization: Remarkably, a policy trained on up to 8 agents was able to control 16 agents carrying a massive table without any additional fine-tuning.
Critical Analysis & Conclusion
Takeaway
TeamHOI proves that decentralized coordination is sufficient for complex physical tasks if the agents have a flexible way to "attend" to their peers. The Masked AMP strategy is a brilliant workaround for the "missing data" problem in robotics and animation, suggesting we can build complex multi-agent behaviors by essentially "kit-bashing" single-agent motion priors.
Limitations & Future Work
While impressive, the interactions are still somewhat simplified (the agents have "ball hands" without fingers). Future work could integrate more complex grasping and perhaps heterogeneous teams (e.g., humans and robots working together).
In short: TeamHOI provides the skeletal architecture for the next generation of multi-agent interactive environments, from video games to collaborative industrial robotics.