How can prioritized motion retargeting transfer human motion to humanoid robots?

Why weighted objectives cannot guarantee priority

Motion retargeting converts human motion-capture data into joint trajectories a robot can execute. The human and the robot differ in size, joint limits and proportions, so the transfer cannot copy joint angles directly; it must reconstruct the defining features of the source motion on a new body. For single-character motion, matching keypoints and respecting joint limits is often enough. Once a person carries a box or coordinates with another person, the motion also contains contacts and spatial relationships that must survive the transfer, and errors there produce physically infeasible references that make downstream learning and control harder.

Most retargeting pipelines express these requirements as a weighted sum of loss terms: keypoint matching, foot placement, contact preservation, joint limits. Weights encode only relative preference. Improving a lower-weighted term can still degrade a higher-weighted one, so a large weight does not guarantee that foot contacts stay intact. The paper's argument is that this is a structural property of the weighted-sum formulation, not a tuning failure, and that it becomes more damaging as extra characters and objects add more contacts to preserve.

Projecting lower-priority updates into the nullspace

PriMoRetarget orders objectives from highest to lowest priority and enforces the order locally at each optimization step. Consider a higher-priority error vector, for example the 3D position errors of several corresponding bodies, and a lower-priority gradient. The method linearizes the higher-priority error and asks for an update that leaves that error unchanged to first order, which means the update must lie in the nullspace of the higher-priority Jacobian. A nullspace projector removes from the lower-priority gradient any component that would disturb the higher-priority error.

Preserving the full error vector, rather than only its scalar loss, matters: it prevents one error component from compensating for another. The paper contrasts this with a weighted sum, which converges to a compromise between competing objectives, whereas the projected update is restricted to directions that keep the higher-priority objective at its current value. In practice the Jacobian can be rank deficient, so a damped projector with a small damping factor is used. The hierarchy itself is specified manually; the paper presents it as a general mechanism rather than a fixed ordering.

Per-frame projection makes long sequences tractable

Applying the hierarchy to a whole motion sequence naively is expensive, because a dense projection matrix grows quadratically with the number of frames. Many retargeting objectives are separable across frames: the error at each frame depends only on that frame's motion parameters. PriMoRetarget exploits this by building the hierarchy from per-frame objectives, which makes the full Jacobian block diagonal in time and decomposes the projection into independent per-frame systems solved as a batched operation. The paper reports the complexity dropping from O(T³m³) to O(Tm³) for T frames and m higher-priority error components per frame.

Two design choices support this. Temporal smoothness and other regularizers do not fit a strict hierarchy: given a high priority they can consume the nullspace, and given a low priority they are heavily restricted after projection. The paper therefore treats them as free objectives whose gradients are added directly to the accumulated projected gradient. Separately, because nullspace projection makes it hard to traverse large distances in parameter space, optimization begins with a short warmup using a uniform weighted sum to move the motion closer to a suitable solution before the hierarchy is applied.

Contact preservation separates prioritized from weighted sums

The evaluation covers four datasets: single-character motion (LAFAN1), character–object manipulation (OMOMO), character–character interaction (Inter-X), and an in-house multi-character-plus-object set. Kinematic quality is measured by penetration duration and depth, foot sliding velocity, foot floating height, jitter from the third derivative of body positions, and contact preservation, the fraction of source hand contacts reproduced after retargeting. Baselines are PHC, GMR and OmniRetarget; for multi-character motion the baselines are applied independently per character.

The clearest split appears where an additional contact-preservation priority level exists. On OMOMO, PriMoRetarget reaches 96.9% contact preservation versus 67.2% for uniform weighting and 88.5% for a 10× weight separation, while foot sliding is 3.7 cm/s versus 9.4 and 3.4. On the in-house multi-character-plus-object set it reaches 92.4% contact preservation versus 88.8% and 71.7%, with jitter of 67.5 versus 52.9 and 59.1 m/s³. For single-character locomotion, where conflicts are limited and only two priority levels are present, the variants perform similarly. The authors read this pattern as evidence that prioritization matters most when interactions add objectives that compete.

Better references train more successful tracking policies

Kinematic metrics matter because retargeted motion becomes a reference for reinforcement learning. Following DeepMimic, the paper trains a separate PPO policy per motion on a simulated Unitree G1 with fixed hyperparameters across retargeting methods. Policies observe target poses and output joint PD targets; object-interaction motions additionally observe object position and target trajectory, and multi-robot motions use a single policy controlling all robots. Across 50 motions, success requires completing the reference without any robot or object deviating by more than 0.5 m, and tracking errors are computed over the full motion with early termination disabled.

Policies trained on PriMoRetarget motions achieve the highest success rates and lowest body tracking errors across all datasets, with the benefit more evident for interaction-rich motions. The authors attribute this to more consistent references when contacts and coordination are preserved. They also note that collaborative object manipulation in the in-house dataset remains challenging, which marks the limit of what better retargeting alone delivers: jointly controlling multiple interacting robots and objects is still hard.

Warmup and free objectives carry measurable weight

The ablations show which components do the work. Removing the weighted-sum warmup while keeping the number of prioritized steps fixed changes little when initialization is already close to a suitable solution, but degrades substantially when large corrections are needed. On LAFAN1, penetration duration rises from 12.2% to 90.1% and foot sliding from 3.5 to 10.2 cm/s; on the in-house set, contact preservation falls from 92.4% to 78.5%.

Placing the free objectives inside the final priority level instead of leaving them free also costs quality: projection through higher-priority objectives limits their influence, particularly on temporal smoothness, raising jitter across settings. On Inter-X, jitter rises from 103.9 to 155.0 m/s³. Contact preservation improves slightly in some cases, but the paper calls those gains marginal. Per-frame projection, meanwhile, is a computational rather than a quality choice: on an NVIDIA RTX 5090 the dense formulation's cost grows rapidly with sequence length while the batched per-frame version stays nearly constant over the evaluated range, with similar behavior reported for LAFAN1 motions exceeding 5000 frames.

What the hierarchy preserves and what it cannot fix

The method is not tied to one robot. The same prioritized formulation is demonstrated on a Unitree G1 and a DEEP Robotics DR02 jointly carrying a box, a Unitree H1-2 carrying a box down a ramp, and the same OMOMO motion transferred to a Booster T1 and a Unitree G1. When the manipulated object is enlarged by 1.5×, the retargeted hand contacts adapt to the new geometry to preserve the high-priority contact objective rather than strictly following the source motion. Policies are also deployed on real robots for box manipulation.

The limits follow from what the hierarchy optimizes. PriMoRetarget primarily preserves the structure of the source motion, so artifacts already in the capture, such as jitter, interpenetration or inconsistencies in the captured environment, can propagate to the retargeted motion. The paper notes that higher penetration and foot floating in some settings may reflect artifacts present in the source. Object trajectories largely follow the original source motion, which limits adaptation when that trajectory suits the target embodiment poorly. The priority order is also specified manually; automatically determining or adapting it, incorporating learned motion priors, and jointly optimizing robot and object trajectories are left as future work.

Prioritization pays off when interactions add contacts

The decision-relevant comparison in this paper is not prioritized versus weighted optimization in general, but where the two diverge. With only two priority levels and limited objective conflict, as in single-character locomotion, a tuned weighted sum is close to the prioritized result. Once an interaction adds a contact-preservation level on top of foot placement and body pose, the weighted variants trade away the higher-priority property: uniform weighting drops OMOMO contact preservation to 67.2% and a 10× separation reaches only 88.5%, against 96.9% for the hierarchy. The paper's own framing is that the hierarchy is a general mechanism for enforcing relative importance, with the ordering supplied by the user. For a practitioner, that makes the priority list, not the weight vector, the object to get right, and makes interaction-rich motion the setting where that choice changes the outcome most.

Sources used in this answer