[CVPR 2025 Submission] Robust Trajectory Prediction: Solving the Skeleton Occlusion Dilemma with Self-Supervision

TL;DR

Predicting where a human will walk is much easier if you can see their body language (skeleton). However, in the real world, "seeing" is obstructed by obstacles. This paper introduces a two-stage framework that uses Self-Supervised Masked Autoencoding to learn robust skeleton representations. The result? A model that doesn't just "survive" missing data but actively uses partial body cues to improve prediction accuracy.

The Problem: The Accuracy-Robustness Trade-off

Most SOTA trajectory predictors use skeletal data to infer "intent" (e.g., a person leaning before making a turn). But there's a catch: Occlusion. When joints are missing, models usually fail.

• Standard approach: Train on clean data $\to$ Breaks during real-world occlusion.
• Corruption training: Train on noisy data $\to$ Robustness increases, but accuracy on clean data drops because the model learns to "distrust" the skeleton.

Methodology: Mastery through Masking

The authors argue that robustness should be handled at the representation level, not the output level. They propose a two-stage pipeline:

1. Self-Supervised Pretraining (The "Healer")

Using an ST-GCN (Spatio-Temporal Graph Convolutional Network), they mask out up to 50% of the joints. The model is forced to reconstruct the full skeleton. This teaches the encoder the underlying "physics" and "grammar" of human motion.

Figure 1: The two-stage framework: (a) Pretraining the encoder via reconstruction, and (b/c) Integrating the frozen robust encoder into the trajectory predictor.

2. Downstream Integration

The pretrained encoder is frozen and plugged into a Social-TransMotion architecture. Instead of feeding raw (and potentially broken) joint coordinates, the system feeds stable, high-dimensional "latent tokens."

Experimental Evidence

The model was tested on the JTA Dataset, a massive synthetic environment with complex crowd behaviors.

Quantitative SOTA Comparison

The proposed method consistently outperforms baselines in "Clean" to "Moderate" missingness. While other models' performance tanks as joints disappear, this method's degradation curve is significantly flatter.

The "Skeleton Reliance" Acid Test

One of the most insightful parts of the paper is the ablation where they disable the skeleton. The authors' model showed a much larger performance drop when the skeleton was removed compared to baselines. Why is this good? It proves that the model has learned to rely more on body language for intent, but it is smart enough to extract that intent even when half the joints are obscured.

Figure 2: In curved trajectory cases, the proposed method correctly anticipates the turn even when skeleton data is masked, whereas baselines predict a straight path.

Critical Insight: High-Level Latents vs. Coordinate Completion

A common fix for missing data is "Coordinate Completion" (filling in the blanks then predicting). The authors show this is sub-optimal because reconstruction errors propagate. By using latent representations (vectors) instead of coordinates, the downstream predictor is shielded from the "jitter" of imperfect reconstruction.

Conclusion

This research shifts the focus from "how to fix broken data" to "how to learn representations that aren't broken by missing data." For autonomous robots navigating busy malls or surveillance systems tracking intent in crowds, this provides a more stable, hardware-ready path forward.

Key Limitation: At extreme missingness (>70%), the problem becomes "ill-posed"—not even the best latent representation can conjure intent from a single visible ankle joint.