[CVPR 2025] Stepping VLMs onto the Court: Scaling Spatial Intelligence via Sports Geometry
Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports
CourtSI is the first large-scale spatial intelligence dataset and benchmark specifically tailored for high-intensity sports scenarios (badminton, tennis, and table tennis). It contains over 1M QA pairs derived from a semi-automatic 3D reconstruction engine, achieving a significant 23.5% accuracy boost for models like Qwen3-VL-8B after fine-tuning.
TL;DR
Researchers have introduced CourtSI, the first large-scale (1M+ samples) spatial intelligence dataset dedicated to sports. By using the fixed metric dimensions of tennis and badminton courts as "ground truth anchors," they've built a pipeline to teach Vision-Language Models (VLMs) how to perceive 3D distances, human poses, and ball trajectories with centimeter-level precision.
Evolution of Spatial Intelligence: From Furniture to Fair Play
Most "spatial" benchmarks for AI (like SpatialVLM or VSI) focus on static, indoor environments—asking if a mug is next to a laptop. However, the real test of spatial intelligence lies in dynamic, high-speed human interaction.
The authors argue that sports provide the perfect "natural laboratory":
- Fixed Geometry: Every tennis court globally has the same dimensions, providing a built-in "ruler" for the AI.
- Complex Motion: Human bodies undergo non-rigid deformation (diving, jumping), challenging the model's understanding of anatomy in 3D space.
- Metric Sensitivity: In sports, "out by an inch" is a world of difference. Models must move beyond "left/right" to "2.54 meters."
The "Data Engine": Turning 2D Broadcasts into 3D Truth
The core technical contribution is a semi-automatic reconstruction pipeline. Since monocular depth estimation (like DepthAnything) often fails to provide metric-scale accuracy for small objects like shuttlecocks, the team used a human-in-the-loop approach:
- Court Anchoring: Annotators mark court corners; a PnP solver calculates the camera's precise position in 3D space.
- Perspective-Aware Scaling: To fix the depth of players and balls, they used a "projection line" tool, allowing humans to easily mark where an airborne ball's shadow would fall on the ground, mathing out the exact 3D coordinate.
Figure: The CourtSI Data Engine pipeline, translating monocular broadcast frames into world-grounded 3D states.
Benchmarking the Giants: Humans vs. GPT-5.2
The team tested 25 models, including GPT-5.2, Gemini-3-Pro, and Qwen3-VL.
- The Verdict: Even the best proprietary models struggle with perspective ambiguity—where objects look close on screen but are far apart on the court.
- Metric Failure: Most general VLMs were found to be "metric-blind," failing tasks that required calculating absolute distances between players.
Figure: Performance comparison across various VLM families. Note the massive 23.5% jump after fine-tuning on CourtSI.
Beyond Counting: Spatial-Aware Commentary
One of the most exciting applications shown is Spatial-aware Commentary. Traditional AI commentary says, "The player hits the ball." A CourtSI-enhanced model says, "The player lunges 2.1 meters towards the net to save a low-flying shuttlecock."
The user studies showed that fine-tuning on CourtSI significantly improves a model's "spatial awareness" without losing the natural flow of language—a key step toward automated, professional-grade sports broadcasting.
Conclusion & Future Outlook
CourtSI proves that geometry is a powerful supervisor. By leveraging the structured nature of the physical world (courts, fields, tracks), we can scale the spatial intelligence of VLMs without requiring expensive LiDAR or multi-view camera setups. The next frontier? Moving this from net sports to "in-the-wild" human activities where the "ruler" isn't as clearly defined.
For more details, check out the CourtSI Project Page.