[CVPR 2026] SpaceSense-Bench: Scaling Spacecraft Perception with Multi-Modal Diversity

TL;DR

Perceiving a non-cooperative satellite in the harsh vacuum of space is a high-stakes challenge where "memorizing" a target's look isn't enough. SpaceSense-Bench introduces a massive multi-modal dataset featuring 136 satellite models, time-synchronized RGB/Depth/LiDAR data, and dense 7-class part-level semantics. The study proves that massive geometric diversity is the key to zero-shot generalization for autonomous on-orbit servicing.

Problem & Motivation: The "Memorization" Trap

Current autonomous space navigation relies heavily on deep learning, but models are often trained on datasets like SPEED+, which only feature a handful of satellite models. In the real world, a chaser spacecraft might encounter thousands of different geometries.

Existing benchmarks fail in three areas:

1. Geometric Overfitting: Models learn specific textures (e.g., the "Tango" satellite) instead of functional parts (e.g., what a "solar panel" looks like generally).
2. Sensor Monoculture: Most datasets are RGB-only, failing under orbital "High Dynamic Range" conditions where sun-glint or total eclipse makes cameras useless.
3. Annotation Scarcity: Lack of millimeter-precision depth and point-level labels prevents the development of robust fusion-based 3D perception.

Fig 1: The SpaceSense-Bench pipeline. From 3D asset decomposition in Blender to high-fidelity UE5 rendering.

Methodology: The Geometry-First Approach

The authors bypassed manual labeling—an impossible task for 90k frames—by performing Instance-level Part Decomposition directly on 3D meshes. By assigning "Stencil IDs" to components like thrusters, antennas, and docking rings, the Unreal Engine 5 renderer generates pixel-perfect ground truth automatically.

Key Technical Specs:

• Scale: 136 models (from 0.27m CubeSats to the 112m ISS).
• Modality: Synchronized 1024x1024 RGB, Depth maps, and 256-beam LiDAR.
• Taxonomy: 7 functional classes defining the "anatomy" of a spacecraft.
• Environment: Ray-traced reflections, Earth Albedo, and star-catalog backdrops.

Fig 2: Multi-modal ground truth (RGB, 6D Pose, Semantic Mask, LiDAR Point Cloud).

Experiments & Results: The Power of Scale

The researchers benchmarked several SOTA architectures (SegFormer, Mask2Former, YOLO26, PMFNet) using a strict Zero-Shot Protocol: the test satellites were never seen during training.

1. Scaling Works (But Doesn't Saturate)

The most striking finding was the scaling law of spacecraft geometry. As the number of training satellites increased from 9 to 117, the mean IoU (mIoU) jumped by 73%. This suggests we haven't reached the limit of how much data diversity can help "teach" a model the concept of a satellite.

Fig 3: Performance gains as training data diversity increases.

2. The "Small Part" Bottleneck

While models were great at spotting "Solar Panels" (88% IoU), they struggled with "Omni-antennas" and "Thrusters" (often <10% IoU). These small parts are mission-critical for docking but occupy less than 1% of the total pixels, creating a massive long-tail distribution challenge.

Critical Analysis & Future Outlook

SpaceSense-Bench effectively shifts the focus from "how to design a better CNN/Transformer" to "how to build a better world model for space."

• Takeaway: Diversity in geometry is more potent than diversity in pixels. By forcing models to see 117 different "bodies," the researchers successfully decoupled texture from structure.
• Limitations: While UE5 provides high fidelity, "Sim-to-Real" remains a hurdle. Performance on real orbital footage (with sensors noise and cosmic radiation artifacts) is the next frontier.
• Future Work: The authors propose using 3D generative AI to automate the creation of thousands of new satellite designs, potentially solving the small-component recognition problem through sheer volume.

For researchers in autonomous navigation and robotic servicing, SpaceSense-Bench provides the most rigorous testbed yet for testing if your model can truly survive the "unknown" of deep space.