Turning Every Smartphone into a Periscope: Real-Time NLOS Imaging with Consumer LiDAR

TL;DR

Researchers from MIT and Dartmouth have unlocked the ability to see around corners using the low-cost LiDAR sensors already found in your smartphone or vacuum robot. By introducing a Motion-Induced Aperture Sampling (MAS) model and using Particle Filtering, they overcome the hardware limitations of consumer sensors—low power and low resolution—to achieve real-time 3D tracking and reconstruction of hidden objects.

The Problem: The High Cost of Seeing the Unseen

For years, Non-Line-of-Sight (NLOS) imaging was a "lab-only" miracle. It required ultra-fast femtosecond lasers and sensitive detectors costing tens of thousands of dollars. Consumer LiDARs, like the ones in an iPhone or a $100 STMicroelectronics sensor, are seemingly "garbage" for this task:

• Low SNR: Eye-safety regulations limit laser power.
• Low Resolution: Instead of megapixels, you get ~100 pixels.
• Motion Blur: Handheld devices are never perfectly still.

Past algorithms assumed a static, high-resolution "virtual aperture" on a wall. In the real world—where both the camera and the hidden object might be moving—these methods fail.

Methodology: The MAS Model & Multi-Frame Fusion

The core innovation is the Motion-Induced Aperture Sampling (MAS) model. The authors realized that instead of a bug, motion is a feature.

1. Light-Cone Transform (LCT) Intuition

The LCT is a mathematical trick that re-maps time and space so that the relationship between a hidden object and its reflection on a wall behaves like a 3D convolution. This makes the math of "un-blurring" the reflection much simpler.

2. Decomposing Motion

The MAS model separates three variables:

• Canonical STIR: The "signature" of the object's shape.
• Object Shift ($\Delta$t): Where the object is moving.
• Camera Sampling: Which part of the "virtual mirror" the device is looking at.

Figure: The MAS model unifies object shape, motion, and camera pose into a single measurement framework.

3. Particle Filtering for Tracking

Since a single frame from a consumer LiDAR is too noisy to "see" the object, the system uses a Particle Filter. It maintains 1,000 "guesses" (particles) of where the object is. As new frames come in at 30Hz, the system keeps the guesses that match the measured light and throws away the rest. This naturally handles uncertainty and noise.

Results: Better than many Lab-Grade systems

The researchers tested three main applications:

1. 3D Reconstruction: Building a point cloud of a hidden mannequin.
2. Object Tracking: Tracking a person's hands or moving objects around a corner in real-time.
3. Camera Localization: Using a hidden object as a "landmark" to navigate in a room with featureless white walls—where traditional GPS and visual SLAM fail.

Figure: Quantitative tracking shows that even with a tiny 100-pixel sensor, the error stays within a few centimeters.

Why It Matters

This isn't just a cool physics trick. This "democratization" of NLOS means:

• Warehouse Robots can see a forklift coming around a blind corner before a collision occurs.
• AR/VR Headsets can track a user's body pose even when limbs are occluded.
• Search and Rescue teams can use off-the-shelf phones to locate victims in collapsed buildings.

Critical Analysis & Limitations

While impressive, the MAS model currently assumes the object is retroreflective (like a bike reflector or traffic cone) for the best results. While they demonstrated success with diffuse (normal) objects, the SNR drops significantly. Future work involves using machine learning to learn "score functions" that can handle complex materials like cloth or skin more effectively.

Future Outlook

The shift from "expensive lab setups" to "plug-and-play" is the final hurdle for NLOS tech. This paper proves that the hardware is already in our pockets; we just needed the right math to unlock it.

Reference: Somasundaram et al., "Imaging Hidden Objects with Consumer LiDAR via Motion Induced Sampling", 2026.