Turning ToF into X-Ray Vision: NLOS Imaging with $100 Consumer LiDAR

TL;DR

Non-Line-of-Sight (NLOS) imaging—the ability to see around corners—has long been the "holy grail" of computational photography, but it usually requires $50k laser setups and laboratory conditions. This paper from MIT and Dartmouth changes the game by using consumer-grade LiDAR (like those in iPhones) to track hidden objects and localize cameras by treating motion as a feature, not a bug.

Background: The Price of Seeing Around Corners

Most LiDAR systems today only "see" what is directly in front of them (Line-of-Sight). However, light bounces off surfaces like walls, hits hidden objects, and returns. Research-grade systems capture these tiny, late-arriving photons with extreme precision.

Consumer LiDARs are too weak and noisy for this—until now. The authors argue that while a single frame from a phone is useless, a burst of frames combined with the natural motion of a handheld device can synthesize a "virtual aperture" large enough to reconstruct the hidden world.

The Problem: The SNR and Resolution Gap

Why can't your phone see around corners out of the box?

1. Low SNR: To protect your eyes, phone lasers are weak. The signal from a hidden object is often buried under noise.
2. Sparse Sampling: Handheld LiDARs often have only ~100 pixels, providing a very low-resolution "view" of the wall.
3. Motion Hyper-Complexity: In a real-world scenario, the camera is moving (handshake) and the object might be moving. This creates a mess of spatio-temporal shifts.

Methodology: Motion-Induced Aperture Sampling (MAS)

The core innovation is the MAS model. It utilizes the Light-Cone Transform (LCT), which maps the physics of light propagation into a convolutional space.

In this transformed space, the relationship between the object's shape and the measurement becomes a simple 3D convolution. Crucially, the authors proved that an object's translation in 3D space corresponds to a linear shift in the measurement space.

Figure: The MAS model decouples object shape (time-independent) from motion (time-dependent), allowing efficient indexing for real-time processing.

The Particle Filter "Brains"

To solve the tracking problem, the authors use a Particle Filter. Instead of trying to calculate the exact position (which is noisy), they maintain 1,000 "particles" (guesses).

• Propagation: Move particles based on a motion prior (where the object was last seen).
• Evaluation: "Render" what the LiDAR should see if the object were at that particle's location and compare it to real data.
• Resampling: Keep the guesses that match the data; kill the ones that don't.

Experimental Battle-Record

The authors tested their approach on several high-stakes applications:

1. 3D Tracking: Tracking people or objects around corners in real-time.
2. Diffuse Imaging: Even without specialized retroreflective "stickers," the system could "see" diffuse (matte) objects, though with slightly more noise.
3. NLOS Localization: Using a hidden object as a "landmark" to tell the camera where it is—perfect for navigating white, featureless hallways where standard CV algorithms fail.

Figure: (Left) The NLOS concept. (Right) Success in 3D reconstruction and tracking using smartphone-grade hardware.

Critical Insight: Motion is a "Virtual Lens"

The most profound takeaway is the concept of Opportunistic Blur. Usually, blur is the enemy of sharp images. But in sparse LiDAR, the multi-bounce "blur" actually acts as a form of optical anti-aliasing. By moving the camera, we aren't just getting more frames; we are building a "synthetic aperture" that is much larger than the physical sensor.

Conclusion & Future Work

The paper successfully demonstrates "Plug-and-Play" NLOS. While it currently assumes some knowledge (like the object's general shape), future iterations could move toward a Full NLOS-SLAM, where a robot maps both the visible and hidden rooms simultaneously as it moves.

Limitations: The model struggles with complex multi-bounce occlusions (objects hiding behind other hidden objects) and currently performs best with retroreflective materials. However, as SPAD (Single-Photon Avalanche Diode) sensors continue to improve in mobile chips, your phone’s "X-ray vision" is no longer a matter of if, but when.