[Apple Research] LITO: Breakthrough in 3D Light Field Tokenization for Realistic View-Dependent Appearance

TL;DR

LITO (Light Field Tokenization) represents a paradigm shift in 3D generation. Unlike traditional models that only care about shape and static color, LITO learns a 3D latent space that mimics the Surface Light Field. By encoding sparse RGB-D samples into latent tokens, it can reconstruct and generate 3D objects with stunning view-dependent effects like specular highlights and Fresnel reflections, outperforming previous SOTA models like TRELLIS in visual fidelity.

The Blind Spot of Current 3D Gen: Static Appearance

Most current 3D generative pipelines (e.g., TripoSR, TRELLIS) treat objects as "matte" surfaces. They either ignore appearance entirely or average out the color across multiple views, resulting in a view-independent diffuse look.

In reality, the way an object looks depends heavily on your viewing angle—think of the glint on a metallic helmet or the sheen on a polished apple. The authors of LITO argue that to capture the "richness" of the real world, we need to model the 5D Surface Light Field $\ell (\mathbf{x},\hat{\mathbf{d}})$, which defines the radiance at every surface point $\mathbf{x}$ in every viewing direction $\hat{\mathbf{d}}$.

Methodology: Tokenizing the 5D Space

The core innovation of LITO is its ability to handle the massive amounts of data required to describe a light field.

1. 3D Patchification

Processing 1 million input points is computationally prohibitive for standard Transformers. LITO introduces 3D Patchification:

• It selects $k$ query points and groups the million input samples using K-Nearest Neighbors (KNN).
• This mimics the "patch" structure in 2D Vision Transformers (ViT) but adapts it to the irregular, sparse topology of 3D surfaces.

2. The Hybrid Decoder

To turn these latents back into 3D, LITO uses two specialized paths:

• Geometry Path: A flow-matching velocity decoder that treats the surface as a probability density function.
• Appearance Path: A Gaussian decoder that outputs 3D Gaussians equipped with Spherical Harmonics (SH) of degree 3. This SH degree is crucial—it's what allows the model to "store" different colors for different viewing angles within the same spatial point.

Figure 1: LITO Encoder-Decoder Pipeline. The pink block represents the Perceiver IO encoder condensing the surface light field.

Experimental Performance: Beyond "Matte" Objects

The results show a massive jump in rendering quality. On the Toys4k dataset, LITO achieved a PSNR of 34.16 dB, significantly higher than TRELLIS's 31.12 dB.

View-Dependent Fidelity

The most striking evidence is in the ablation of Spherical Harmonics (SH). As shown in the paper's appendix, increasing the SH degree from 0 (diffuse) to 3 (specular) allows the model to separate "baked" lighting from the true material properties.

Figure 2: Single-image to 3D results. Note the specular highlights on the metallic fox and the clean geometry compared to previous baselines.

Critical Analysis & Future Outlook

Strengths:

• Single-Stage Generation: Unlike TRELLIS, LITO doesn't require a separate model to predict coarse occupancy; the latent itself is expressive enough to guide the geometry directly.
• Input Fidelity: By aligning the world coordinate system with the input view during training, the model achieves much lower FID scores (6.22 vs 12.84), meaning the generated 3D object "looks" much more like the 2D input image.

Limitations:

• High-Frequency Specularity: Since it relies on SH degree 3, extremely sharp mirrors or perfectly transparent glass are still difficult to represent.
• Compute Intensity: Training on 500k objects with 150 views each for 20 days on 128 H100s highlights the massive data scale required for this fidelity.

Conclusion

LITO proves that the future of 3D vision is not just about "shapes," but about the interaction between light and geometry. By tokenizing the light field, LITO bridges the gap between 2D photographic realism and 3D structural consistency.

Senior Editor's Note: LITO represents Apple's strong entry into the 3D foundation model space, emphasizing the PBR (Physically Based Rendering) qualities that are essential for AR/VR applications.