[CVPR 2025] Fase3D: Breaking the Dependency on Heavy 3D Encoders with Fourier Logic

TL;DR

Fase3D is a breakthrough in 3D Large Multimodal Models (LMMs). It is the first architecture to successfully remove the traditional, heavy 3D visual encoder (like a 3D U-Net) while maintaining SOTA performance. By leveraging Fast Fourier Transforms (FFT) and Space-Filling Curves (SFC), it approximates complex global attention at a fraction of the cost—reducing FLOPs by up to 20x and vision parameters by 10x.

Problem & Motivation: The "Encoder Tax"

In the current 3D LMM landscape (e.g., LL3DA, 3D-LLaVA), the standard recipe involves a "heavy head": a pre-trained geometric encoder that processes millions of points before a single token reaches the LLM.

This approach has three major flaws:

1. Computational Bottleneck: High-resolution 3D scenes require massive memory and compute.
2. Semantic Mismatch: Frozen encoders often output features that aren't perfectly aligned with the LLM's reasoning space.
3. Inflexibility: Unlike 2D pixels, 3D points are unordered. Directly feeding them to a decoder-only LLM without an encoder usually leads to a loss of global structure.

Fase3D's core insight is that global context doesn't require self-attention. By moving to the Frequency Domain, we can capture the "layout" of a scene (low frequencies) much more efficiently than by comparing every point to every other point.

Methodology: The Fourier Tokenizer

Fase3D transforms a messy point cloud into a clean sequence of tokens via a three-step pipeline:

1. Structured Superpoints

Instead of raw points, the model aggregates local geometry into superpoints. This reduces input length by an order of magnitude while preserving local semantic coherence.

2. Spectral Context Enhancement (The Core)

To give these superpoints "global vision" without a heavy encoder, Fase3D uses:

• Serialization: Mapping 3D points to 1D using Hilbert/Z-order curves.
• FFT Mixing: Applying 1D FFT on the serialized sequence. By filtering for low-frequency components, the model captures the relationship between distant objects (e.g., the sink's position relative to the toilet) with $O(M\log M)$ complexity.

Figure 1: The Fase3D pipeline featuring the FFT-based context enhancer and Graph-based token merging.

3. Fourier-augmented LoRA

The innovation continues even inside the LLM. Instead of standard LoRA, the authors use global frequency-aware adapters. These modules filter the hidden states in the spectral domain, ensuring the LLM remains "aware" of the 3D frequency distribution throughout the reasoning process.

Experiments: Superior Efficiency

The results are striking. Fase3D matches the performance of the most popular 3D LMM (3D-LLaVA) on benchmarks like ScanQA and SQA3D, but the efficiency gap is enormous.

Figure 2: Qualitative benchmark on ScanQA. Fase3D shows higher semantic accuracy in identifying specific object counts and colors compared to heavier baselines.

Critical Insight: Why Does It Work?

The success of Fase3D suggests that Inductive Bias matters more than Parameter Count. By explicitly teaching the model how to look at frequencies and spatial locality (via SFCs), it learns 3D geometry from scratch more effectively than a generic Transformer learns it via brute-force self-attention.

Conclusion & Limitations

Fase3D is a significant "monolithic" step forward. It treats 3D data as just another sequence, provided you use the right mathematical tools (FFT) to manage that sequence.

Limitations: Like other serialization methods (e.g., PTv3), it may struggle with extremely non-Euclidean relations in highly cluttered scenes where SFCs might create "jumps" that are hard for the model to reconcile.

Final takeaway: The "Encoder-Free" movement has officially arrived in 3D. Expect future SOTA models to ditch the CLIP/U-Net backbones for these more agile, spectral architectures.