BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Clouds

[CVPR 2024] BuildAnyPoint: Shifting the Paradigm of 3D Building Reconstruction via Generative Priors

BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Clouds

Summary

Problem

Method

Results

Takeaways

Abstract

BuildAnyPoint is a novel generative framework for structured 3D building reconstruction from diverse, unstructured point clouds (e.g., LiDAR, SfM). It introduces a Loosely Cascaded Diffusion Transformer (Loca-DiT) that achieves SOTA performance by decoupling geometric distribution recovery from autoregressive mesh generation.

TL;DR

BuildAnyPoint solves a long-standing challenge in urban modeling: reconstructing clean, artist-ready 3D meshes from "messy" real-world sensor data (Airborne LiDAR, SfM). By introducing a Loosely Cascaded Diffusion Transformer (Loca-DiT), it first cleans and completes the point cloud distribution using diffusion, then "writes" the mesh structure using an autoregressive transformer. The result? Lower complexity (low-poly), higher fidelity, and unprecedented robustness to noise.

1. The "Dirty Data" Problem in Urban Reconstruction

Reconstructing 3D buildings isn't new, but doing it from any source is. Traditional methods are usually "picky":

Optimization-based (e.g., City3D): Rely on clean plane detection; they crumble under noisy SfM data.
Grammar-based (e.g., ArcPro): Force buildings into rigid templates, failing on complex or slanted roofs.
End-to-End Learning (e.g., Point2Building): Try to go straight from points to mesh, often resulting in "visual chaos" because the model overfits to the input noise.

The fundamental insight of BuildAnyPoint is that we shouldn't go directly to a mesh. Instead, we should probabilistically recover the ideal distribution of the building first.

2. Methodology: The Loca-DiT Architecture

The framework operates on a "Loosely Cascaded" principle, bridging the gap between unstructured points and structured tokens through three latent transitions.

Overall Architecture

Step A: Hierarchical Latent Diffusion

The model first encodes the noisy input into a Dense Latent Grid ( $G_{d}$ ). This allows the model to "reason" about empty space and missing structures. It then refines this into a Sparse Latent Grid ( $G_{s}$ ) using a Diffusion Transformer. This stage essentially acts as a "hallucination machine" that fills in the gaps of a sparse LiDAR scan to create a dense, uniform point cloud.

Step B: Autoregressive Mesh Generation

Once we have a clean geometric prior ( $P_{o u t}$ ), it is tokenized ( $T_{P}$ ) and fed as a prefix to a decoder-only Transformer (based on MeshAnything V2). The Transformer then predicts the mesh sequence ( $T_{M}$ )—defining vertices and faces in a topologically consistent manner.

3. Results: Generalization is the Killer Feature

The most impressive part of BuildAnyPoint is its zero failure rate (0% FR). While other models fail to produce any valid geometry when the input is too sparse (up to 16% failure for City3D), BuildAnyPoint remains stable.

Qualitative Comparison

Key Metrics:

Surface Accuracy: Chamfer Distance (CD) improved by ~16% compared to the previous best learning-based method.
Simplicity: It generates "artist-like" low-poly meshes, reducing vertex counts significantly compared to traditional solvers.
Completion: In building point cloud completion benchmarks, it outperformed 8 dedicated baselines (like PoinTr and SnowflakeNet), particularly in Uniformity, where it scored an order of magnitude better (0.04 vs 0.25+).

4. Why it Works: The Power of Intermediate Priors

The ablation studies prove that the "cascaded" design is the secret sauce. Removing the coarse latent grid ( $G_{d}$ ) leads to chaotic outputs, while removing the fine grid ( $G_{s}$ ) creates "double-surface" artifacts. By separating "what the shape is" (Diffusion) from "how the mesh is structured" (Transformer), the model avoids the modality gap that plagues end-to-end systems.

Ablation Results

5. Conclusion & Future Outlook

BuildAnyPoint sets a new SOTA by treating 3D reconstruction as a sequence of probabilistic refinements. While it currently struggles with extremely intricate architectural details (due to dataset biases), its modular nature means it can easily be upgraded as more powerful Diffusion or Transformer backbones emerge.

For researchers in Digital Twins and Urban Planning, this work provides a robust blueprint for converting heterogeneous, low-quality sensor data into structured, actionable 3D assets.

Primary Citation: Hua, T., Gong, H., et al. "BuildAnyPoint: 3D Building Structured Abstraction from Diverse Point Clouds." Project Page: https://ai4city-hkust.github.io/BuildAnyPoint/

Find Similar Papers

Try Our Examples

Search for recent papers on 3D building reconstruction that utilize latent diffusion models for point cloud completion or surface refinement.
Which paper first introduced the concept of autoregressive mesh generation using tokenized sequences, and how does BuildAnyPoint's Loosely Cascaded design evolve this concept?
Explore studies that apply Diffusion Transformers (DiT) to large-scale urban scene modeling or digital twin generation from heterogeneous sensor data.

Contents

[CVPR 2024] BuildAnyPoint: Shifting the Paradigm of 3D Building Reconstruction via Generative Priors

1. TL;DR

2. 1. The "Dirty Data" Problem in Urban Reconstruction

3. 2. Methodology: The Loca-DiT Architecture

3.1. Step A: Hierarchical Latent Diffusion

3.2. Step B: Autoregressive Mesh Generation

4. 3. Results: Generalization is the Killer Feature

4.1. Key Metrics:

5. 4. Why it Works: The Power of Intermediate Priors

6. 5. Conclusion & Future Outlook