DiLAST: Awakening the Hidden Power of 3D Latents for Arbitrary Style Transfer
Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion
The paper introduces DiLAST, a test-time optimization framework designed for high-fidelity 3D style transfer. It leverages a pretrained 2D Diffusion Model (Stable Diffusion) to guide the structured 3D latents of generative backbones like Trellis, achieving state-of-the-art results in transferring out-of-distribution (OOD) styles to 3D assets.
TL;DR
DiLAST (2D Diffusion-based Latent Awakening for 3D Style Transfer) is a plug-and-play framework that enables 3D assets to adopt "out-of-distribution" (OOD) styles—like Cyberpunk or Ukiyo-e—with high fidelity. By using a 2D Stable Diffusion model to "teach" a 3D generative model through attention distillation, it bypasses the creative limitations of native 3D training data, proving that existing 3D latents are far more expressive than previously thought.
Background: The "Distribution Trap" in 3D Generation
We are currently witnessing a shift from 2D-guided 3D generation (like Score Distillation Sampling) to Native 3D Generative Models (like Trellis or Hunyuan3D). These models denoise directly in a structured 3D latent space, offering superior geometry.
However, these models suffer from a generalization bottleneck. Because 3D datasets are tiny compared to 2D datasets (like LAION-5B), 3D models only "understand" a narrow range of styles. When you ask them to apply a radical artistic style, the attention mechanisms break, resulting in artifacts or flat textures.
The Core Insight: 2D Models as Style Teachers
The authors of DiLAST realized that we don't need to retrain 3D models. Instead, we can use the vast stylistic knowledge embedded in 2D Diffusion models to "guide" the 3D denoising process.
The Insight: The 3D latent space itself is expressive enough to represent almost any style; it just doesn't know how to get there. DiLAST acts as a GPS, steering the 3D latent toward the desired stylistic coordinates.
Methodology: How DiLAST Works
The framework operates as a test-time optimization process. It doesn't change the weights of the 3D model; it optimizes the latent vector during the denoising steps.
1. The Differentiable Bridge (3DGS)
To let the 2D "teacher" see what the 3D "student" is doing, DiLAST decodes the 3D latent into 3D Gaussian Splatting (3DGS) primitives. This allows for high-speed, differentiable rendering of the 3D asset into 2D views.
2. Triple-Branch Attention Distillation
DiLAST sets up three parallel branches in a 2D LDM:
- Content Branch: Extracts structural features from the original object.
- Style Branch: Extracts artistic features from the reference style image.
- Optimization Branch: The current state of the 3D asset being stylized.
By computing the gradient of an Attention Distillation Loss, DiLAST forces the optimization branch to match the "Query-Key-Value" interactions of the style branch. This ensures the brushstrokes and textures are transferred correctly without losing the object's identity.
3. Cleaning the Mess (Regularization)
Style transfer can be "violent" to geometry. To prevent artifacts, DiLAST introduces:
- Floater Removal Loss: Prunes stray Gaussians that appear as "floaters" in space.
- Color Speckle Suppression: Smoothes the latent field to prevent saturated color "clipping" (e.g., random neon pixels).
Experimental Results: A New Ceiling for 3D Style
The performance jump is dramatic. In user preference studies, DiLAST is chosen 90% of the time over current SOTA baselines.
SOTA Baselines vs. DiLAST
While prior methods like MorphAny3D often fail to capture the color palette or destroy the geometry of the object, DiLAST maintains a perfect balance. It is truly "plug-and-play," working across various backbones such as UniLat3D and Hunyuan3D.
Critical Analysis & Conclusion
The "Transfer-then-Generate" Myth
The paper debunks the intuitive idea of "stylizing a 2D image first, then generating 3D from it" (TTG). TTG often introduces 3D inconsistencies because the stylized 2D image becomes an "OOD" input for the 3D generator. DiLAST’s strategy of direct latent steering is clearly superior for consistency.
Limitations
The main cost is inference time. Since this is a test-time optimization, it is slower than a single forward pass. It also relies on the 2D LDM being "smarter" than the 3D model; if the 2D model fails to understand a style, the 3D guidance will also fail.
Future Outlook
DiLAST proves that we are sitting on a goldmine of unutilized potential in current 3D latent representations. Future work could extend this "Awakening" concept to 3D object editing, relighting, and even structural morphing, all guided by the massive priors of 2D foundation models.