Relit-LiVE: Achieving Physically Consistent Video Relighting Without Camera Pose

TL;DR

Relit-LiVE is a breakthrough in video editing that allows users to change the lighting of a video realistically without knowing the camera's path. By training a model to predict both the new video and how the light source "moves" relative to the camera (environment video), it solves the long-standing problem of misaligned shadows and reflections in "in-the-wild" footage.

The Problem: The "Intrinsic" Bottleneck and Pose Dependencies

Current video relighting usually follows a "decompose-then-render" pipeline. You take a video, break it down into its "intrinsics" (like albedo and surface normals), and then shove it through a renderer with new lighting.

However, this approach faces two fatal flaws:

1. Lossy Decomposition: Intrinsic estimation is rarely perfect. For complex materials like glass or metallic surfaces, the decomposition often fails, resulting in "matte" or distorted looks.
2. The Pose Requirement: To make an environment map (the source of light) look right, the model needs to know exactly where the camera is for every single frame. Calculating these "poses" for a random phone video is computationally expensive and prone to error.

Methodology: The Fusion of RGB and "Environment Videos"

Relit-LiVE introduces two key technical innovations to bypass these hurdles.

1. RGB-Intrinsic Fusion Renderer

Instead of relying solely on G-buffers (depth, normal, etc.), the authors propose using the raw reference image as a guide. This raw image contains semantic and lighting cues that are hard to capture in a simple normal map. By fusing the raw RGB space with the intrinsic space, the model can "correct" the rendering errors of the decomposition stage.

2. Joint Environment Video Generation

This is the "special sauce." Rather than asking for a camera pose, Relit-LiVE predicts a transferred environment map for every frame.

• The Logic: If the camera moves left, the reflection on a car should move right. By generating the "environment video" (the light's perspective) alongside the relit video, the model implicitly learns the geometry of the scene.

Fig 1: The Relit-LiVE framework showing joint generation of relit video and environmental warping.

Advanced Training Strategies

The paper doesn't just stop at architecture; it introduces clever self-supervised loops:

• IPE (Intrinsic Perception Enhancement): The model uses "pseudo-realistic" data—interpolating between pure rendering and relit outputs—to learn how to decouple lighting more effectively.
• SIC (Self-supervised Illumination Consistency): It performs a "round-trip" check. If you relight a video with Light A, and then try to relight the output back with the original Light B, the result should match the original. This cycle-consistency ensures temporal stability.

Experimental Results: SOTA Performance

Compared to heavyweights like NeuralGaffer and Diffusion Renderer, Relit-LiVE shows a massive leap in quality.

The visual results are even more striking. The model successfully handles complex refractions in transparent objects and moves shadows accurately as the camera pans.

Fig 2: Qualitative comparison showing Relit-LiVE managing consistency where text-prompt methods fail.

Critical Insight & Conclusion

Relit-LiVE proves that implicit geometric reasoning (learning the environment warp) is more practical for real-world applications than explicit geometric estimation (SfM or SLAM).

Takeaway: By moving away from the rigid "decompose-render" formula and embracing a joint generation paradigm, the authors have created a tool that feels more like a "universal video engine" rather than just a niche rendering research project.

Limitations: The model is currently heavy. Generating a 57-frame video takes about 10 minutes on an A800 GPU. For real-time applications, significant optimization or distillation will be necessary.