What makes a reconstruction controllable? Deformation models guided by human priors.
The most direct way to make a 4D reconstruction controllable is to build in a deformation model that is guided by a human prior—like a 3D face or body model. Rig3DGS does exactly this: it learns to deform 3D Gaussians from a fixed canonical space to match a target facial expression and head pose, using a 3D face morphable model as a guide. This design lets you control facial expressions and head poses independently, and it produces photorealistic renders of the whole scene. The key is that the deformation is not free-form; it's anchored to a known structure, which makes the output predictable and adjustable.
Similarly, DressRecon combines a generic human body prior with a video-specific 'bag-of-bones' deformation, learned via test-time optimization. This allows it to handle extremely loose clothing and object interactions while still maintaining a controllable body model. The separation of body and clothing deformations into different motion layers is what gives you control: you can adjust the body pose without messing up the clothing, and vice versa. This layered approach is a powerful design choice for controllability.
How fast can you iterate? Speed is a control feature.
Controllability isn't just about what you can adjust; it's also about how quickly you can see the result of your adjustments. 4D-Fly addresses this by making the reconstruction process much faster—it can reconstruct a 4D scene from a monocular video of hundreds of frames in about 6 minutes, which is more than 20× faster than previous optimization methods. This speed is achieved by directly applying data priors to explicit Gaussian primitives and using an anchor-based propagation strategy. For a user, this means you can tweak parameters and see the effect almost in real time, which makes the whole process feel much more controllable.
In contrast, many existing methods require hours of optimization, which makes interactive control impractical. The trade-off is that 4D-Fly's speed comes from a streaming paradigm that propagates Gaussians forward in time, which might not offer the same level of fine-grained control as a full optimization. But for applications where speed is crucial, this design choice is a game-changer.
What are the trade-offs? Best-case vs. typical-case evidence.
The evidence shows a clear gap between what's possible in the best case and what's typical. For example, Rig3DGS demonstrates high-quality, controllable portraits from casual monocular videos, but it's limited to facial expressions and head poses—not full-body motion. DressRecon handles loose clothing and object interactions, but it requires test-time optimization, which can be slow. 4D-Fly is fast, but it may not offer the same level of control as methods that optimize longer.
The papers also highlight the challenge of monocular video: it's inherently under-constrained. 4D-Fly notes that monocular videos suffer from limited multi-view information, which makes reconstruction ill-posed. To mitigate this, they use data-driven priors, but that adds complexity. The takeaway is that controllability often comes at the cost of speed or generality, and the best design choice depends on your specific needs—whether you prioritize fine-grained control, speed, or the ability to handle complex clothing and interactions.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2025, 3 from 2024 or later, collectively cited 180 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 54 papers retrieved from a database of over 500 million.
Sources used in this answer
Rig3DGS: Creating Controllable Portraits From Casual Monocular Videos
Rig3DGS uses a 3D face morphable model to guide deformation of 3D Gaussians, enabling control over facial expressions and head poses from monocular videos, and outperforms prior art while being orders of magnitude faster.
DressRecon: Freeform 4D Human Reconstruction from Monocular Video
DressRecon combines a generic human body prior with a video-specific 'bag-of-bones' deformation, separating body and clothing motion layers, to reconstruct loose clothing and object interactions from monocular video, achieving higher-fidelity reconstructions than prior art.
4D-Fly: Fast 4D Reconstruction from a Single Monocular Video
4D-Fly reconstructs a 4D scene from a monocular video of hundreds of frames in about 6 minutes, more than 20× faster than previous optimization methods, by applying data priors directly to explicit Gaussian primitives and using an anchor-based propagation strategy.
Humans in 4D: Reconstructing and Tracking Humans with Transformers
4DHumans uses a transformer-based network (HMR 2.0) for human mesh recovery and 3D tracking, achieving state-of-the-art results for tracking people from monocular video, including handling multiple people and occlusions.
Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model
Ub4D introduces a coordinate-based implicit neural representation with differentiable volume rendering and a dynamic scene flow loss, enabling reconstruction of large deformations and shape completion in occluded regions from monocular RGB video, improving surface reconstruction accuracy and robustness.