[CVPR 2026] CT-1: Bridging Spatial Reasoning and Video Generation via Vision-Language-Camera Models
CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation
The paper introduces CT-1 (Camera Transformer 1), a Vision-Language-Camera (VLC) model that predicts physically plausible 3D camera trajectories from a single image and text prompt. By integrating these predicted trajectories into diffusion-based video generators, the framework achieves state-of-the-art camera-controllable video synthesis.
Executive Summary
TL;DR: CT-1 (Camera Transformer 1) is a specialist model that acts as a "director," translating simple text instructions like "zoom into the subject" into professional-grade 3D camera trajectories by looking at a single reference image. By decoupling trajectory prediction from video pixel synthesis, it achieves a 25.7% boost in control accuracy compared to existing end-to-end models.
Background: In the landscape of generative AI, we’ve moved from "Text-to-Video" to "Controllable Video." However, most models struggle to understand where the camera should move based on the depth and layout of a specific image. CT-1 occupies a unique spot in the academic archipelago: it is a Vision-Language-Camera (VLC) model that transfers spatial reasoning knowledge to the generative process.
Problem & Motivation: The "Blind" Camera Problem
Current video foundations (like Wan2.2 or CogVideoX) treat camera instructions as just another text tag. They lack a physical "understanding" of the scene. For instance, if you ask for a "dolly-in" on a desk, the model needs to know the 3D distance between the camera and the desk to make the motion plausible.
The authors identify two main bottlenecks:
- Manual Labor: Precise control currently requires raw SE(3) parameters, which no average user can provide.
- Semantic Mismatch: Text-only models don't know the "constraints" of the image (e.g., you can't "truck right" if there's a wall in the way).
Methodology: The Architecture of a Digital Director
CT-1 isn't just a regressor; it's a generative trajectory model. It understands that for any given image and prompt, there isn't just one correct way to move the camera—there is a distribution of plausible paths.
1. Multimodal Encoding
The model uses a dual-branch vision encoder (DINOv2 for structure, SigLIP for semantics) and feeds them into a LLaMA-2 backbone. The secret sauce is the <CAM> token—a learnable bottleneck that condenses the entire scene's spatial logic into a single vector.
2. The Diffusion Transformer (DiT) Head
Unlike prior works that use simple MLPs to predict coordinates, CT-1 uses a DiT. It starts with Gaussian noise in the shape of a trajectory and iteratively refines it into a smooth 3D path.
3. Wavelet-based Regularization (WavReg)
To prevent the "jittery" camera motion common in AI videos, the authors introduced WavReg.
- The Intuition: Most camera motion is "low-frequency" (smooth trends). Noise and errors are "high-frequency."
- The Solution: By applying a 1D Discrete Wavelet Transform (DWT), the loss function penalizes errors in the global "trend" more heavily than tiny local fluctuations.
Figure 1: The CT-1 framework: from Multimodal Encoding to Diffusion-based Trajectory Synthesis.
CT-200K: The Data Powerhouse
You can't train a VLC model without data. The authors built CT-200K, featuring 47 million frames. They used a "teacher" model (VGGT) to extract ground-truth camera parameters from existing videos and used LLMs to "clean" the text descriptions, ensuring the camera instructions match what is actually happening in the pixels.
Experiments & Results: Success in the Wild
The results are striking. On CameraBench, CT-1 achieves an 81.6% success rate in following complex instructions, whereas even the most advanced video models like Wan2.2 hover around 62-64%.
| Method | Success Rate (Avg) | Complex Motion |
|---|---|---|
| Wan2.2 (w/ Prompt Expansion) | 64.9% | 68.5% |
| Ours (CT-1) | 81.6% | 81.5% |
Figure 2: Performance comparison across different camera motion types.
The "Reasoning" scenarios are where the model shines. If the prompt is "move towards the wall with posters," CT-1 identifies the wall in the image and calculates the trajectory to move toward that specific 3D coordinate.
Critical Insight & Conclusion
The Takeaway: The modularity of CT-1 is its greatest strength. Because it outputs standard camera parameters, it can be "plugged into" any controllable video model (CameraCtrl, MotionCtrl, etc.).
Limitations: While CT-1 is excellent at spatial reasoning, its inference adds a slight overhead (about 5.3% of total generation time). Furthermore, it currently relies on a single starting frame; future iterations might benefit from multi-frame "anchor" inputs to handle even more complex cinematic pans.
By treating camera control as a linguistic-spatial reasoning problem rather than a pixel-matching problem, CT-1 brings us one step closer to truly professional-grade AI cinematography.