Identity-as-Presence: Bridging the Gap in Personalized Joint Audio-Video Generation

TL;DR

Personalized video generation has long struggled with "identity schizophrenia"—the face looks right, but the voice is generic or mismatched. Identity-as-Presence is a pioneering framework from Tencent WeChat Vision that treats identity as a unified multimodal anchor. By injecting both facial appearance and vocal timbre into a dual-tower Diffusion Transformer (DiT), the model generates hyper-realistic, synchronized, and identity-consistent audio-visual content, even for complex multi-subject interactions.

Problem & Motivation: The Identity Entanglement Trap

Current "Cameo" style applications (where a specific person speaks arbitrary content) usually follow a sequential path: generate audio $\to$ generate video $\to$ lip-sync. This modular cascade is brittle; it limits cinematic dynamics and often ignores the holistic nature of identity.

The core challenges the authors identify are:

1. Identity Entanglement: In a scene with two people, how does the model know which voice belongs to which face?
2. Data Scarcity: There are plenty of TTS (voice) datasets and plenty of video datasets, but high-quality paired identity-labeled audio-video data is rare.
3. Modality Disparity: Audio is 1D temporal; Video is 3D (2D spatial + 1D temporal). Aligning them without losing fine-grained identity details is non-trivial.

Methodology: The Core Architecture

The framework is built on a dual-tower DiT architecture, but the "secret sauce" lies in how identity is injected.

1. Unified Identity Injection

Instead of using separate adapters, the authors treat identity signals (facial images and audio clips) as additional input tokens.

• Shared Identity Embeddings: For each identity $k$, a unique learnable vector $e_{id}^{(k)}$ is added to both visual and auditory tokens. This acts as a "glue" that implicitly tells the model, "these two different signals belong to the same person."
• Structured Position Embeddings: To handle the spatial-temporal mismatch, the authors map facial tokens to "virtual temporal indices" and synchronize them with audio tokens. This creates an invariant geometric anchor for the attention mechanism to exploit.

Figure 3: Overview of the Identity-as-Presence framework, highlighting the joint injection of visual and auditory identity signals.

2. Asymmetric Self-Attention

A common pitfall in conditioning DiTs is "noise leak," where the diffusion noise contaminates the clean identity tokens. The authors solve this with Decoupled Parameterization (separate projection weights for reference and noisy tokens) and a Structural Mask that allows noisy latents to "see" identity but prevents identity tokens from being distracted by the noise.

3. Three-Stage Training Strategy

To overcome the data bottleneck, the model learns in stages:

• Stage 1 (Unimodal): Trains the audio tower on TTS data and the video tower on video data.
• Stage 2 (Multimodal): Activates cross-modal fusion on the scarce paired data.
• Stage 3 (Fine-tuning): Uses high-quality multi-view data to ensure identity holds up during large head rotations.

Experiments & Results: SOTA Across the Board

The model was tested against top-tier baselines like Ovi, LTX-2, and HunyuanCustom.

• Direct Identity Match: The use of Multi-View inputs significantly reduced identity "drifting" compared to one-shot methods.
• Multi-Subject Control: The "Subject Anchors" and "Identity Embeddings" proved critical. Without them, the model often suffered from voice-swapping or simultaneous talking.

Figure 5: Ablation study showing how Subject Anchors (SA) and Identity Embeddings (IE) prevent identity confusion and voice-face mismatch.

As shown in the table below, Identity-as-Presence achieves a VID-SIM of 0.667, a significant leap over R2I+Ovi (0.383).

Table 2: Comparison with state-of-the-art methods across Audio, Video, and Consistency metrics.

Critical Insight: Why This Matters

The fundamental contribution here isn't just "better quality"—it's the spatial-temporal binding. By mapping visual identity to a virtual timeline that aligns with audio, the authors provide the Transformer with a strong inductive bias that mirrors how humans perceive identity.

Limitations & Future Work

While the results are impressive, the reliance on high-quality multi-view data for Stage 3 suggests that truly "zero-shot" performance across radical poses still benefits from specific architectural priors. Future iterations might focus on moving away from discrete "ID embeddings" toward fully continuous latent identity representations that can generalize to entirely unseen populations without fine-tuning.

Takeaway

Identity-as-Presence proves that in the realm of AIGC, identity is not just a feature—it is a constraint. By enforcing this constraint at the architectural level through unified injection and geometric alignment, we can finally move toward truly personalized, cinematic digital personas.