StreamingTalker: Low-Latency 3D Facial Animation via Autoregressive Diffusion
StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
StreamingTalker is an audio-driven 3D facial animation framework that introduces a novel Autoregressive (AR) Diffusion Model to generate realistic, synchronized facial motions in a streaming manner. By utilizing an AR condition predictor and a lightweight diffusion head, it achieves SOTA performance on BIWI and VOCASET datasets while maintaining low latency (25ms) for arbitrary-length audio inputs.
TL;DR
StreamingTalker addresses the critical "latency vs. quality" trade-off in speech-driven 3D facial animation. By shifting from a full-sequence diffusion approach to an Autoregressive (AR) Diffusion Model, it achieves real-time, streaming performance (25ms latency) without sacrificing the naturalness of expressions. It effectively handles arbitrary-length audio and maintains temporal stability even for sequences exceeding 60 seconds.
Problem & Motivation: The "Long Sequence" Bottleneck
Previous SOTA methods like FaceDiffuser or DiffSpeaker rely on processing the entire audio sequence simultaneously. While diffusion models are great at capturing the non-deterministic nature of human expressions (one-to-many mapping), they suffer from two fatal flaws:
- Fixed Horizons: They struggle with sequences longer than what they saw during training.
- Latency Spikes: The computational cost scales with sequence length, making "streaming" or "interactive" digital humans impossible.
The authors at Zhejiang University and Ant Group recognized that a streaming interface requires a model that can "start talking" almost immediately after the first few bits of audio arrive.
Methodology: The AR Diffusion Framework
StreamingTalker splits the task into two core components: a VQ-VAE to learn a compact facial motion prior and an AR Diffusion Model to handle the speech-to-motion mapping.
1. The AR Condition Predictor
Instead of attending to the whole sequence, the model looks at a fixed-length history buffer (60-120 frames). This "sliding window" approach uses a Transformer decoder to fuse:
- Historical Motion: Encoded latents from previous frames.
- Current Audio: High-level acoustic features from a pre-trained HuBERT encoder.
- Speaker Identity: A style embedding to maintain person-specific traits.
2. The Lightweight Diffusion Head
Unlike heavy UNet or Transformer denoisers, StreamingTalker uses a lightweight MLP-based diffusion head. Because the AR Condition Predictor has already done the heavy lifting of modality fusion, the diffusion head only needs to focus on refining the latent representation of the next frame.
Figure 1: The dual-stage pipeline featuring the AR Condition Predictor and the Diffusion Head.
Experimental Performance
The model was validated on the BIWI and VOCASET datasets.
- Lip Synchronization: StreamingTalker consistently beats CodeTalker and DiffSpeaker in Lip Vertex Error (LVE), particularly in handling bilabial consonants (like 'm', 'p', 'b') where lips must fully close.
- Temporal Stability: While other models "drift" or become jittery on long sequences, StreamingTalker remains coherent after 2,000+ frames.
Real-Time Efficiency
The most striking result is the latency comparison. As shown in the chart below, traditional diffusion models (red and orange lines) see latency explode as audio length increases. StreamingTalker (blue line) stays flat at roughly 25ms.
Figure 2: Latency remains constant regardless of audio duration, enabling true streaming interactions.
Visual Results & Qualitative Nuance
Qualitative comparisons show that the model excels at articulatory accuracy. Vowel shapes ('o', 'u') are rounder and more defined than in FaceFormer or CodeTalker.
Figure 3: Comparison of lip shapes across different models. StreamingTalker captures the subtle closure required for accurate speech.
Critical Insight & Conclusion
By integrating ALiBi (Attention with Linear Biases) and a fixed history window, the model achieves a form of "temporal extrapolation." This is a significant step forward for the industry, moving away from academic "batch processing" toward production-ready "streaming processing."
Limitations: The model is currently "expression-lite," focusing primarily on the mouth and jaw. Future iterations will need to integrate emotional cues (like EMOCA) to truly bring digital humans to life with affective facial dynamics.
Final Takeaway: StreamingTalker proves that for real-world interactions, Autoregression is not just a legacy technique but a necessary architectural choice for low-latency Diffusion models.