[CVPR 2025] MIBURI: Breaking the Real-Time Barrier for Expressive AI Avatars

TL;DR

Researchers have introduced MIBURI, the first fully causal, real-time framework capable of generating expressive full-body gestures and facial expressions synchronized with live dialogue. By directly utilizing the internal tokens of the Moshi speech-text foundation model, MIBURI eliminates the latency of traditional pipelines, delivering natural motion at just 36ms per frame.

Background Positioning: This work bridges the gap between high-quality offline gesture synthesis (like diffusion models) and the strict low-latency requirements of Embodied Conversational Agents (ECAs). It shifts the paradigm from "audio-to-gesture" to "token-to-gesture."

The "Broken" Pipeline: Why Current Avatars Feel Like Robots

Modern digital assistants understand what we say, but they lack "embodiment." If you look at current state-of-the-art gesture models, they suffer from two fatal flaws:

1. Future-Context Dependency: Models like EMAGE or ConvoFusion are non-causal; they need to "know" the end of a sentence to figure out how to start a gesture. This makes them useless for a live, unscripted conversation.
2. The Pipeline Tax: Conventional systems convert LLM text to speech, then tokenize that speech to drive gestures. Each step adds 100ms+ of latency, killing the fluidity of human interaction.

MIBURI's core insight: Why wait for the audio? By tapping into the internal embeddings of a speech LLM, the system can predict motion as the "thought" (token) is being generated.

Methodology: Hierarchical Motion and Two-Tier Transformers

The technical brilliance of MIBURI lies in how it handles the complexity of human movement without exploding the computational cost.

1. Body-Part Aware Codecs

Instead of tokenizing the whole body as one block, MIBURI uses Residual VQ-VAE to split movement into three streams: Face (FLAME), Upper Body (Hands/Arms), and Lower Body. Using RVQ allows the model to capture both coarse "jerks" and fine "finger-level" nuances across multiple quantization levels (K-levels).

2. The 2D Transformer Architecture

To avoid the quadratic complexity of modeling $TimesK$ tokens, MIBURI uses a disentangled approach:

• Temporal Transformer: Models the "rhythm" and dynamics across time steps.
• Kinematic Transformer: Fills in the skeletal details (the hierarchy of joints) within a single time step.

3. Solving "Mean Pose" Collapse

Autoregressive models often drift toward a static, "average" pose. MIBURI solves this with:

• InfoNCE Contrastive Loss: Encourages the model to stay close to the diversity of ground-truth motion.
• Voice Activation Loss: A binary classifier that helps the model distinguish between "listening" and "speaking" modes, preventing "phantom gestures" when the agent should be still.

Experimental Battleground: SOTA Performance

MIBURI was tested on the BEAT2 and Embody3D datasets.

• Latency: On an RTX 3090, MIBURI hits ~36ms. Compared to diffusion models that take seconds to denoise a sequence, MIBURI is effectively instantaneous.
• Quality: It achieved an FGD of 0.480 (lower is better), outperforming all other real-time baselines.
• Subjective Feedback: In user studies, participants found MIBURI’s gestures significantly more natural and better aligned with speech than existing real-time methods.

Critical Insights: The Causality-Quality Trade-off

The authors acknowledge a deep philosophical challenge: Where do gestures originate? In humans, gestures often precede speech (the "Width" of intent). Because MIBURI is strictly causal, it cannot "foresee" a gesture that should start before the corresponding word is tokenized.

Future Outlook: The next frontier is Intent Modeling. If we can disentangle "intent" from the LLM before it even picks a word, we can generate the "stroke" of a gesture exactly when a human would, achieving the holy grail of non-verbal communication.

Conclusion

MIBURI represents a massive leap for Embodied AI. By combining efficient Residual Quantization with a dual-transformer setup, it proves that we don't have to sacrifice expressiveness for speed. We are moving closer to a world where our digital assistants don't just speak—they interact with the physical presence of a human.