DashengTokenizer: One Layer is Enough for Unified Audio Understanding and Generation

In the rapidly evolving landscape of Generative AI, audio foundation models have long been divided into two camps: Encoders that "understand" (like Whisper or HuBERT) and Tokenizers that "generate" (like Encodec or DAC). Bridging this gap usually involves complex distillation or lossy quantization.

DashengTokenizer, introduced by the MiLM Plus team at Xiaomi, shatters this status quo. Their discovery? You don't need complex VAEs or multi-stage distillation to achieve unified audio intelligence—one linear layer for acoustic injection is enough.

TL;DR

• The Core Innovation: Inverting the paradigm. Instead of distilling semantic knowledge into an acoustic model, they inject acoustic details into a frozen, powerful semantic encoder.
• Performance: SOTA across 22 understanding tasks (Speech, Music, Sound) and faster convergence in TTA/TTM generation than traditional VAEs.
• Architecture: A frozen 630M Transformer encoder + a tiny 0.66M acoustic injector + a 173M Vocos-based decoder.

The Problem: The Representation Conflict

Existing unified models typically follow one of two suboptimal paths:

1. Redundant Dual-Systems: Using two separate models for understanding and generation, which is computationally expensive.
2. Bottlenecked Unified Models: Training a single model (often a VQ-VAE) to do both. However, the pressure to reconstruct audio often washes away the abstract semantic features needed for complex tasks like Emotion Recognition or Scene Classification.

The authors identify this as a "Semantic-Acoustic Conflict."

Methodology: Acoustic Injection

Unlike prior work like Ming-UniAudio which uses a complex three-stage pipeline, DashengTokenizer adopts a incredibly simple approach:

1. Frozen Semantics: Take a pre-trained, high-performance semantic encoder (MiDashengLM) and freeze it.
2. Acoustic Injection: Extract low-level acoustic features from a Mel-spectrogram and pass them through a lightweight linear projection ($\varphi$).
3. Additive Fusion: Simply add the acoustic features ($z_{ac}$) to the semantic features ($z_{sem}$).

To ensure the "understanding" part of the brain isn't overwhelmed by "hearing" details, they introduce a Semantic Preservation Loss: ${\mathcal{L}}_{sem}=\Vert z_{sem}-z_{ac}{\Vert }_2^2$

Figure 1: DashengTokenizer [C] eliminates the multi-stage training required by distillation-based methods [B].

Experimental Results: The Best of Both Worlds

1. Superior Understanding

DashengTokenizer doesn't just "match" previous encoders; it often beats them because the injected acoustic information adds paralinguistic depth (pitch, texture) that pure semantic models might miss.

Table 1: Performance on the X-ARES benchmark. DashengTokenizer outperforms both specialized encoders and neural codecs.

2. High-Fidelity Generation

In Text-to-Audio (TTA) and Text-to-Music (TTM) tasks, the model was plugged into a Flow-based Diffusion Transformer (DiT).

• Convergence: It reached target performance levels 190k steps earlier than VAE-based models in CLAP and KL metrics.
• Quality: Achieved a lower FAD (3.06 vs 4.27) in TTA, indicating significantly better audio quality and text alignment.

3. Speech Enhancement

As a "latentspace denoiser," DashengTokenizer outperformed standard baselines (PESQ of 2.66 vs 2.24 for the prior Dasheng Denoiser), proving that its continuous latent space is highly robust for signal processing.

Why It Works: The Physics of "High-D" Latents

Most tokenizers try to cram audio into a small latent dimension (e.g., $d=128$). DashengTokenizer uses a high-dimensional unified embedding ($d=1280$).

While this seems inefficient, it actually provides a much "roomier" manifold for the DiT to explore during generation. By not forcing a bottleneck (the "VAE requirement"), the model preserves the nuances of the original signal while keeping the semantic anchors provided by the frozen encoder.

Critical Analysis & Conclusion

Takeaway: DashengTokenizer proves that the "representation gap" is more of a "design choice" than a fundamental law. By leveraging frozen semantic encoders, we can build universal audio models with significantly less compute and complexity.

Limitations:

• The unified feature is high-dimensional ($d=1280$), which might be heavy for some deployment scenarios compared to ultra-low bitrate discrete codecs.
• It shows a slight performance dip in "pure" semantic tasks like ASR (LS100h) compared to Whisper, suggesting that acoustic variance can occasionally act as noise for linguistic abstraction.

Future Outlook: This approach paves the way for a new generation of Audio LLMs that don't need to "switch modes" between listening and speaking—they use the same latent language for both.

Check out the official checkpoints and further research by the MiLM Plus team to see DashengTokenizer in action.