OTAD: Fixing the Structural Blindness of Audio Evaluation Metrics

TL;DR

Current audio metrics like Fréchet Audio Distance (FAD) are structurally "blind" to rare sound artifacts because they simplify complex distributions into smooth Gaussians. Optimal Transport Audio Distance (OTAD) fixes this by using a learned Riemannian adapter to sharpen the representation and Sinkhorn Optimal Transport to ensure that even a single bad sample can't hide in the crowd. The result is a metric that correlates much better with human judgment and can point out exactly which generated files are broken.

The Problem: Why FAD and KAD Fail

In the world of Text-to-Audio (TTA) generation, we don't just care about the "average" quality; we care about the "deal-breakers"—the random clicks, pops, or tonal breaks that ruin a sample.

The authors identify two fatal "primitives" in existing metrics:

1. The Primitive of Cost (The Encoder): We usually use frozen backbones (like VGGish or PANNs). These encoders were trained for classification, meaning they are designed to ignore small noise—the very artifacts we need to detect.
2. The Primitive of Coupling (The Distance): FAD assumes the data is Gaussian. Under this assumption, a "rank-1" error (an outlier in one direction) gets averaged out across the entire feature dimension.

As shown in the paper's Theorem 1, this "Bures averaging" systematically attenuates artifacts. KAD (Kernel Audio Distance) tried to fix the coupling by using a kernel, but it still suffered from the frozen encoder's blind spots.

Methodology: Two Primitives, Two Fixes

OTAD proposes a dual-track solution:

1. The Riemannian Ground-Metric Adapter

Instead of trusting the frozen encoder (Φ), OTAD adds a lightweight residual adapter: $g_{heta}(z)=z+f_{heta}(z)$.

• Intuition: It "warps" the space. As Proposition 1 suggests, this creates a local Riemannian metric that adds "curvature" precisely where the original backbone was blind to artifacts.
• Volume Control: Proposition 2 explains that the Jacobian of this map acts as a density reweighting factor. It effectively "spreads out" mass in the embedding space to make outliers more visible.

2. Entropic Sinkhorn Optimal Transport

Instead of fitting a Gaussian, OTAD uses discrete Optimal Transport.

• Sensitivity: By using the Sinkhorn divergence, the metric maintains "discrete-OT-grade" sensitivity. This means it doesn't matter if the artifact is in a high-dimensional space; it won't be diluted.

Caption: The OTAD framework: Raw audio passes through a frozen encoder, is refined by a learned Riemannian adapter, and then reference and test sets are compared via a discrete Sinkhorn coupling.

Key Results: Sensitivity and Human Alignment

Rank-1 Contamination

The authors tested what happens when you inject a single consistent artifact (rank-1 contamination).

• Result: Sinkhorn's sensitivity exceeds FAD's by a factor of 1.9x to 3.6x. FAD's response is nearly zero in regimes where humans would clearly hear the error.

Correlating with Human MOS

Testing on the DCASE 2023 Foley synthesis task, OTAD shows a superior Spearman correlation with human Mean Opinion Scores (MOS).

• Granularity: While FAD and KAD do okay at ranking broad "systems," their performance collapses when you look at specific "categories" (e.g., just "dog barks" or "footsteps"). OTAD remains stable and accurate.

Caption: Comparison of Spearman correlation with human MOS. OTAD (blue) significantly outperforms FAD and KAD, especially at the per-category granularity.

Per-Sample Diagnostics: Finding the Needle

One of the most powerful features of OTAD is the Transport Plan. Unlike FAD, which gives you a single number for an entire dataset, OTAD can decompose the cost for each individual sample ($c_j$).

• Detection: It achieves an AUROC ≥ 0.86 for detecting cross-class replacements and 1.0 for detecting silence or noise. This allows researchers to generate a "worst offender" list for debugging their models.

Depth Insight: Cost vs. Measure

The paper includes a fascinating 2x2 factorial decomposition. They ask: Is the improvement coming from the new adapter (cost) or the Sinkhorn method (measure)?

• The Findings: At lower dimensions ($d\le 512$), the Cost correction (the adapter) is the most important lever. At higher dimensions ($d=2048$), a Synergy emerges—you need both the adapter and the Sinkhorn coupling to get the best results.

Conclusion and Future Work

OTAD is a more rigorous, sensitive, and diagnostic tool for audio evaluation. While it is limited by the fact that no adapter can recover information entirely discarded by the original encoder (like in the case of EnCodec), it provides the most faithful "distilled" distance metric available today.

The authors have released the otadtk toolkit as a drop-in replacement for FAD pipelines. For researchers in TTA or music generation, moving from FAD to OTAD might be the difference between missing a failure mode and fixing it.