A benchmark built on deliberately conflicting audio
DEAF asks whether an audio model actually listens to the signal or infers the answer from words. To test this, the authors built 2,756 conflict stimuli in which what is said disagrees with how it sounds, spanning three acoustic dimensions: emotional prosody (ESC), background sounds (BSC), and speaker identity (SIC).
The design matters because ordinary speech benchmarks reward correct answers without revealing their source. If a clip's words and its acoustics agree, a model can score well by reading the transcript alone. DEAF removes that shortcut by making the acoustic evidence the only route to the correct answer, and by pairing each conflict with a question whose answer the text would get wrong.
Three interference levels separate content bias from prompt pressure
Each conflict type is evaluated at three levels of increasing textual influence. At Level 1, only the audio contains a semantic–acoustic conflict, with no misleading prompt, so a wrong answer can be attributed to the spoken content. At Level 2, the audio is acoustically matched to its content and the only pressure is a misleading prompt describing the wrong acoustic feature, isolating prompt-induced sycophancy. At Level 3, the semantic conflict returns and a prompt reinforces it, so both text channels push toward the wrong answer.
This ladder is the paper's key methodological move. Because the levels differ in which textual channel is active, the authors can attribute failures to content, to prompt, or to their combination rather than treating all text influence as one phenomenon. Levels 2 and 3 reuse pre-generated clips, so the comparison does not require new audio.
The acoustic robustness score counts answers that follow the signal
The headline metric is the Acoustic Robustness Score (ARS), reported as a percentage. It measures how often a model's answer follows the acoustic evidence when text points elsewhere; higher values mean stronger acoustic grounding under textual interference. The paper also reports raw accuracy on mismatched samples, which is a different quantity and should not be read as the same thing.
That distinction is visible in the numbers. On ESC, no model exceeds 14.6% accuracy at Level 1 or 3.4% at Level 3, well below the 25% expected from guessing uniformly among four emotion categories. Responses are open-ended and scored by DeepSeek-R1 acting as an LLM judge, with each sample evaluated zero-shot and repeated three times.
Speaker identity survives text pressure better than prosody
At Level 1, where only the spoken content conflicts, a clear hierarchy appears. Speaker identity yields the highest ARS, between 39.6% and 47.1% for every model except GPT-4o-Audio, followed by emotional prosody (0.9%–24.8%) and background sounds (1.3%–24.5%). The authors read this as voice characteristics such as gender and age being more perceptually accessible, while prosodic and environmental cues are more easily overridden by conflicting words.
Model differences are large. Qwen3-Omni and Audio Flamingo 3 show the strongest Level 1 performance, while GPT-4o-Audio sits near zero across all three dimensions, which the authors suggest may reflect an audio pipeline that effectively reduces to speech recognition followed by reasoning.
A misleading prompt sometimes raises robustness
Level 2 produces a counterintuitive result: for emotional prosody and background sounds, several models score higher than at Level 1. Qwen2-Audio rises from 14.7 to 34.0 on ESC, and Qwen3-Omni from 24.5 to 41.6 on BSC. The explanation is structural rather than a sign of better listening: Level 2 uses neutral-content audio, removing the in-audio semantic conflict, so the prompt is the only textual pressure and the audio itself is unambiguous.
Speaker identity moves the other way. ARS drops sharply for several models, with Audio Flamingo 3 falling from 44.6 to 15.0 and SALMONN from 39.6 to 13.6. The authors conclude that misleading identity prompts, such as asserting the speaker is an elderly woman, are particularly effective at overriding voice-based judgments.
When both text channels agree, acoustic grounding collapses
Level 3 is the hardest condition, and it is where the text-dominance pattern becomes stark. ESC falls below 7% for every model and BSC below 15%. Only speaker identity retains moderate robustness, between 32.6% and 44.1% excluding GPT-4o-Audio, consistent with voice gender and age cues being more salient than prosodic or environmental ones.
Even the strongest models capitulate. Qwen3-Omni, among the best at Levels 1 and 2, drops to 0.2% on ESC and 14.5% on BSC at Level 3. The authors also report that explicit versus implicit semantic cues make limited difference overall, suggesting the level of interference, not lexical specificity, drives text dominance. Human evaluation confirmed that the critical acoustic cues in these stimuli remain perceptually identifiable.
What the benchmark does and does not cover
The scope is deliberately narrow. DEAF covers emotion, background sounds, and speaker identity, leaving out temporal reasoning, multi-speaker interaction, and complex acoustic scenes. Most stimuli are generated through text-to-speech and controlled synthesis pipelines, which allow precise manipulation but may not capture the variability and noise of real-world speech. Scoring relies on an LLM judge, which scales evaluation but may introduce bias in ambiguous cases without large-scale human verification, and the experiments cover seven models in a zero-shot setting only.
These boundaries qualify how far the result travels. The finding is about behavior on controlled acoustic–semantic conflicts, not a claim that these models cannot process audio at all. The paper's own framing is a gap between high scores on standard speech benchmarks and genuine acoustic understanding, and the conflict design is what makes that gap measurable.
Judge audio models on conflict cases, not benchmark averages
The practical priority DEAF establishes is a comparison, not a single score. A model's standing on standard speech benchmarks says little about whether it will follow the acoustic signal when words disagree, and the level structure shows why: the same model can look reasonably grounded at Level 2 and near zero at Level 3.
Among the seven models, the ordering that matters is by dimension and interference level. Speaker identity is the cue most likely to survive text pressure, emotional prosody and background sounds the least, and dual interference the condition where nearly all models fail. Readers evaluating an audio model should therefore ask which acoustic dimension and which text channel a reported result actually tests, because averaging across them hides the collapse that DEAF was built to expose.
Sources used in this answer
DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models
Across seven audio multimodal LLMs, DEAF's 2,756 conflict stimuli reveal consistent text dominance: models respond to acoustic variation but their predictions are driven mainly by textual content and prompts, with acoustic robustness collapsing under combined semantic and prompt interference.