[AudioCapBench] Gemini vs. GPT: A New Benchmark Exposes the Strengths and "Silence" of Audio LMMs

TL;DR

The landscape of Large Multimodal Models (LMMs) is shifting from text-centricity to native audio-to-text understanding. Salesforce AI Research introduces AudioCapBench, a comprehensive benchmark spanning Sound, Music, and Speech. Evaluating 13 models, the study finds Gemini 3 Pro leading the pack, while also uncovering a "conservative bias" in OpenAI’s mini models that prioritize silence over the risk of being wrong.

Background Positioning

In the trajectory of AI, audio understanding was long treated as a two-step "transcribe-then-process" task. With the advent of GPT-4o and Gemini 1.5/2.0/3.0, we entered the era of Native Audio LMMs. However, evaluating these "black-box" APIs is difficult. AudioCapBench settles the score by moving beyond simple word matches (BLEU/METEOR) toward a semantic "LLM-as-Judge" approach.

Problem & Motivation: Why Word-Overlap Fails

Traditional metrics are brittle. If a reference says "Fireworks exploding" and a model predicts "Multiple loud pops from pyrotechnics," BLEU-4 would give it a near-zero score despite being perfectly accurate.

The researchers identified that existing benchmarks (like Clotho or AudioCaps) were too narrow. A general-purpose AI should distinguish a C-major chord from a dog bark and a frustrated speaker from a calm one. Thus, AudioCapBench was born to test the boundaries of environmental sound, musicality, and paralinguistic speech features.

Methodology: The Orthogonal Judge

The core innovation is the LLM-as-Judge framework. Using GPT-4o as the evaluator, the system scores predictions from 0-10 on three distinct axes:

1. Accuracy ($s_{acc}$): Precision of events described.
2. Completeness ($s_{comp}$): Recall of key attributes found in the reference.
3. Hallucination ($s_{hall}$): The Inverse False Positive Rate (higher is better, meaning less fabrication).

Figure 1: The mapping of evaluation dimensions to Information Retrieval theory (Precision, Recall, and FPR).

The framework also utilizes category-specific guidance, ensuring that music is judged on genre/mood while speech is judged on emotion/transcript.

Experiments & Results: The Gemini Dominance

The results provide a clear leaderboard. Gemini models (blue bars in the charts) consistently outperform OpenAI (green) across the board.

Figure 2: Gemini models occupy the top slots, with Gemini 3 Pro reaching a score of 6.00/10.

The "Mini" Model Paradox

A fascinating discovery is the Accuracy vs. Hallucination trade-off. Models like gpt-4o-mini-audio-preview achieved the highest hallucination scores (near 8.0), but the lowest accuracy. Why? Because they are incredibly brief. By saying very little, they avoid making mistakes, but they fail to describe the audio content significantly.

Figure 3: Visualization of the trade-off. Gemini models (bottom right) take more risks and achieve higher accuracy, whereas OpenAI mini models (top left) are safer but less useful.

Key Category Insights:

• Music is the hardest: Models struggle to identify specific instrumentation and tempo compared to simple environmental sounds.
• Speech is the easiest: Models likely leverage the transcript to infer the context and emotion naturally.
• Realtime Models lag behind: OpenAI’s Realtime API is optimized for low-latency dialogue, not descriptive precision, resulting in lower scores than their Chat Completion counterparts.

Critical Analysis & Conclusion

Takeaway

AudioCapBench proves that the "intelligence" of an audio model isn't just about transcription; it's about holistic acoustic scene understanding. Gemini currently leads in "hearing" the world with more detail, while OpenAI takes a more conservative, grounded approach in its smaller models.

Limitations & Future Work

The study relies on a single LLM judge (GPT-4o), which might have its own biases toward certain phrasing styles. Future iterations should include open-weight models (like Qwen-Audio or SLAM-LLM) to see how they stack up against the proprietary giants.

For developers, the message is clear: when choosing an audio model, decide whether you value detailed completeness (Gemini) or conservative grounding (OpenAI).