Why word error rate alone misses the point
Word error rate (WER) has been the standard metric for decades, but it treats every word equally and ignores meaning. A 2026 paper argues that WER often fails to reflect the semantic correctness of an utterance at the sentence level [1]. For example, a transcription that swaps a key word could have a low WER but completely change the meaning, which WER would not catch.
To capture subjective quality, you need a metric that evaluates whether the transcription conveys the intended meaning. The same study proposes using an LLM-as-a-Judge, where a large language model scores the semantic fidelity of the output [1]. This approach aligns with how humans perceive quality—by understanding, not just by counting word matches.
How to combine metrics for better benchmark optimization
For benchmark optimization, you should not abandon WER but supplement it with semantic metrics. The 2026 study conducted experiments on standard benchmarks and found that both objective (WER) and subjective (LLM-as-a-Judge) evaluations showed improvements when using their interactive framework [1]. This suggests that a dual-metric approach gives a more complete picture of model quality.
Additionally, consider robustness across diverse inputs. A 2023 benchmarking study of speech-to-text services found significant performance variations depending on the type of audio (interviews, lectures, speeches) [4]. Therefore, your benchmark should include a variety of audio conditions to ensure that optimizing for one metric does not degrade performance on others.
Practical steps to implement subjective evaluation
Start by defining what 'subjective quality' means for your use case—semantic accuracy, user satisfaction, or robustness to noise. Then, integrate an LLM-as-a-Judge metric into your evaluation pipeline, as proposed in [1]. This involves feeding the reference and hypothesis transcriptions to an LLM and asking it to rate semantic coherence.
Also, test your model on a diverse set of inputs, as recommended by [4]. Use datasets that include different speaking styles, noise levels, and demographic groups. A 2024 robustness benchmark (Speech Robust Bench) provides 114 input perturbations to simulate real-world corruptions and also revealed disparities across demographic subgroups [5]. Including such tests helps ensure your optimization does not sacrifice robustness for average quality.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2023 to 2026, 4 from 2024 or later, 1 in Q1 journals, collectively cited 74 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 58 papers retrieved from a database of over 500 million.
Sources used in this answer
Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition
Proposes LLM-as-a-Judge as a semantic-aware evaluation metric and an interactive agent framework; experiments on GigaSpeech, WenetSpeech, and ASRU 2019 show improvements in both objective and subjective evaluations.
Overview of the IWSLT 2011 Evaluation Campaign
Reports on the IWSLT 2011 evaluation campaign, which included subjective evaluation of speech translation and speech recognition, providing a detailed methodology for human assessment.
The Faetar Speech Recognition Benchmark
Introduces the Faetar benchmark for low-resource ASR, with only 5 hours of transcribed noisy field recordings and inconsistent transcriptions, achieving a best phone error rate of 30.5% using a multilingual foundation model.
Benchmarking open source and paid services for speech to text: an analysis of quality and input variety
Benchmarks open-source and paid speech-to-text services on six datasets, finding significant performance variations based on input type, with paid services generally more accurate but still dependent on audio characteristics.
Speech Robust Bench: A Robustness Benchmark For Speech Recognition
Presents Speech Robust Bench, a benchmark with 114 input perturbations, and finds that model size, discrete representations, and self-training improve robustness, but also reveals disparities across demographic subgroups.