What do standard TTS quality tests actually miss?
Traditional TTS evaluation focuses on how natural and intelligible the speech sounds—but that says little about whether the model can follow a specific instruction. For example, a model might produce perfectly fluent audio while completely ignoring a request to speak in a 'whispery, sad tone' or to 'sound like a news anchor.' The new InstructTTSEval benchmark, which includes 6,000 test cases across three tasks (acoustic-parameter specification, descriptive-style directives, and role-play), found that current instruction-following TTS systems have 'substantial room for improvement' [1]. That means even the best models often fail to execute complex, multi-part instructions—something a simple quality score would never reveal.
Another gap is emotional control. EmoVoice, a model designed for fine-grained emotion control via natural language, was tested on a 40-hour English emotion dataset and a Chinese dataset. The researchers found that existing emotion evaluation metrics don't align well with human perceptual preferences [2]. So a model might score high on an automated emotion metric, but human listeners still find the emotion wrong or exaggerated. Quality tests that don't include human judgment on emotional appropriateness can therefore give a false sense of success.
Why can't quality tests catch prosody or precision failures?
Prosody—the rhythm, stress, and intonation of speech—is a key control dimension that quality tests often ignore. A model might sound natural but fail to produce the exact pitch contour or emphasis you requested. EdiTTS, a score-based editing method, showed that targeted pitch edits can be made without retraining, but the evaluation relied on listening tests and speech-to-text back transcription to verify that edits were applied correctly [3]. That's a much more granular check than a generic quality score, and it's rarely part of standard TTS evaluation.
Similarly, prosody control in spontaneous speech—like creaky voice or fillers—requires precise alignment between text and acoustics, which is hard to achieve. A neural HMM-based system was able to add utterance-level prosody control without degrading synthesis quality, but the authors had to perform perceptual tests to confirm that the control didn't harm quality [4]. This shows that quality and controllability are orthogonal: you can have one without the other, and quality tests alone can't tell you if the control is accurate.
What about disentanglement—can quality tests reveal if controls interfere?
Controllable TTS models often aim to disentangle factors like speaker identity, language, and style, so you can change one without affecting the others. But quality tests don't check whether changing the style inadvertently changes the speaker's voice or the language. A 2022 study proposed a method to minimize mutual information between latent representations to improve disentanglement, and they validated it with subjective listening tests that measured both quality and controllability [5]. The fact that they had to explicitly test controllability—not just quality—underscores that these are separate dimensions.
The papers collectively show that quality tests are necessary but not sufficient. They can tell you if the audio is pleasant and intelligible, but they can't tell you if the model followed your instructions, conveyed the right emotion, hit the exact prosody, or kept controls independent. That's why new benchmarks like InstructTTSEval are pushing for automated judges (like Gemini) to evaluate instruction-following ability directly [1]. Until such evaluations become standard, a model can pass quality tests while failing at the very thing it was designed to do—being controllable.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2025, 2 from 2024 or later, collectively cited 126 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 55 papers retrieved from a database of over 500 million.
Sources used in this answer
InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
Introduced InstructTTSEval, a benchmark with 6,000 test cases across three tasks (acoustic-parameter specification, descriptive-style directive, role-play) in English and Chinese, and found that current instruction-following TTS systems have substantial room for improvement in following complex natural-language instructions.
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
Proposed EmoVoice, an LLM-based emotional TTS model, and found that existing emotion evaluation metrics do not align well with human perceptual preferences, suggesting that automated metrics may not reliably reflect human judgment of emotional expression.
EdiTTS: Score-based Editing for Controllable Text-to-Speech
Presented EdiTTS, a score-based editing method for TTS that allows targeted pitch and content edits without retraining, and showed via listening tests and speech-to-text back transcription that it outperforms baselines in satisfying user-imposed requirements.
Prosody-Controllable Spontaneous TTS with Neural HMMS
Developed a neural HMM-based TTS system with utterance-level prosody control for spontaneous speech, and demonstrated through objective and perceptual tests that prosody control does not degrade synthesis quality while enabling reproduction of creaky voice types.
A Controllable Multi-Lingual Multi-Speaker Multi-Style Text-to-Speech Synthesis With Multivariate Information Minimization
Proposed a multivariate information minimization method to disentangle latent representations for multi-lingual, multi-speaker, multi-style TTS, and validated via subjective listening tests that it improves both quality and controllability.