How can subjective quality be evaluated for controllable text-to-speech models?

Human listening tests are the gold standard—but they need structure to be useful

The most direct way to evaluate subjective quality is to ask people to listen and rate what they hear. The standard metric is Mean Opinion Score (MOS), where listeners rate naturalness or overall quality on a scale (often 1–5). Several studies here use MOS as a primary measure: for example, the P2VA framework improved MOS by 0.33 points when converting persona descriptions into voice attributes [1], and DiffVoice used subjective evaluations to show it beats the best publicly available systems in naturalness [9]. But a single MOS number can hide why a system fails. That's why the best evaluations pair MOS with more diagnostic tests.

For controllable TTS, you often want to know not just 'does it sound good?' but 'did it follow the instruction?' or 'is the emotion right?' That requires task-specific listening tests. For instance, the CoT-TTS challenge combines human subjective assessment with objective metrics and multimodal LLM-based evaluation to judge whether a system infers the intended speaking manner from context [4]. Similarly, EdiTTS used listening tests to confirm that its editing method produces samples that satisfy user-imposed requirements [8]. The lesson: design your listening test around the specific control you're evaluating, not just overall quality.

Combine human ratings with objective metrics to catch what ears miss

Human listening is essential, but it's expensive and can be inconsistent. That's why the studies here consistently pair subjective tests with objective metrics—especially Word Error Rate (WER) from automatic speech recognition (ASR) and other automatic scores. For example, P2VA reduced WER by 5% alongside the MOS improvement, showing that better voice attribute conversion also improves intelligibility [1]. The encoder-free style-control model achieved comparable WER and automatic evaluation scores to a baseline with a heavy text encoder, demonstrating that lightweight representations can maintain perceived quality [10]. These objective metrics act as a proxy for quality and can catch issues like mispronunciations or dropped words that listeners might not consciously notice.

Objective metrics are also useful for localizing problems. One study used frame-level quality scores to detect synthesis artefacts in two state-of-the-art TTS systems, and then confirmed with listening tests that listeners rated those segments as poor more often than a random control set [2]. This shows that objective predictors can guide where to listen, making subjective evaluation more efficient. However, objective metrics aren't perfect—they can't capture expressiveness or emotional nuance. That's why the hybrid evaluation protocol in MINT-Bench jointly assesses content consistency, instruction following, and perceptual quality, combining automatic and human measures [3]. The takeaway: use objective metrics to screen and diagnose, but always validate with human ears.

Match the evaluation to the control dimension you care about

Controllable TTS can vary along many axes—emotion, accent, speaking rate, pitch, and more. The right evaluation depends on which axis you're testing. For emotion control, EmoSphere++ used both objective and subjective evaluations to show fine-grained control over emotional style and intensity, including for unseen speakers [7]. For accent control, a study on accented TTS used both objective and subjective measures to show that fine-tuning with limited data improved prosodic rendering and accent similarity [5]. For prosody and pitch editing, EdiTTS used listening tests and speech-to-text back transcription to verify that edits were applied correctly [8].

The key is to define your control dimension and then choose metrics that directly measure it. For example, if you're testing instruction following, you need a benchmark like MINT-Bench that has a hierarchical taxonomy and evaluates across ten languages, revealing that compositional and paralinguistic controls are major bottlenecks [3]. If you're testing context awareness, the CoT-TTS challenge requires systems to output a chain-of-thought reasoning analysis alongside the speech, and evaluates both [4]. And if you're testing fine-grained control like emotion intensity at the phoneme or word level, you need hierarchical emotion distributions as in the hierarchical emotion prediction study [6]. The bottom line: there's no one-size-fits-all evaluation; design your protocol around the specific control you're claiming.

About These Sources

This answer is built on 10 peer-reviewed studies — published from 2022 to 2026, 8 from 2024 or later, collectively cited 120 times — selected as the most relevant from 13 studies that passed quality screening, drawn from 64 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions