Why does the same voice get different quality scores depending on the test setup?
Subjective quality isn't a fixed property of a voice—it depends on the situation the listener imagines. A 2024 crowdsourced study tested four TTS voices under different task framings (domestic robot, child's voice replacement, audiobook, and long informative text) and found that mean opinion scores (MOS) varied significantly across framings, with individual voices performing differently in each context [5]. This means a voice that sounds great for an audiobook might score poorly as a robot assistant, so any evaluation should specify the intended use case and framing, or the results won't generalize to real applications.
The practical implication: when you design a subjective test for expressive speech, don't just ask 'how natural is this?'—give listeners a concrete scenario. The 2024 study explicitly warns that decontextualized MOS evaluations are not reliable and that reports should always state the framing used [5]. So, for your own evaluation, define the target application (e.g., conversational agent, audiobook narrator) and instruct raters accordingly.
What specific aspects of expressiveness should you ask listeners to rate?
Expressiveness is multi-dimensional, so a single overall score isn't enough. Research on Mandarin audiobook synthesis used a multi-scale approach, extracting style embeddings at three levels (likely sentence, paragraph, and discourse) and found that modeling these levels improved both naturalness and expressiveness in subjective tests [2]. This suggests you should evaluate style at multiple granularities—e.g., how well the emotion carries across a sentence versus a whole paragraph—not just one global rating.
Another study on self-supervised style representation for audiobook TTS found that using implicit, context-aware style embeddings improved subjective ratings on both in-domain and out-of-domain test sets, and also made emotion transitions in long paragraphs sound more natural [3]. So, when evaluating, include long-form passages to assess emotional continuity, not just isolated sentences. Ask raters about emotional consistency and transition smoothness, not just 'expressiveness' as a vague whole.
How can objective measures support subjective judgments?
Objective prosody metrics—like pitch, duration, and energy—can back up subjective ratings and even predict them. In a 2021 study on cross-speaker style transfer, the proposed model achieved prosody accuracy on par with a speaker-dependent model (trained on the target speaker's own data) and outperformed baselines in both objective and subjective evaluations [4]. This shows that measuring prosody objectively (e.g., correlation with ground-truth pitch) can serve as a proxy for style fidelity, complementing human listening tests.
However, objective metrics alone don't capture perceived quality. The same study used subjective evaluations to confirm that the prosody bottleneck improved style transfer quality, indicating that you should always pair objective measures with human ratings [4]. For your evaluation, collect both: compute prosody error metrics (e.g., F0 RMSE, duration error) and run listening tests with the context framing discussed earlier. If they agree, you have strong evidence; if they diverge, investigate why—maybe the objective metric misses something listeners care about.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2024, 2 from 2024 or later, collectively cited 54 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 76 papers retrieved from a database of over 500 million.
Sources used in this answer
Generative Expressive Conversational Speech Synthesis
GPT-Talker, a generative conversational speech synthesis system, outperformed other state-of-the-art systems in both subjective and objective evaluations of naturalness and expressiveness on a new 236-hour natural conversational dataset (NCSSD).
Towards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis
A multi-scale speaking style modeling method, using three levels of style embeddings and a hierarchical context predictor, significantly improved naturalness and expressiveness in subjective evaluations on a Mandarin audiobook dataset.
Self-supervised Context-aware Style Representation for Expressive Speech Synthesis
A self-supervised style representation learned from plain text using contrastive learning and deep clustering improved subjective ratings over human-annotated style tags on both in-domain and out-of-domain audiobook test sets, and made emotion transitions in long paragraphs more natural.
Cross-Speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis
A cross-speaker style transfer TTS with an explicit prosody bottleneck achieved prosody accuracy on par with a speaker-dependent model and outperformed cycle-consistency and GMVAE baselines in both objective and subjective evaluations.
Assessing the impact of contextual framing on subjective TTS quality
In a crowdsourced MOS evaluation of four TTS voices, systematically varying task framing (robot, child's voice, audiobook, long text) significantly changed MOS scores, with individual voice performance varying across framings, indicating that decontextualized evaluations do not generalize.