Why word-level emphasis sliders are the clearest win
The most intuitive way to make TTS feel controllable is to let users adjust emphasis on specific words, not just the whole sentence. A 2022 study from Apple tested three ways to encode emphasis in a neural TTS system: pitch/duration variance within words, a wavelet-based feature combining pitch, energy, and duration, and a learned mix of both. All three methods let users increase emphasis on a word by simply raising a value, and listeners perceived the emphasis change with little loss in quality—in fact, they preferred the emphasized utterances over flat ones in pairwise tests [2]. This means a product could offer a simple 'emphasis slider' per word, and it would work as users expect.
The same study found that the learned combination of features performed best, but even the simplest variance-based method was effective. That's good news for product designers: you don't need a complex model to give users a useful control. The key is that the control maps to a perceptible acoustic change—pitch and duration—rather than a hidden latent variable that users can't predict [2].
Utterance-level prosody controls: the backbone of expressiveness
Beyond individual words, users want to shape the overall melody and rhythm of a sentence. A 2023 study added utterance-level prosody control to a neural HMM-based TTS system, allowing users to adjust features like pitch range and speaking rate for the whole utterance. The system maintained stable alignment even with spontaneous speech (which includes fillers and disfluencies), and perceptual tests showed that prosody control did not degrade synthesis quality [1]. This suggests that a global 'expressiveness' dial—controlling pitch range, for example—can be added without breaking naturalness.
A 2024 study on emotion-controllable TTS went further by combining utterance-level prosodic factors with word-level prominence. They achieved an average linear controllability score of 0.95 (where 1.0 is perfect), meaning users could smoothly dial emotion intensity and prosodic variation, and the output followed accordingly [5]. This two-stage design—first setting emotion, then fine-tuning prosody—is a strong template for products that want both coarse and fine control.
Emotion and style controls: from soft labels to lightweight prototypes
Emotion is a high-level control that users care about, but it's tricky because emotions vary in intensity and blend. A 2024 study used 'emotion soft labels'—continuous values instead of a single category—to condition a TTS model. This allowed users to control emotion strength continuously, and the system achieved 51% emotion-distinguishable accuracy across three emotions, meaning listeners could tell which emotion was intended most of the time [5]. The same study also showed that adding word-level prominence on top of emotion labels preserved inter-emotion controllability while increasing intra-emotion diversity [5].
For style control, a 2026 study found that you don't need a heavy text encoder to understand style prompts. Instead, they used precomputed 'prototype attribute vectors'—averages of embeddings from many style descriptions—and achieved comparable word error rates and quality scores to a baseline with a full text encoder, while reducing model size by 38% [3]. This is a practical design choice for products that want to offer style presets (e.g., 'cheerful', 'sad') without the computational cost of a large language model.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2026, 3 from 2024 or later, collectively cited 74 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 65 papers retrieved from a database of over 500 million.
Sources used in this answer
Prosody-Controllable Spontaneous TTS with Neural HMMS
Adding utterance-level prosody control to a neural HMM-based TTS system preserved synthesis quality in perceptual tests, even with spontaneous speech containing disfluencies, and enabled synthesis of creaky voice types.
Emphasis Control for Parallel Neural TTS
In a comparison of three latent features for emphasis control, all allowed word-level emphasis by increasing feature values, with listeners preferring emphasized utterances over non-emphasized ones in pairwise tests, and the learned combination performed best.
Encoder-Free Style-Controllable Text-to-Speech with Voice Attribute Vectors
Replacing a large text encoder with precomputed prototype attribute vectors reduced model size by 38% while achieving comparable word error rates and automatic evaluation scores, indicating that lightweight style representations are viable.
Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching
Matcha-TTS, using optimal-transport conditional flow matching, achieved the highest mean opinion score in a listening test among strong baselines, with the smallest memory footprint and fast synthesis, though it does not focus on controllability.
Emotion-controllable Speech Synthesis Using Emotion Soft Label, Utterance-level Prosodic Factors, and Word-level Prominence
A two-stage emotion-controllable TTS model using emotion soft labels and prosodic factors achieved 51% emotion-distinguishable accuracy and average linear controllability scores of 0.95, while maintaining audio quality comparable to conventional models.