How does intonation perception differ between real and synthetic speech?

Two tasks, three manipulated factors

The study asks whether AI-generated voices are processed differently from natural voices, and how intonation and familiarity with the speaker change that processing. Seventeen native Mandarin speakers took part in a speaker-identity similarity task, and 11 of them also completed an intonation-recognition task on a different day. Both tasks used a 2 × 2 × 2 within-subject design: speech type (natural vs. synthetic), familiarity (familiar vs. strange speaker) and intonation (statement vs. question).

The stimuli were 10 neutral six-word Mandarin sentences, plus one practice sentence. Two strangers and two people familiar to the participants read them as statements and as questions; synthetic versions were produced with an F0-conditioned singing voice conversion model, which takes the pitch contour and content of a source recording and re-renders them in a target speaker's timbre. Two phonetics experts selected the synthetic versions with the best intonation expression.

The two tasks measure different things and should not be merged. In the similarity task, listeners heard a pair of sentences and judged whether the same person produced both, then rated similarity from 1 to 7. In the intonation task, listeners heard one sentence and judged whether it was a statement or a question. Accuracy and response time were recorded in both.

Mixed natural-synthetic pairs are harder to match

For accuracy in the similarity task, logistic regression found a significant main effect of speech type. Pairs containing two natural recordings were judged correctly with a mean accuracy of 0.917 (SE = 0.014), compared with 0.851 (SE = 0.018) for pairs mixing one natural and one synthetic recording. That gap is the clearest sign in this task that the cloned voice carries a detectable difference from the person it imitates.

The paper also reports a significant interaction between speech type and intonation. Within natural-only pairs, question stimuli were judged more accurately (mean 0.946, SE = 0.014) than statements (mean 0.875, SE = 0.022). Within mixed pairs, question (0.826, SE = 0.025) and statement (0.872, SE = 0.022) did not differ significantly. The authors interpret this as question intonation possibly serving as a cue for speaker identification, one that may be weakened when synthetic features are present. That reading is the authors' explanation, not a separately measured mechanism.

Familiar voices are faster but rated less similar

Familiarity changed response time and similarity ratings in the same task, and the two moved in opposite directions. Responses to strange speakers were slower (mean 0.848 s, SE = 0.089) than to familiar speakers (mean 0.748 s, SE = 0.078). Yet strange speakers received higher similarity ratings (mean 3.930, SE = 0.131) than familiar speakers (mean 3.560, SE = 0.135).

The authors explain this pattern by arguing that familiar voices are matched against stored acoustic representations, making processing faster and discrimination finer, so listeners apply a stricter standard to them. This is an interpretation consistent with the reported pattern, not a direct test of the underlying memory mechanism. Speech type also affected ratings: natural-only pairs scored higher (mean 3.900, SE = 0.136) than mixed pairs (mean 3.600, SE = 0.130), a main effect reported at p = 0.048.

Intonation recognition is near ceiling in every condition

Accuracy in the intonation task was very high throughout, which limits how much the interaction can tell us. Firth's bias-reduced logistic regression found a significant interaction between familiarity and speech type. For natural speech, strange voices scored 0.987 (SE = 0.006) and familiar voices 0.982 (SE = 0.008), a non-significant difference. For synthetic speech, strange voices scored 0.973 (SE = 0.012) and familiar voices 0.992 (SE = 0.007), also non-significant on its own.

The interaction is therefore a difference in direction between two small, individually non-significant contrasts, on accuracies clustered near the top of the scale. The authors read it as speech type changing how much familiarity contributes to voice processing. Readers should treat that as a hypothesis raised by the pilot rather than a settled effect, because the simple effects did not reach significance and the task left little room for errors.

Response time showed only a marginal effect of intonation: questions took longer (mean 0.270 s, SE = 0.014) than statements (mean 0.249 s, SE = 0.013). The authors connect this to pitch differences between statement and question being more discernible at sentence ends, and note that no interaction was observed, so the pattern held for both real and synthetic speech.

The cloning model kept speaker similarity roughly constant

Before the listening experiments, the researchers checked the synthetic voices objectively. A speaker encoder extracted embeddings from the converted speech and from the target speaker's reference speech, and cosine similarity between them served as the speaker-similarity score, ranging from -1 to 1. Statements averaged 0.55 (SD = 0.28) and questions 0.51 (SD = 0.30), which the authors describe as comparable across sentence types.

This matters for interpreting the accuracy results. The synthetic voices were selected to be reasonably similar to their targets and to have good intonation expression, so the differences listeners showed are not simply the product of obviously poor clones. At the same time, the similarity scores were moderate rather than near-perfect, and the selection of the best-scoring samples means the tested synthetic voices are likely better than what the model produces on average.

A small pilot with narrow scope

Several features bound what these results can support. The sample was 17 participants for similarity and 11 for intonation, all native Mandarin speakers aged 22 to 41, and all four stimulus speakers were female. The familiarity contrast was binary: voices of acquaintances versus strangers. The synthetic voices came from one singing voice conversion model applied to Mandarin sentences, so the findings describe that system and that language, not voice cloning in general.

The intonation task's near-ceiling accuracy is the most consequential constraint, because it compresses the room for any factor to show an effect. The similarity task, with accuracies in the 0.85 to 0.95 range, offers more usable variation. The paper itself frames the work as a preliminary exploration and calls for more extensive, multidimensional investigation.

Intonation and familiarity interact, not add up

The practical priority this study suggests is not a simple verdict that cloned speech is worse. The informative differences were interactions: question intonation helped similarity judgments only for natural pairs, and familiarity's contribution to intonation accuracy pointed in opposite directions for natural and synthetic voices. A test that averaged over intonation or familiarity would have missed both patterns.

For anyone assessing whether synthetic voices can stand in for natural recordings in language materials, the paper's evidence points to two specific checks. First, include question intonation alongside statements, since the natural-advantage pattern appeared there. Second, test both familiar and unfamiliar speakers, because the direction of the familiarity effect differed by speech type. With 11 to 17 listeners and near-ceiling intonation accuracy, these checks are worth running at larger scale before treating the observed interactions as stable.

Sources used in this answer