Do speech representations preserve regional accent across read and spontaneous speech?

Whisper sets the matched-condition ceiling

The study asks whether regional accent cues survive a change of speaking style. It uses RVG1-v3, a German corpus with 500 region-labelled speakers from nine regions, of whom 498 also have spontaneous recordings and 432 have geocoded hometown coordinates. Classification therefore uses 500 read and 498 spontaneous speakers, read-spontaneous analyses use 498, and geolocation uses 432. The nine classes are imbalanced (44/103/31/73/53/63/20/13/100 speakers), so unweighted average recall (UAR) is the primary classification metric.

Ten frozen speaker-level representations from seven families are compared: Whisper-large-v3, three self-supervised encoders (WavLM-large, wav2vec 2.0-XLSR, HuBERT-large), two speaker embeddings (x-vector, ECAPA-TDNN), a phonological descriptor (PhonoQ), acoustic features (eGeMAPS), the Mimi codec latent, and Qwen2-Audio hidden states. Encoder activations are mean-pooled into one vector per speaker at fixed layers chosen by preliminary nine-way probing.

Under matched conditions Whisper ranks first at every granularity. It reaches 0.489 nine-way UAR and 0.568 accuracy on read speech, and 0.767 three-way UAR. Its nine-way advantage holds across all five matched splits and all nine other representations, with mean gains of roughly 0.09 UAR over the strongest self-supervised model and 0.27 over Mimi. Because the five 70:30 splits overlap in speakers, the authors treat these paired tests as descriptive supporting evidence rather than exact inference.

Performance rises monotonically for all ten representations as the label space is coarsened, and an untuned logistic-regression probe reproduces a very similar ordering, suggesting the ranking is not an artifact of classifier tuning. Matched spontaneous-speech performance follows the same ranking (Whisper 0.431, 0.572, 0.756, 0.832 for 9-, 6-, 3-, and 2-way), which indicates that the later transfer drop reflects the style shift rather than an inherent difficulty of spontaneous speech.

Geolocation error hides large regional differences

Whisper is also the strongest matched-condition geolocation representation, reaching a median great-circle error of 148 ± 16 km across the five splits with a two-layer MLP trained on great-circle loss. For comparison, direct coordinate regression gives 175 km and the mean-location baseline gives 242 km. Adding an auxiliary regional-classification head yields no further gain (148 km; p = 0.42).

That single median conceals substantial heterogeneity. Nine-way recall correlates strongly with class size (Spearman ρ = 0.87, p = 0.003), ranging from 0.10 for South Franconian to 0.65 for Bavarian, while median geolocation error ranges from 116 to 208 km. Leave-one-region-out errors show geographic structure, but because recall tracks class size, the authors describe these maps as descriptive rather than evidence that some dialects are intrinsically harder.

The granularity results matter for interpretation: coarse three-way structure is much easier to recover than fine nine-region distinctions, so a headline number should always be read together with the label space it was computed on.

Read-spontaneous transfer breaks the ranking

The central result appears when training and testing cross styles with held-out speakers. For Whisper, nine-way UAR falls to 0.11 for read to spontaneous and 0.18 for spontaneous to read, and geolocation error rises from 148 km to 363 km, which is worse than the 242 km mean-location baseline. Self-supervised encoders degrade similarly (wav2vec2-XLSR: 0.14/0.19 UAR), whereas speaker embeddings and the codec retain more regional information (ECAPA: 0.25 to 0.23 UAR, 196 to 208 km).

Two controls rule out simpler explanations. Sentence-disjoint evaluation reduces Whisper nine-way UAR from about 0.48 to 0.40 (three-way 0.77 to 0.73), so prompt overlap accounts for part but not all of the gap. Duration matching leaves read UAR essentially unchanged (0.486 to 0.488) and transfer near 0.11, so speech amount alone does not explain the drop. Age, sex, sentence overlap, and duration controls do not account for the gap, although channel characteristics contribute.

The practical consequence is that a matched-condition score and a cross-style score answer different questions. Whisper's 0.489 and its 0.11 are not two estimates of one quantity; they measure discriminability within a style and robustness across styles respectively.

Style invariance tracks robustness better than in-domain strength

To explain the reversal, the authors measure style invariance: how little a representation shifts between read and spontaneous speech. Cross-style speaker retrieval is used as an interpretable proxy. Each spontaneous-speech speaker vector queries read-speech vectors by cosine similarity, and rank-1 accuracy is the fraction whose nearest neighbour is the same speaker (498 queries; chance ≈ 0.002). No model is fit; speaker labels only define matches.

Retrieval is strongly associated with transfer performance across the ten representations: classification (Spearman ρ = +0.74, p = 0.014) and geolocation (ρ = −0.68, p = 0.030). Speaker embeddings reach about 0.99 retrieval and the codec 0.72, whereas content-oriented representations stay at or below 0.10. The authors read this as regional information being better retained when speaker-related information is preserved across conditions.

A second pattern is a trade-off: for geolocation, lower in-domain error is associated with higher transfer error (ρ = −0.71, p = 0.022), robust to leave-one-representation-out analysis and to removing both speaker embeddings. The same trade-off is weaker and non-significant for classification (ρ = −0.37). With only ten representations from partly related families, the authors treat all four correlations as exploratory: jointly they do not survive Holm correction (smallest p = 0.014 > 0.05/4). The two central associations remain stable under leave-one-representation-out (ρ in [0.64, 0.81] for classification; [−0.85, −0.64] for geolocation) but weaken to non-significance when both speaker embeddings are removed. Auxiliary multi-task objectives did not significantly improve geolocation or transfer.

Discriminability and robustness are different targets

The paper's own framing is that the results distinguish matched-condition discriminability from robustness across read and spontaneous speech. Whisper, the self-supervised models, and Qwen2-Audio perform strongly in-domain but degrade markedly under transfer, whereas speaker embeddings and, to a lesser extent, Mimi remain more stable. For geolocation, the strongest in-domain representations are also among the least robust.

This is an association between representation properties and transfer behaviour, not a demonstrated causal mechanism. The authors present style invariance as a factor underlying robustness and motivate future work on whether speaker-oriented representations retain geographic information relevant to anonymization.

The scope is deliberately narrow: one imbalanced German corpus, with region labels treated as evaluation categories rather than sharp linguistic dialect boundaries. Read speech consists mainly of digit, number, and command strings plus phonetically balanced sentences, while spontaneous speech is a prompted monologue imagining an address to someone from the speaker's home region, so read-spontaneous transfer also crosses a register boundary. Broader validation would require additional datasets.

Evaluate accent systems in both speech styles

The decision-relevant takeaway is a change of evaluation priority rather than a new best model. If a system is meant to handle regional accent in realistic use, matched-condition accuracy on read speech is not sufficient evidence: Whisper's 0.489 nine-way UAR coexists with 0.11 and 0.18 across the two transfer directions, and its 148 km geolocation error becomes 363 km, worse than predicting the mean location.

The comparison that should guide choices is the pair of numbers, in-domain and cross-style, read together with the label granularity. A representation that looks weaker in-domain, such as ECAPA at 0.25 to 0.23 UAR and 196 to 208 km, can be the more dependable option when the speech style is unknown, and cross-style speaker retrieval is the paper's interpretable proxy for that stability.

The authors' own conclusion follows the same line: accent systems should be evaluated explicitly in both read and spontaneous speech rather than relying solely on matched-condition performance. The open question they leave is whether speaker-oriented representations retain geographic information relevant to anonymization, which is where the robustness advantage of speaker embeddings becomes a privacy concern as much as a benefit.

Sources used in this answer