Does layer selection bias speech depression detection performance estimates?

Naive layer selection inflates AUC by up to 0.09

The paper audits a routine practice in clinical speech modeling: freezing a pretrained speech encoder, extracting representations from each of its layers, and reporting the layer that performs best. When that choice is made on the same cross-validation folds used to report performance, the layer choice becomes part of the evaluation procedure rather than part of the model.

On DAIC-WOZ, with repeated nested cross-validation across five deep encoder families, naive best-of-25-layer probing inflates AUC by 0.05–0.09 points (mean 0.07). The largest gap appears for wav2vec2, where AUC drops from 0.65 under naive selection to 0.56 under nested selection. The paper stresses that this absolute difference changes interpretation: 0.65 suggests meaningful predictive performance, while 0.56 is only slightly above chance.

The audit covers wav2vec2, data2vec, WavLM, Whisper, and HuBERT Large variants of comparable scale (about 0.3B parameters). For each encoder the authors extract the input embedding plus the outputs of all 24 transformer blocks, giving 25 layers of 1024-dimensional representations, each mean-pooled into one vector per participant and layer.

Selection alone produces above-chance AUC

To separate selection from signal, the authors run 300 signal-free simulations with 25 Gaussian feature sets unrelated to the label. Naive best-of-25 selection systematically yields above-chance apparent AUC (0.64±0.03), exceeding 0.55 in every trial, while nested selection stays at chance (0.50±0.07).

A second control keeps the real latent representations but shuffles the labels. Across 500 permutations, preserving inter-layer correlation, dimensionality, and participant structure, naive selection still produces above-chance AUC (mean 0.59) whereas nested selection does not surpass chance (mean 0.50; paired ΔAUC +0.08, 95% CI [0.08, 0.09]). Because the labels carry no information, the residual AUC can only come from the selection step.

The paper also shows the bias is not a fixed constant. Apparent AUC rises monotonically with the number of candidate layers K (0.50→0.59 for K=1→25), consistent with the classical σ√(2 log K) growth of an expected maximum, and the optimism increases as sample size falls (Δ from 0.08 at n=140 to 0.14 at n=40) while nested selection stays close to chance. The authors read this as isolating selection, not signal, as the cause.

The bias replicates in a second language

The same shuffled-label setup is repeated on the Italian Androids corpus (116 participants, 62 diagnosed with depression). Naive selection still reaches above-chance AUC (0.58) while nested selection remains at chance (0.50), a Δ of +0.08 with CI [0.08, 0.08].

Notably, this happens even though the encoders genuinely separate depression on the original, unshuffled labels: pooled AUC is 0.96 under naive and 0.93 under nested selection. The authors use this to argue that selection optimism is not specific to DAIC-WOZ, to English, or to low-signal regimes.

Androids is restricted to this control for two stated reasons: its non-English speech would make the emotion-encoder affect comparison an unfair cross-language test, and it lacks the PHQ-8 item labels needed for the symptom-level analysis.

Leakage-free comparison changes the ranking

Once layer selection is nested, the paper compares representation families on DAIC-WOZ cross-validation (n=140, mean±std over 10 repetitions). A 119-dimensional interpretable acoustic representation, Affect+Timing+eGeMAPS, reaches AUC 0.75±0.03 (95% CI [0.68, 0.84]; shuffled-label null 0.52±0.07), the highest cross-validation AUC among the considered systems, while requiring only about 120 trainable parameters.

The components matter differently. Affect alone attains 0.65±0.02 and adding timing gives only a marginal increase, but adding eGeMAPSv02 features raises performance to 0.75±0.03. The paper interprets this 0.65→0.75 gain as voice-quality features, including H1-H2, supplying complementary information beyond affect, while timing, near-null on binary AUC, carries symptom-specific signal instead.

Under nested layer selection the best pretrained encoder, data2vec, reaches 0.66±0.05, comparable to affect alone but below the best interpretable setting. Qwen2-Audio-7B does not exceed the interpretable representations either: zero-shot evaluation reaches 0.60, and an additional frozen-encoder probe using its latent embeddings performs worse (0.56±0.03). Kyutai Mimi is weakest, with 0.47±0.03 for continuous SEANet latents and 0.58±0.02 for discrete RVQ-token statistics.

The paper also reports what does not help. Four deep heads (transformer, cross-attention, multitask, supervised-contrastive) over encoder-layer tokens did not improve on simpler representations; the best, cross-attention with data2vec, reached only 0.60±0.06 under 10-repeat cross-validation, with high-variance training. Adding 38-dimensional affect-dynamics trajectory descriptors reduced performance from 0.75 to 0.70, and multitask symptom learning and SMOTE likewise did not improve it. An exhaustive sweep over all 15 feature subsets confirmed Affect+Timing+eGeMAPS as uniquely CV-optimal (AUC 0.75; next-best Affect+eGeMAPS, 0.69).

Held-out test scores are high but unstable

On the 47-speaker held-out test set, the fixed affect+logistic model reaches AUC 0.89 [0.76, 0.98], showing that a high individual test estimate can coexist with substantial ranking instability. The cross-validation ranking does not carry over: the CV-best Affect+Timing+eGeMAPS drops from 0.75 to 0.72, while Whisper, last among the pretrained deep encoders in CV (0.54), reaches 0.77 on test. Qwen2-Audio zero-shot rises from 0.60 in CV to 0.84 on test, although the same output logits varied substantially (0.89/0.76) by classifier configuration alone.

Across all systems, CV and test rankings show only weak agreement (Spearman ρ=0.25) and are strongly anti-correlated among the five pretrained encoders (ρ=−0.97). The paper treats these rank inversions as evidence of the instability of model comparisons on a small held-out test set, and as motivation for using repeated nested cross-validation as the primary basis for ranking systems.

The test column is a single, config-unstable held-out estimate with roughly ±0.13 AUC, so it should not be read as a stable leaderboard.

Symptom-level bias and calibrated severity

The downstream models decompose into a PHQ-8 symptom profile at no additional cost, letting the authors extend the audit to a finer-grained level. Pooled across the five encoders, naive selection inflates AUC for all eight symptoms (mean 0.63 vs. 0.56 for nested). The nested protocol performs worse than the Affect+Timing+eGeMAPS reference on all eight symptoms (mean AUC gap 0.09, range [0.02, 0.17]), whereas naive selection exceeds that reference on 4 of 8 symptoms (sleep, fatigue, appetite, psychomotor; mean gap 0.04, range [0.01, 0.09]).

Using weighted-F1 with score>0 binarization, the symptom predictions match or exceed the CNN/LSTM approach of prior work (0.65 vs. 0.59; better on 6 of 8 symptoms). Summing per-item cumulative-ordinal expectations yields severity with 4-class κ 0.46±0.03, versus 0.34 for direct regression. The probabilities are over-confident (ECE 0.15±0.02), improved by Platt scaling to 0.09±0.02.

The authors argue that this per-symptom profile, coherent severity, and calibrated probabilities are what a screening tool needs but a black-box latent embedding score lacks.

Where the conclusion holds and stops

The selection-bias finding replicates across two clinical speech corpora and two languages, but the broader representation comparison and the symptom analyses remain single-corpus, based on DAIC-WOZ. The Androids replication is limited to the shuffled-label control, so it supports the bias claim rather than the ranking claim.

The robustness check on sex does not explain the main result: although the features predict sex with AUC 1.0 and sex prevalence differs between depressed and non-depressed participants (0.39 vs. 0.23), depression detection remains strong within both females and males (AUC 0.73/0.77) and is essentially unchanged after residualizing the features for sex (0.76→0.75).

The paper's own conclusion is a protocol recommendation: nested model-selection protocols should be standard when probing encoder layer representations on small clinical speech datasets. The evidence supports that recommendation specifically for the audited setting of frozen encoders, 25 candidate layers, and small clinical corpora, not as a general claim about all representation learning.

What to trust when ranking systems

The paper's most decision-relevant contrast is between two numbers that look like the same measurement but answer different questions. A naive best-of-25 AUC of 0.65 for wav2vec2 and a nested AUC of 0.56 both describe the same encoder and the same folds, yet only the second estimates how the layer choice would transfer to new participants. When the gap is 0.09, the practical difference is between a system that looks meaningfully predictive and one that is barely above chance.

The second contrast is between cross-validation ranking and held-out test ranking. With Spearman ρ=0.25 overall and ρ=−0.97 among the five pretrained encoders, a single 47-speaker test score cannot settle which representation family is better; the paper's own example is Whisper, last in CV (0.54) and at 0.77 on test.

The third is between complexity and a compact task-aligned representation. Under leakage-free selection, the 119-dimensional Affect+Timing+eGeMAPS model reaches CV AUC 0.75 with about 120 trainable parameters, above data2vec (0.66), Qwen2-Audio zero-shot (0.60), and both Mimi variants (0.47 and 0.58), while also producing per-symptom outputs with 4-class κ 0.46±0.03 and Platt-scaled ECE 0.09±0.02. For readers comparing speech representations on small clinical datasets, the paper's priority is therefore clear: fix the selection protocol first, then compare families, and treat symptom-level interpretability and calibration as part of the comparison rather than an afterthought.

Sources used in this answer