Do Cantonese-adapted language models better predict Cantonese reading?

Adaptation depth matters more than adaptation itself

The paper asks whether training a language model specifically on Cantonese makes its probability estimates better at predicting how people actually read Cantonese. The answer is conditional: the model with extensive Cantonese continued pretraining and instruction tuning, CantoneseLLM-7B, produced the largest improvements in predicting fixation durations, while a model with only light Cantonese adaptation, JED351, showed no general advantage over the base model it was derived from.

That split is the study's central result. The authors describe it as a qualified association rather than a uniform benefit of Cantonese adaptation: more extensive Cantonese-specific training can coincide with stronger predictive fit, but adaptation by itself does not guarantee it. They also caution that the two model pairs differ in architecture and scale, so the comparison reflects broader adaptation strategies rather than one isolated training ingredient.

Four measures separate anticipation from surprise

The study derives four information-theoretic quantities from each model for every target word, using only the preceding within-sentence context. Lexical surprisal is the negative log probability of the word that actually appeared, so an unexpected word gets a high value. POS surprisal applies the same logic to the word's syntactic category, aggregating model probabilities over candidate words that share that category in the Cifu Cantonese lexicon.

The other two measures concern uncertainty rather than the observed word. Entropy before is the spread of the model's next-token distribution before the target is seen, capturing how uncertain the model was in advance. Entropy reduction is entropy before minus entropy after the target is added, so it records how much observing the word sharpened the model's expectations about what comes next. The authors treat these as complementary views of pre-target anticipation, target-level expectation, and post-target updating, not as one-to-one mappings onto cognitive stages.

This distinction matters for reading the results, because the four measures do not rank the models the same way. Lexical surprisal and the joint four-metric model favor CantoneseLLM-7B, followed by Qwen2.5-7B, CKIP, and JED351, whereas entropy reduction favors CKIP. A reader who collapsed these into a single notion of model quality would miss the paper's main methodological point.

Eye movements from 30 native Cantonese readers

The human data come from the Cantonese portion of MCFIX, a parallel Mandarin and Cantonese eye-tracking corpus collected from 30 native speakers. Participants read in two conditions: natural reading for comprehension (NR) and a task where they searched the text for specified information (TSR). The word-level outcomes analyzed are first fixation duration (FFD), second fixation duration (SFD), and total fixation duration (TFD).

Prediction used CatBoost regression with five-fold cross-validation at the word level, repeated across split seeds 42 to 46, with the same row identities and fold assignments for all four models. The no-LLM baseline included reading task, POS dummies, distance to root and head, dependency depth, neighbourhood size, current and previous frequency, current and previous syllable count, and relative word position. The authors then added one LLM metric at a time, all four together, and ran leave-one-block-out ablations. The primary measure was mean absolute error (MAE), so a positive change means lower prediction error in milliseconds.

Because FFD, SFD, and TFD have different baseline error scales, the authors also report a baseline-normalized percentage improvement. This prevents the larger numerical scale of TFD from making its raw gains look more important than they are.

LLM measures add modest value beyond linguistic predictors

The four measures do carry information beyond conventional predictors, but the increment is small. Each metric improves prediction in at least some model–outcome combinations, and the joint four-metric model outperforms every corresponding single-metric model. Yet even the largest baseline-normalized joint improvement stays below 2%, which the authors call incremental rather than transformative.

The ablation results explain why. Current syllable count and relative word position remain the strongest predictor blocks across models, while the LLM-derived measures generally sit in the middle of the ranking. Their conditional contributions are smaller than their single-metric additions, consistent with partial overlap among the four measures. Within-model correlations support that overlap: entropy before and entropy reduction correlate at 0.55 to 0.67 across models, while lexical and POS surprisal correlate only weakly.

For readers weighing whether to add model-derived surprisal to an eye-tracking analysis, the practical reading is that these measures complement established lexical and positional predictors rather than replace them, and that using several theoretically distinct measures is more informative than picking one.

CantoneseLLM-7B leads its base on most measures

In the large-model pair, CantoneseLLM-7B exceeds Qwen2.5-7B on the joint four-metric model by 0.132, 0.066, and 0.430 ms for FFD, SFD, and TFD respectively, with positive contrasts across all five split seeds. Entropy before shows similarly consistent advantages, and lexical and POS surprisal also favor CantoneseLLM-7B on average, though less uniformly across seeds. Entropy-reduction contrasts remain close to zero.

The raw single-metric table shows the same ordering. For lexical surprisal on TFD, the mean improvement is 0.683 ms for CantoneseLLM-7B, 0.599 ms for Qwen2.5-7B, 0.249 ms for CKIP, and 0.229 ms for JED351, with all five seeds positive in every case. For the joint model on TFD, the values are 1.251, 0.821, 0.651, and 0.409 ms respectively.

The authors explicitly warn against reading this as a scale effect. They note that more training and larger models have been reported to worsen surprisal fit in controlled English studies, and that similar results appeared for other languages. The advantage here is tied to the specific adaptation strategy, not to size alone.

Light Cantonese fine-tuning does not beat its base

The small-model pair points the other way. JED351 descends from CKIP GPT-2 Tiny Chinese; its intermediate checkpoint patched the tokenizer and embedding matrix with Cantonese characters, and the version used here was then fine-tuned for ten epochs on roughly 50 MB of Cantonese Wikipedia text. Despite that Cantonese vocabulary expansion and fine-tuning, JED351 does not show a general predictive advantage over CKIP.

For the joint four-metric model, the JED351–CKIP contrast is negative for FFD, SFD, and TFD, providing no evidence of an overall advantage from the lighter adaptation strategy. The single-metric picture is mixed rather than uniformly worse: entropy reduction favors CKIP clearly, with a TFD improvement of 0.362 ms versus 0.159 ms for JED351, while lexical surprisal is close between the two.

The contrast between the two pairs is the paper's most decision-relevant comparison. A small amount of Cantonese text and vocabulary patching did not improve psycholinguistic alignment, whereas extensive continued pretraining plus instruction tuning did. Because each pair bundles several changes at once, the study cannot attribute the difference to any single component such as vocabulary size, data volume, or instruction tuning.

Which fixation measure and reading task you model changes the ranking

The strongest raw improvements often appear for TFD, but that partly reflects its larger baseline error scale. After baseline normalization, the joint four-metric improvement is largest for FFD in every model. The authors connect this to the different processing stages the outcomes index: FFD captures the first fixation, while second-pass measures such as SFD can be more sensitive to later structural demands.

Reading task also matters, most clearly for FFD. All four models show larger mean joint-model improvements in TSR than in NR for this measure, which the authors say is compatible with goal-directed information search increasing the relevance of lexical expectations during initial processing. That pattern does not extend uniformly to SFD or TFD: CantoneseLLM-7B shows larger TSR gains for these later measures, whereas the other models often show larger NR gains. The task-stratified numbers come from models trained jointly on NR and TSR observations, not from separately trained task-specific models.

The takeaway is that a claim about which model predicts reading best is incomplete without naming the fixation measure and the reading condition. The paper's own rankings shift across those dimensions.

The comparison bundles several training changes at once

The central limitation sits inside the design rather than at its edges. JED351 differs from CKIP in vocabulary, embeddings, and Cantonese Wikipedia fine-tuning, while CantoneseLLM-7B differs from Qwen2.5-7B in continued Cantonese pretraining and instruction tuning; the two pairs also differ in architecture and scale. The results therefore describe broader adaptation strategies, not the causal effect of any single training component.

Two further constraints bound the measurement. The extraction pipeline tokenizes context and target separately, imposing a lexical boundary that differs from natural concatenated tokenization on 246 of 5,072 Qwen-family rows. And entropy values are not directly comparable across model–tokenizer systems with different continuation spaces, which limits how far the entropy-based rankings can be pushed. The human data are also limited to one literary translation and one participant sample.

Given those boundaries, the paper's most useful priority is the contrast it can support: extensive Cantonese continued pretraining plus instruction tuning coincided with better fit on lexical surprisal and the joint model, while light vocabulary patching and Wikipedia fine-tuning did not. Anyone choosing a model to predict Cantonese reading should check the specific measure and reading condition they care about, because entropy reduction ranked the models differently from every other measure in this study.

Sources used in this answer