How well calibrated are language models' confidence in their relational knowledge?

Calibration asks whether confidence matches accuracy

Knowledge probes usually ask how much a model knows: they score accuracy and precision on cloze-style factual statements such as "Balach Marri died in [MASK]." This paper argues that such scores describe attainment but not reliability. A model that answers correctly 60% of the time while stating 90% confidence is accurate yet poorly calibrated, and it gives no usable signal when it is about to fail.

Calibration compares a stated confidence score with the observed probability of being correct. In a perfectly calibrated system, a confidence of 0.8 means the answer is right 80% of the time. The paper measures calibration error with ACE, the average gap between confidence and accuracy, and with the Brier score, which penalizes both overconfidence and underconfidence. The authors treat ACE below 0.1 as well calibrated, meaning confidence deviates from accuracy by at most 10 percentage points; a Brier score of 0.25 corresponds to a random baseline that always predicts 0.5.

Three ways to read a model's confidence

The study evaluates confidence in a closed answer-set setting, where each statement comes with a fixed list of candidate answers. It compares three modalities. Intrinsic confidence comes from the model's raw log-probabilities over those answers. Paraphrase consistency asks whether the model gives the same answer across semantically equivalent rephrasings of the statement. Linguistic confidence injects explicit uncertainty markers, either verbal ones such as "possibly" and "certainly" or numerical ones such as "I'm 0% confident that...".

The intrinsic estimates are built from a single template. The base score, CBase, uses the model's probability for the answer. The margin score, CMargin, uses the gap between the top answer and the next candidate, which the authors expect to separate confident from uncertain predictions more sharply. The paraphrase estimates aggregate predictions across five templates per relation, and the entropy variant converts the spread of answers across templates into a confidence score.

Paraphrase consistency gives the best-calibrated scores

Across the models tested, the average paraphrase-consistency estimate, CAverage, outperformed every other approach on both ACE and Brier score. For opt-6.7b it reached an ACE of 0.024, and for Qwen3.5-9B(IT) an ACE of 0.034, both close to perfect calibration. The calibration curves for these estimates deviate only slightly from the identity line where confidence equals accuracy.

The margin estimate is the strongest single-template option. With CMargin, gemma-7b, Llama-3.1-8B(IT) and Qwen3.5-9B(IT) all fall below an ACE of 0.05. CBase, by contrast, is markedly overconfident and does not produce enough low-confidence scores to cover the full confidence range. The authors note that CMargin can push the curve above the identity line at low confidence, making the model underconfident there, which they consider preferable to overconfidence in practice. The other paraphrase estimates, simple answer agreement and entropy, are badly miscalibrated and discriminate poorly, which the authors read as evidence that agreement alone does not capture confidence.

Masked models are more overconfident than causal models

The analysis covers 21 causal and 8 masked language models. To separate architecture from accuracy, the authors pair causal and masked models with similar accuracy and compare their calibration. Across all five matched pairs, causal models are substantially better calibrated, with differences often around 0.1 to 0.2 in Brier score for CMargin and CAverage. For both estimates, every causal model reaches a Brier score of 0.2 or lower, while no masked model does.

The authors attribute this gap to training objective. Masked models do not explicitly model a sentence-level log-likelihood during pre-training, so the study relies on pseudo log-likelihoods. The authors believe this absence is the main reason for the large calibration discrepancy, while noting that the scoring is still sufficient to distinguish correct from incorrect answers on the probe. Scale matters too: for most model families, larger models show lower ACE and Brier scores, and all models above 1B parameters reach an ACE of 0.1 or lower with their best estimate. The newer gemma-3 models are less accurate and worse calibrated than the original gemma family, and xlm-roberta-large is the outlier among masked models, which the authors link to its multilingual training data.

More answer options improve calibration

The paper also varies the number of answer options, sampling subsets of the 60 options used in the 1:1 relations of the BEAR dataset. For causal models smaller than 1B parameters, adding distractors significantly reduces calibration error. Both average confidence and accuracy fall as options increase, but confidence falls faster, narrowing the gap that ACE measures. In larger causal models and in masked models the effect is weaker but still present.

The pattern is not uniform at low option counts. For Qwen3.5-4B and bert-base-cased, accuracy drops faster than confidence when there are few options, which raises ACE. After roughly 15 to 20 options, ACE stabilizes and larger models gain no further calibration benefit. The authors conclude that robust confidence estimation needs a fairly high number of distractors, at higher computational cost. They also use this to explain why the Wiki-FACTOR dataset shows much larger calibration errors: it offers only four options. When BEAR is sampled to five options, the accuracy-to-ACE relationship nearly matches FACTOR, for example gpt2-large at 48.97% accuracy with an ACE of 0.33 on BEAR versus opt-6.7b at 49.33% accuracy with an ACE of 0.323 on FACTOR.

Injected uncertainty markers help inconsistently

Adding verbal and numerical confidence expressions to the statements produced mixed results. Verbal weakeners such as "possibly" improved calibration more consistently than numerical weakeners such as "I'm 0% confident...", which had smaller and more variable effects across models. The authors suggest that numerical expressions are less common in training data and therefore less reliably interpreted as signals of uncertainty.

Instruction tuning did not systematically shift log-likelihoods in line with the intended meaning of the injections. For gemma-3-4b-it and gemma-3-12b-it, the strengthener "certainly" actually reduced overconfidence and lowered ACE. For the base gemma-3 models, "possibly" moved the calibration curve closer to the identity line, while "certainly" moved it further away. The authors present these as inconsistent effects rather than a dependable calibration lever.

Where this calibration probe stops applying

The probe is a diagnostic for closed answer-set scenarios, and the authors state that its calibration error values are unlikely to transfer directly to more practical settings. Real use often lacks predefined answer options, and paraphrase consistency in particular requires multiple formulations of the same statement to compute. The setup also captures output uncertainty only; unreliable responses can also come from vague prompts, which this framework does not measure.

A further boundary is relation type. BEAR contains no N:M relations, so quantifying calibration with respect to the top answer is sufficient here. A complete reliability assessment would need to account for such relations and evaluate calibration over all answer options, for example with a Brier score over one-hot ground-truth labels or static calibration error. Within those limits, the paper's practical priority is clear: when a closed answer set is available, prefer confidence built from paraphrase consistency, and expect masked-model confidence to overstate correctness more than causal-model confidence does.

Sources used in this answer