How can Bayesian inference remain valid under model misspecification?

The likelihood is almost never exactly right

A standard Bayesian analysis combines a prior over parameters with a likelihood and returns a posterior through Bayes' rule. The paper states the assumptions behind that step plainly: the prior must exactly encode the user's beliefs about parameter plausibility before seeing data, and the likelihood must exactly correspond to how the data were generated, or at least to how the user believes they were generated. The author argues these conditions are almost never met. Eliciting the infinitely many probability statements a prior density makes is not feasible within the time frame of a particular problem, and Bayesian analyses increasingly involve high-dimensional or observational data, where modelling the generation process correctly is difficult.

The paper is careful about what follows from this. Analyses proceed with working priors and likelihoods that approximate the ideal ones, and that is meaningful only if the resulting posterior is not too dissimilar to the ideal posterior, meaning inferences are not driven by the simplifying functional assumptions. The paper also draws a distinction that shapes everything after it: in finite parametric Bayesian models, the influence of the prior shrinks with sample size, but the effect of model misspecification generally cannot be alleviated by collecting more data. Misspecification is defined formally as the case where no parameter value in the model class reproduces the data-generating distribution.

The posterior as a trade-off between prior and loss

The paper's central justification comes from rewriting the posterior as the answer to an optimisation problem rather than as a consequence of a correct likelihood. The posterior is the density that minimises the expected negative log-likelihood under the candidate posterior plus the Kullback-Leibler divergence from that candidate to the prior. The first term measures closeness to the data through the log-score; the second measures closeness to prior beliefs.

This reframing matters because the optimisation problem never asks whether the model is true. It asks which posterior beliefs best trade off fit to observed data against fidelity to the prior, which gives the posterior a decision-theoretic foundation. The paper explains why each ingredient is chosen: the Kullback-Leibler term yields a coherent update, in which updating on one part of the data and then using the result as a prior for the rest gives the same answer as updating on both parts at once; the negative log-likelihood is the only local proper scoring rule, where locality means the score depends only on the model density at the observed data, and propriety means the score is minimised when the model matches how the data were generated. On this reading the posterior remains valid under misspecification provided one accepts that the optimisation problem adequately measures the suitability of candidate posterior beliefs.

Uncertainty quantification breaks before point estimation does

The variational justification covers the location of the posterior, not the width of its intervals. Under a correctly specified model and standard regularity conditions, the Bernstein-von Mises theorem makes the posterior asymptotically Gaussian, centred at the data-generating parameter with covariance equal to the inverse Hessian. The posterior then agrees asymptotically with the sampling distribution of the maximum likelihood estimate, so credibility intervals are also confidence sets.

Under misspecification the asymptotic Gaussian shape survives, but its centre and spread change meaning. The posterior concentrates at the parameter in the model class whose Kullback-Leibler divergence from the data-generating distribution is smallest, with covariance again given by the inverse Hessian at that parameter. The maximum likelihood estimator is centred at the same parameter but has a sandwich covariance. The two no longer agree, so posterior credibility intervals are not confidence sets, and the paper describes the posterior's uncertainty quantification as not robust to disagreements between model and data. This is the sharpest limitation in the paper: the variational argument licenses the posterior as a belief update, while the coverage guarantee that many users assume is lost.

A learning rate can widen the posterior toward coverage

One response is to generalise Bayes' rule by raising the likelihood to a positive power, usually written as a learning rate. Values below one produce a more diffuse posterior and values above one a more concentrated one; the literature the paper reviews focuses on values below one to guard against under-coverage and over-confidence. The intuition given is that a rate below one down-weights the likelihood relative to the prior, so for a fixed prior the posterior becomes more robust to disagreement between likelihood and data.

The paper lists named variants that fall under this heading, including tempered, fractional and power posteriors, the safe-Bayes paradigm for improved empirical risk in linear modelling, coarsened posteriors that condition on data in a neighbourhood of the exact observations rather than on the exact data, and a quasi-likelihood specification for linear regression that fixes only the mean and variance functions. It also notes how the rate can be chosen: by matching experimental information, by calibrating to the Bayesian bootstrap, or by targeting coverage under repeated sampling. The paper adds a limit on what a single rate can achieve: adjusting the curvature of the log-likelihood or replacing it with a quadratic form built from its gradient and estimated gradient variance can make the posterior's asymptotic variance agree with the sandwich covariance, but a single learning rate cannot generally do this for all parameters at once.

The parameter the posterior learns about is tail-sensitive

Even when the posterior behaves predictably, the parameter it converges to may not be the one a user wants. The paper explains that the Kullback-Leibler-closest parameter is generally not robust to changes in the tails of the likelihood model or to outlying observations. The reason is that minimising this divergence is equivalent to minimising the logarithmic score in expectation over the data-generating distribution, and the negative log-likelihood diverges to infinity as the model density at an observation goes to zero. Slightly changing the model's tails, or removing an outlier, can therefore produce very different inferences.

The paper contrasts this with traditional Bayesian practice, which favours heavy-tailed models: replacing a Gaussian with a Student's t, or using model classes with flexible heavy-tailed behaviour. It cites an analysis of linear regression with Student's t errors finding cases where those tails were still not heavy enough to ignore outliers, which motivated an even heavier-tailed log-Pareto-tailed normal distribution. The point is not that heavy tails fail, but that the default target of Bayesian learning under misspecification carries a specific sensitivity that users should recognise.

Generalised updates trade the likelihood for a chosen loss

Because the variational form does not require the parameter to index a probability model, the log-likelihood can be replaced by a general loss function that scores how suitable a parameter is for the observed data. The resulting update is often called a Gibbs posterior. The paper traces uses of this form: misclassification error in logistic regression, where it was proved to improve predictive performance under misspecification and later extended beyond classification; M-estimators that swap squared error for Huber's or Tukey's loss in linear regression; Bayesian M-estimation formalised further, including frameworks that condition on statistics that are not sufficient and methods to select the M-estimation hyperparameters; and an application that avoids specifying a model when estimating a treatment effect's minimum clinically important difference.

A second family chooses losses derived from divergences other than the Kullback-Leibler divergence. Early examples used the Hellinger or total-variation distance but required a non-parametric density estimate, which prevented scaling beyond low-dimensional problems. More convenient are divergences whose minimisation can be replaced by minimising a proper scoring rule, including beta-divergence with the Tsallis score, Fisher's divergence with the Hyvärinen score, and gamma-divergence. A third family uses discrepancy measures such as maximum mean discrepancy and kernel-Stein discrepancies, which compare simulations from the model with the data and so allow inference on models with intractable densities while providing robustness. The paper notes that many of these losses would still learn the data-generating parameters if the model were correct, at some cost in statistical efficiency, but under misspecification they learn about different parameters with different robustness properties.

A loss function has no built-in probability scale

Swapping the log-likelihood for a loss creates a problem the paper states directly: the generalised update provides no guarantee that its uncertainty quantification is meaningful. Probability densities must integrate to one, but loss functions carry no such constraint. Multiplying a loss by a constant other than one leaves the parameter it learns about unchanged while changing the uncertainty quantification, so the scale of the loss has to be calibrated.

The common fix places a learning rate in front of the loss, mirroring the tempered posterior. Under regularity conditions a Bernstein-von Mises result holds for this update, but its asymptotic variance is the inverse of the second derivative of the loss, which does not agree with the repeated-sampling asymptotic distribution of the loss minimiser. Work in this area has therefore concentrated on setting the rate to better match repeated-sampling asymptotics, or on modifying the loss so that it produces the same asymptotic distribution. The paper observes that this literature generally does not distinguish between using the log-score with a misspecified model and using a general loss function, which is why the calibration question applies across both.

Fix the interval before chasing a better point estimate

The paper's own ordering of priorities is the useful takeaway. Its variational argument is what keeps Bayes' rule defensible when the likelihood is wrong, and that argument concerns which beliefs the posterior represents, not how tightly it concentrates. The failure it documents is specific: under misspecification the posterior's asymptotic covariance is the inverse Hessian at the Kullback-Leibler-closest parameter, while the maximum likelihood estimator's is the sandwich covariance, so credibility intervals are not confidence sets. A user who needs interval coverage should therefore treat the learning rate or a modified loss as the operative choice, and should note that a single rate cannot generally match the sandwich covariance for all parameters at once.

A user who instead needs robustness to outliers or tail behaviour faces a different choice, because the default target itself is tail-sensitive: the negative log-likelihood diverges as the model density at an observation approaches zero, so small changes in tails or a single removed outlier can move inferences substantially. That is the case for a divergence-based loss or a discrepancy measure, accepting some loss of efficiency if the model happens to be correct. The paper closes by noting that a better-specified model would remove many of these issues, but that flexible modelling should not come at the expense of the structure and interpretability a parametric model provides, and that misspecification cannot be ruled out even in very complex models.

Sources used in this answer