One weight per model, learned from predictions
OMA treats each candidate model as a fixed source of class probabilities over a shared label space. It forms an ensemble distribution by taking a weighted average of those probabilities, with one nonnegative scalar coefficient per model, and the coefficients sum to one. Only these coefficients are learned; every foundation model stays frozen, and once predictions are cached the method needs neither model internals nor backpropagation through the models. The number of optimized parameters therefore depends on how many candidate models there are, not on their size.
The learning signal is predictive entropy on an unlabeled target set: the objective rewards ensemble predictions that are concentrated rather than diffuse. The paper shows why this cannot be used raw. Because entropy is concave and the mixture is linear in the weights, the unregularized objective over the simplex always admits a single-model optimum, so it does not inherently favor a genuine ensemble, and the selected model need not be the most accurate or best calibrated. Regularization is therefore central rather than cosmetic.
Two regularizers pull weights toward a reference
The paper offers two ways to constrain adaptation. The first is a prediction-space KL penalty toward a reference distribution built from the coordinate-wise median of the candidate probabilities at each target input; this reference is computed once from frozen predictions and stays fixed. It penalizes assigning little probability to classes the reference supports, and it uses only target predictions, not labels. The second is a weight-space L2 penalty toward a fixed reference vector, which may be uniform, approximate Bayesian model averaging (BMA) weights, or weights built from source-domain performance.
The distinction matters for what each penalty can do. The median reference can inherit errors that the candidate models share, and it is not assumed to be calibrated. The source-derived reference reflects model quality on source data, which the paper notes need not reflect target-domain quality under distribution shift. Neither penalty directly optimizes calibration, so the paper measures accuracy and expected calibration error separately instead of treating lower entropy as evidence of better-calibrated predictions.
Text tasks show calibration gains over uniform averaging
On four GLUE text classification tasks (MRPC, RTE, CoLA, SST-2), the paper fine-tunes four models — two BERT-base variants and two funnel-transformer variants — and applies OMA with the KL regularizer. The reported result is a better accuracy–calibration trade-off than uniform averaging. Across all four datasets, OMA tuned for expected calibration error is better calibrated than output averaging and than every base model, with results reported as mean and standard deviation over five seeds.
The paper also reports that OMA outperforms output averaging and a single best model, and it is careful about comparisons: Model Soups and Fisher merging use different architectures, so the paper treats direct comparison as unreasonable and instead emphasizes improvement over each best model. One observation complicates any simple reading of the weights: the learned coefficients at best calibration do not track individual model accuracy, so higher-accuracy models do not necessarily receive higher weights.
Image gains are small and dataset-dependent
For image classification, the paper uses eight OpenCLIP vision models as frozen feature extractors with a trainable linear classifier each, forming a 16-model pool of eight MAP estimates and eight zero-shot models. BMA over the linear classifiers supplies an optional source-derived reference vector, which lets the study examine how source-informed weighting interacts with target entropy under distribution shift. Evaluation covers ImageNet-1K and five shifted datasets: ImageNet-V2, ImageNet-R, ImageNet-Sketch, ImageNet-A, and ObjectNet.
The accuracy changes relative to output averaging are modest and not uniformly positive. With the combined zero-shot and MAP pool, OMA improves top-1 accuracy by 0.37% on ImageNet-1K, 0.26% on V2, 0.29% on R, 2.15% on ImageNet-A, and 2.10% on ObjectNet, but drops 0.11% on Sketch. With only MAP estimates, OMA gains 2.22% on ImageNet-A while losing 0.11% on Sketch. With only zero-shot models, it loses 0.29% on V2 and 0.05% on R. The paper's own summary is that combining both pools gives generally better calibration and accuracy than uniform merging, except on Sketch, where calibration is lower-error but accuracy is worse.
Lower entropy is not the same as calibration
The paper's central caution is that minimizing predictive entropy can favor confidently incorrect models, because increased confidence need not imply improved accuracy or calibration. Regularization can reduce expected calibration error in the evaluated settings, but the paper states that the benefit depends on the task and optimization configuration. Since target labels are unavailable, regularization itself does not guarantee target-domain calibration.
This is why the reported calibration results are framed as selected configurations rather than a general property. The paper reports lower expected calibration error in those configurations and better performance–calibration trade-offs than model output averaging, while explicitly declining to identify reduced entropy with calibrated predictions. Readers comparing methods should therefore keep accuracy and calibration as separate measures rather than reading one as a proxy for the other.
Cheaper than fine-tuning, but not free at inference
The paper positions OMA as a low-cost alternative to fine-tuning and parameter-space merging, and it reports concrete computation comparisons. Fine-tuning a single model in Model Soups takes 3 days on an A100, while BMA on an RTX 4090 needs 86.1% less than Model Soup. Comparing fine-tuning all 58 models against eight models in BMA, Model Soup takes 4176 hours and BMA takes 80 hours, a 98.08% decrease in total runtime. The paper also notes comparable accuracy in that comparison: Model Soup reaches 90.94 on ImageNet-1K, while OMA using BMA weights reaches 89.19.
The cost structure is not one-sided. Output merging, including OMA, adds inference-time computation relative to parameter merging, and the paper acknowledges that all output-merging methods share this burden, pointing to quantization and pruning as possible remedies. The experiments were designed to run on a single 24 GB RTX 4090 for image tasks and a single 16 GB RTX A4000 for text tasks, which is part of the paper's accessibility argument.
When target-adaptive weighting is worth the trade
The paper's evidence supports a specific priority rather than a blanket recommendation. OMA is most clearly useful when candidate models are heterogeneous or their parameters are inaccessible, so parameter-space merging is unavailable, and when unlabeled target predictions are available for adaptation. In that setting, the learned weights can respond to the target distribution in a way uniform averaging cannot, and the paper's strongest accuracy gains appear on the shifted ImageNet-A and ObjectNet pools where zero-shot and adapted models are combined.
The same evidence sets the boundary. Gains over uniform averaging are often fractions of a percentage point, and on some datasets and pools OMA is worse on accuracy even when calibration improves. Because the method optimizes an unlabeled entropy surrogate with a regularizer that does not target calibration directly, the paper's own framing is that regularization controls adaptation rather than guaranteeing better-calibrated or more accurate predictions. The practical reading is to compare OMA against uniform averaging on the specific target distribution and to report accuracy and calibration side by side, since the paper's results show the two can move in different directions.
Sources used in this answer
Revisiting Bayesian Model Averaging in the Era of Foundation Models
OMA learns one nonnegative weight per frozen model by minimizing regularized predictive entropy on unlabeled predictions; it beats uniform output averaging on accuracy in most reported image pools and on calibration in selected text configurations, but gains are small and dataset-dependent.