A shared subspace inside each architecture
The universal weight subspace hypothesis says that networks of the same architecture, trained on different tasks and data, end up using a common set of directions in weight space. The paper calls this shared structure the Universal Subspace. The claim is not that two trained networks have identical weights; it is that their weights can be described with a small number of shared principal directions per layer.
The authors frame this as a parameter-level claim, distinct from earlier universality work that focused on representations or circuits. Their unit of analysis is the weight matrix of a layer, decomposed across a pool of independently trained models. A subspace is called universal when directions estimated from some models also describe models that were not used to estimate it.
LoRA adapters concentrate in 16 directions or fewer
The largest single test uses 500 Mistral-7B LoRA adapters trained on 500 natural instruction tasks. LoRA updates are a demanding case because they encode task-specific directions and show limited alignment with the pretrained weights. Even so, the authors report that most information across all 31 layers is concentrated in 16 or fewer directions.
The paper then reconstructs LoRA parameters for randomly chosen seen and unseen tasks by projecting onto that subspace, and reports robust performance for both. It also reruns the analysis with the leftover components to check that the retained directions, not the discarded ones, carry the task-relevant structure. This is the clearest demonstration in the packet that a small shared basis can stand in for many task-specific adapters.
Reconstruction accuracy tracks retained variance
The ResNet-50 experiments show how much performance depends on how much variance is kept. Nine models trained from scratch on different datasets produced a low-quality subspace estimate, with a suitability score of q_all = 0.0183. Direct reconstruction without fine-tuning reached 77.91% mean top-1 accuracy at an 80% variance threshold, against 81.00% for the original models. At 95% variance, reconstruction reached 80.85%.
A second table isolates lightweight adaptation: keeping the shared basis fixed and optimizing only subspace coefficients. On CIFAR-10, CIFAR-100, and SVHN, original accuracy was 90.63% for IID and 90.59% for OOD. At an 80% threshold, adapted accuracy reached 88.76% and 88.87% respectively, using 2.10M coefficient parameters. The gap between direct reconstruction and coefficient adaptation is the practical distinction: the basis supplies the structure, and a small number of coefficients recover task-specific behavior.
Generative and 3D models show the same pattern
The hypothesis is not limited to classifiers. Twenty StyleGAN2 generative models spanning faces, animals, scenes, artwork, textures, and maps produced a low suitability score of q_all = 0.0572, and the authors retained 9 of 13 common 3x3 synthesis convolution layers. Across the 18 models that could be evaluated, mean Inception Score changed by only -0.0041 for IID and -0.0040 for OOD, with paired reconstruction near 20 dB PSNR.
A separate test uses 20 Geometry Distributions models, each trained from scratch on a different 3D primitive. With a 95% explained-variance threshold and no fine-tuning, the mean pairwise Chamfer distance between samples from original and reconstructed models was 0.0250. These results extend the pattern to generative objectives and 3D output, though both settings rest on small model pools.
Large pretrained pools and merging without tuning
The broadest evidence comes from roughly 500 pretrained Vision Transformers collected from HuggingFace across medical imaging, satellite data, and synthetic domains, with no filtering and no access to original training data. The authors also analyze 50 LLaMA3-8B models and 177 GPT-2 and Flan-T5 models. In the ViT pool, most variance is captured by the top few spectral components per layer, excluding the first and last layers.
The paper applies the same structure to model merging. Its universal subspace method computes merging coefficients analytically from the geometry of the shared low-rank subspace, requiring no iterative tuning or validation data. The authors report higher average accuracy than merging baselines while reducing parameter count, because the merged model plus task-specific coefficients contains fewer parameters than any individual model.
Where the claim holds and where it stops
The universality is architecture-specific. The shared subspace is estimated within a pool of models sharing an architecture, and the paper does not claim that a subspace recovered from ResNet-50 transfers to a transformer. The authors also note that small model pools give poor subspace approximations, which is why the ResNet-50 and StyleGAN2 suitability scores are low even though reconstruction remains usable.
The evidence is empirical and correlational in character. The paper offers a conditional analysis of when a shared subspace can be estimated consistently, but it does not establish why training converges to these directions. The authors explicitly leave controlled experiments that isolate individual factors, such as initialization versus data versus loss, to future work. They also raise the open question of whether these subspaces can be discovered without the extensive data and compute used here.
Why the variance threshold is the key choice
The most decision-relevant variable in this paper is not the model count but the retained-variance threshold. At 60% variance, direct ResNet-50 reconstruction fell to 62.71% IID and 56.69% OOD against an 81.00% original. At 80%, those figures rose to 78.26% and 77.91%. At 95%, both reached 80.85%. The same pattern appears in coefficient adaptation, where 60% yielded 86.25% and 86.74% adapted accuracy and 80% yielded 88.76% and 88.87%.
That curve is the paper's practical message. A shared subspace is useful only when the retained directions are chosen to preserve enough variance, and the cost of a smaller basis is paid in accuracy rather than in a failure of the hypothesis. For readers weighing reuse, merging, or parameter-efficient adaptation, the paper's priority is to treat the variance threshold as the tuning knob and to check OOD reconstruction, not just IID, before trusting a recovered basis.
Sources used in this answer
The Universal Weight Subspace Hypothesis
Spectral analysis of over 1,200 models shows that networks sharing an architecture converge to a common low-rank weight subspace, and projecting weights onto it preserves much of the original performance.