Can training-free transformer merging align operators along the execution path?

Why merging layers independently breaks attention

Training-free model merging combines several fine-tuned models into one multi-task model without further optimization on labeled data. The paper's starting observation is that in transformers this is not a layer-by-layer problem. Within a head, attention scores depend on the composed Query-Key operator, and the information written back depends on the composed Value-Output operator. Merging the individual projection matrices independently therefore acts on objects that are not the functional units of the computation.

The second problem is path mismatch. When earlier components are merged, downstream components receive different activations than they do in the original model, so the merged and original execution paths no longer match. A merge that is locally reasonable at one layer can be evaluated under inputs the deployed model will never produce.

Merging composed operators in topological order

SLOA changes the unit of merging. Instead of combining the individual matrices W_Q, W_K, W_V and W_O, it merges the composed operators: the QK operator that determines token-to-token routing through pre-softmax attention scores, and the VO operator that determines what is written back after attention weights are applied. The paper notes that for any invertible matrix R, the QK product is preserved by reparameterizing the query and key factors, and the VO product is invariant under an analogous transformation. Because SLOA only depends on the composed operator, two different factorizations that realize the same operator become the same object for the method.

The method then proceeds along the execution path of the partially merged model. For each local operator, it solves a regularized least-squares problem in closed form, using calibration data to estimate the local behavior of each functional component and aligning operators sequentially under the intermediate activation of the partially merged model. The paper contrasts this with gradient-based progressive distillation, which trains merging coefficients layer by layer; SLOA does not learn coefficients by gradient descent but directly solves for the merged local operator.

The paper also describes the method as more fine-grained than layer-wise distillation inside transformer blocks: it can separately align LayerNorm affine maps, standard linear maps, QK attention operators and VO attention operators, respecting the internal computational structure of attention rather than treating a whole block as one black-box feature map.

Factorization restores valid parameters and allows rank expansion

After alignment, SLOA factorizes the merged operators back into valid transformer parameters. For a head with original dimension d_k, each task-specific QK or VO operator has rank at most d_k, because each head implements a d-by-d operator through a width-d_k bottleneck. SLOA merges these operators in the ambient space where no rank constraint is imposed, and restores a factorized form only afterwards, using a truncated SVD that keeps the top r singular directions.

This is where rank expansion comes from. Different task models may use different low-dimensional subspaces, and a combination of low-rank operators need not share their rank: the sum of two rank-one operators with different singular directions has rank two, and in general the rank of the summed operators can reach min(d, T d_k). Truncating the merged operator back to d_k can therefore discard directions that are useful for multi-task behavior, while r greater than d_k retains them. The paper argues this does not introduce free parameters, because the factorization only retains singular directions already present in the aligned operator, which is anchored to the task-specific operators by the regularizer. The cost is a wider head at inference.

What the sequential step buys in vision and language

The paper reports that the sequential step reduces error accumulation across layers, and that the approach generalizes across modalities, model scales and varying numbers of tasks, from CLIP and RoBERTa to billion-parameter LLMs. On the larger ViT-B/16 backbone, the reported trend mirrors ViT-B/32: the margin over the best baseline widens as the number of merged tasks grows. On 8 tasks, the data-free baseline Iso-CTS is narrowly strongest (89.01 versus 88.04 for SLOA at r=128); by 14 tasks SLOA has overtaken it (85.16 versus 84.54); and by 20 tasks SLOA leads by nearly 5 points (83.43 versus 78.50).

The paper offers an explanation for why data-free baselines stay competitive at small task counts: prior work cited in the paper shows that larger, better-pretrained vision backbones tend to be more weight-disentangled than smaller ones, so task-specific fine-tuning directions interfere less under simple weight-space combination. As the number of merged tasks grows and interference increases, the paper argues that aligning functional operators rather than raw weights wins by an increasingly clear margin. This is the authors' interpretation of the pattern, not a separately measured mechanism.

The paper also reports per-task comparisons of SLOA at r=128 against data-free and training-free baselines on ViT-B/32 with 20 vision tasks and on RoBERTa with 7 GLUE tasks, and states that SLOA consistently outperforms both groups.

Rank expansion trades accuracy against inference cost

The paper separates two claims about rank. The first is that improvements over strong merging baselines are obtained without requiring rank expansion. The second is that optional expansion provides a further accuracy-inference-cost trade-off. These are different statements about different configurations, and the paper keeps them distinct.

On ViT-B/16, increasing r from 64 to 128 improved performance at every task count, with the gain growing alongside the number of merged tasks. The paper also reports that rank expansion did not reduce average accuracy in any of the settings tested and typically improved it, and that the cost is a wider head at inference. The scaling figures for ViT-B/32, ViT-B/16 and RoBERTa show average performance as the number of calibration samples increases from 2^7 to 2^11, with curves for r_qk = r_ov = 64, 128 and 256.

Where SLOA applies and where it does not

SLOA is training-free, but it is not data-free: it uses calibration data to estimate the local behavior of each functional component. The LLM experiments use a fixed 256 calibration examples per task, drawn from the same training split and formatted with the same template used for fine-tuning. The paper's own comparison notes that ProDistill, a training-based progressive method, optimizes merging coefficients with gradient descent, whereas SLOA solves a closed-form regularized least-squares problem per local operator.

The method extends naturally to LoRA-fine-tuned models. For a LoRA-adapted task weight formed as a base weight plus a low-rank update, the paper states there is no additional difficulty caused by the LoRA parameterization itself. In the LLM setting, the authors did not use an off-the-shelf pool of LoRA experts for LLaMA-2-7B; they trained three LoRA experts independently from the base checkpoint at rank 32, targeting all linear projections, for math, code and instruction-following, and evaluated them on GSM8K, HumanEval and IFEval respectively.

Two boundaries follow from the evidence. The reported gains are measured on the specific backbones, task pools and benchmarks the paper uses, so the size of the margin should not be assumed to transfer to other task mixtures or evaluation suites. And the rank-expansion benefit is described as a trade-off rather than a free improvement, because it widens attention heads at inference.

The practical choice is rank, not just method

The most decision-relevant split in this paper is not simply whether to use a training-free merge, but at what rank to deploy it. At the default r equal to the original head dimension, SLOA is reported to improve over strong baselines without rank expansion, which makes it comparable to other training-free methods on inference cost. At larger r, the merged operator retains singular directions that a d_k-truncation would discard, and the paper reports further accuracy gains that grow with the number of merged tasks, paid for with wider heads at inference.

That makes the number of merged tasks the variable to watch. The paper's ViT-B/16 comparison shows a data-free baseline ahead at 8 tasks, SLOA ahead by 14 tasks, and SLOA ahead by nearly 5 points at 20 tasks, with the benefit of raising r from 64 to 128 also growing with task count. For a reader deciding how to merge many fine-tuned experts, the paper's evidence points to operator-level alignment under the partially merged execution path as the priority, and to rank as the dial that trades multi-task capacity against inference cost.

Sources used in this answer