Can sparse plus low-rank decomposition compress large language models without performance loss?

Splitting each weight matrix into two parts

Sparse plus low-rank decomposition replaces a pre-trained weight matrix W with the sum of two cheaper matrices: a sparse matrix S that keeps only a small fraction of entries, and a low-rank matrix LR formed by multiplying two thin factors. The compressed model stores S and the factors instead of W. The appeal is that the low-rank part can later serve as an initialization for LoRA fine-tuning, so the decomposition is designed to fit an adaptation workflow rather than to stand alone.

The paper's target is one-shot post-training compression: the decomposition is computed from a small calibration set without retraining the model. Prior sparse-plus-low-rank methods for LLMs, notably OATS and HASSLE-Free, rely on alternating minimization, optimizing the sparse and low-rank pieces in turn. The authors argue this leaves quality on the table and propose a different optimizer plus a second refinement stage.

A three-block ADMM replaces alternating minimization

The first component, 3BASiL, minimizes layer-wise reconstruction error, meaning it tries to make each layer's output on calibration inputs match the original layer's output. It does this with a three-block Alternating Direction Method of Multipliers: the sparse component S, the low-rank component L, and a constrained copy D of the sparse component are updated in sequence, followed by a dual update on the multiplier variable V. The copy D is what enforces the sparsity pattern, including N:M structures such as 2:4, where only two of every four weights may be nonzero.

The paper reports closed-form updates for the sparse block and an exact low-rank update for the L block, and states a convergence guarantee for the scheme. The practical claim is that jointly optimizing both components inside one framework yields better layer-wise reconstruction than optimizing them separately, which the authors support with objective-minimization comparisons against competing decomposition methods.

Refinement aligns whole transformer outputs

Layer-wise reconstruction is a proxy. Matching one layer's output does not guarantee that the full stack behaves like the original, because small per-layer errors compound. The second component, transformer matching, addresses this by jointly optimizing the sparse and low-rank components across transformer layers against a loss that aligns outputs at the transformer level, computed in a memory-efficient way.

The authors describe this step as universal: it can refine any sparse-plus-low-rank decomposition, including a purely sparse one, because it operates on the already-decomposed weights rather than on a specific optimizer. That universality matters for interpreting the results, since it means the reported gains combine two separable contributions, the 3BASiL optimizer and the transformer-matching refinement, and the paper reports both 3BASiL alone and 3BASiL-TM.

The headline gain is a narrower perplexity gap

The central quantitative result is a relative comparison of gaps. Under a 2:4 sparse plus rank-64 low-rank configuration, 3BASiL-TM reduces the WikiText2 perplexity gap to the dense LLaMA-8B model by over 30% compared with prior methods. This is a reduction in the distance to the dense model, not a claim that the compressed model matches it; the dense model still has lower perplexity.

The paper's tables show the same direction on smaller models. On Llama-3.2-1B under a 50% sparsity plus rank-128 configuration, 3BASiL-TM records lower WikiText2 and C4 perplexity than OATS, Hf-SparseGPT, and Hf-ALPS, and 3BASiL alone already improves on those baselines before refinement. On Meta-Llama-3-8B, the same ordering holds, with the dense model remaining the reference point at the bottom of the table.

Faster compression and gains that survive LoRA

Runtime is reported separately from quality. The authors state that their method achieves over 2.5x faster compression runtime on an A100 GPU than the state-of-the-art sparse-plus-low-rank method. This is a compression-time measurement, not an inference-speed claim, and the paper does not present an equivalent end-to-end latency result for deployed models in the evidence reviewed here.

A second result concerns what happens after adaptation. The low-rank components can initialize LoRA fine-tuning, and the authors ran limited fine-tuning on roughly 10% of the first C4 training shard, about 15 million tokens. After this LoRA fine-tuning, 3BASiL-TM still outperforms competing decompositions under a 2:8 plus rank-64 configuration, with approximately 8% lower perplexity. The advantage therefore is not erased by adaptation, though the fine-tuning budget is deliberately small.

What the evidence does and does not cover

The quality evidence is perplexity on WikiText2 and C4 for LLaMA-family models at specific sparsity and rank settings, plus the reported gap reductions. The paper does not, in the material reviewed, establish that compressed models match dense accuracy on downstream reasoning or generation benchmarks, so the 30% figure should be read as a relative gap reduction under one configuration rather than a general no-loss guarantee.

The authors also flag open directions: extending the three-block ADMM and transformer matching to quantization or combined quantized-sparse constraints, and algorithmically allocating different sparsity and rank settings to different layers. Both are stated as future work, which bounds how far the current configuration choices can be generalized.

Weigh the gap reduction against the configuration

The decision-relevant comparison in this paper is not compressed versus dense but compressed versus compressed. If the choice is among sparse-plus-low-rank decompositions at a fixed budget such as 2:4 plus rank 64, 3BASiL-TM is the stronger option on the reported perplexity gap and on compression runtime, and its low-rank factors remain usable as LoRA initialization. If the requirement is parity with the dense LLaMA-8B model, the paper's own numbers do not support that, since the gap is reduced rather than closed.

The most useful threshold from the evidence is the configuration itself: the over-30% gap reduction is tied to 2:4 sparse plus rank 64 on LLaMA-8B, and the post-fine-tuning advantage is tied to 2:8 plus rank 64 with about 15 million tokens of LoRA training. Readers comparing methods should match those settings before transferring the claim, and should treat the transformer-matching step as the portable piece, since the authors present it as applicable to any sparse-plus-low-rank decomposition.

Sources used in this answer