Can SlimKV compress KV cache jointly across tokens and features?

Why feature compression stalls at key reconstruction

Long-context serving is limited by the key-value (KV) cache, which grows with context length and creates memory pressure and memory-access latency. The paper groups prior fixes into two families. Token-wise methods reduce how many cached states are kept, but deleting states can lose information under aggressive compression, and eviction-based methods often need the full KV to be built before deletion, which limits prefill savings. Feature-wise methods shrink each token's KV dimensions, but the compressed keys usually must be expanded back to full dimension before positional encoding (RoPE) can be applied.

That reconstruction step is the bottleneck SlimKV targets. RoPE is position-dependent, so it cannot be folded into a single position-independent projection once keys are stored at low rank. The paper's central question follows directly: can compression along both axes avoid this reconstruction cost while still beating single-axis limits?

Beacon states carry implicit position cues

SlimKV builds on beacon-based soft compression, which folds historical context into beacon states rather than hard-deleting tokens. Its first addition is to train the beacon KV projections directly under low-rank constraints, so each beacon state both summarizes historical tokens and forms a compact latent KV representation. A second addition allocates rank per layer according to layer-wise low-rank heterogeneity, using a floor-constrained equal-energy criterion with lambda set to 0.5 to balance the global rank budget against minimum layer capacity.

The more surprising finding concerns position. The authors test what happens when key-side RoPE is removed and find a positional asymmetry: removal degrades both beacon and raw-token keys, but beacon memory tokens degrade far less. They also report that RoPE-free beacon states reconstruct random character sequences well above random guessing, which they read as evidence of non-trivial positional cues acquired without explicit key-side RoPE. This is the paper's own interpretation of an empirical pattern, not a proven mechanism.

Latent-space attention removes the rebuild step

Exploiting that asymmetry, SlimKV trains beacon KV projections under a K-RoPE-free constraint and keeps RoPE on raw-token keys and on all queries. With explicit beacon key RoPE gone, decoding uses latent-space attention: queries are projected into the beacon latent space, attention scores are computed against the cached latent keys, and values are aggregated in latent form before the up-projection. The paper writes this as (Q' U_K) C_K^T for scores and (A C_V) U_V^T for value aggregation, with the query projection computed after grouped-query attention grouping.

The practical consequence is that beacon keys never need to be materialized at full dimension during decoding. The paper also analyzes ALiBi and argues it is algebraically compatible with this scheme, because ALiBi adds its positional bias after the query-key product. That analysis is theoretical; the reported experiments are on RoPE-based backbones.

LongBench gains concentrate at high compression

SlimKV is evaluated on Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct with frozen backbones, training only the beacon-branch QKV projections on 0.5B RedPajama tokens followed by fine-tuning on LongAlpaca, BookSum, and synthetic data, with all samples shorter than 20K tokens. Baselines are Full KV, PALU for low-rank compression, KVZip and SnapKV for eviction, and Activation Beacon for soft compression.

On LongBench, SlimKV outperforms baselines at 16x/32x compression and remains leading at 4x/8x, where it retains over 96% of the uncompressed model's score. The low-compression averages show the pattern concretely: at 4x on Llama-3.1-8B, SlimKV-f2 scores 48.10 against 48.21 for Activation Beacon, 46.66 for PALU, and 36.39 for KVZip; at 8x on Qwen2.5-14B it reaches 49.94 versus 45.50 for Activation Beacon. The advantage is therefore largest where compression is aggressive, and near-parity at the mildest setting.

Task type decides where the gain lands

The paper's task-wise analysis adds a condition that matters for anyone choosing a compression setting. SlimKV-f4 gives its most consistent gains on document QA tasks, including single-document and multi-document QA, which the authors attribute to RoPE-free beacon memory preserving semantic evidence for retrieval-style reasoning while relieving token-wise pressure under the same total budget. Summarization gains are smaller but stable, consistent with those tasks depending more on global semantic aggregation than exact token positions.

Few-shot and code tasks are less uniform and more backbone-dependent: SlimKV improves clearly on Qwen2.5-14B but less so on Llama-3.1-8B, and code results vary, which the authors link to stronger reliance on local order, formatting, and structural cues. The same boundary appears in the positional design itself: raw-token keys keep RoPE, and only beacon keys drop it.

Speedups are real, but memory is set by the ratio

Decoding latency on Llama-3.1-8B over 128 generated tokens separates SlimKV from a reconstruction-based variant that keeps beacon key RoPE and materializes full-dimensional K/V before attention. At 128K context and 16x compression, SlimKV records 6.35 seconds of attention time and 14.24 seconds end-to-end, against 8.73 and 16.79 for the reconstruction variant and 46.64 and 48.08 for Full KV. At 32x the corresponding SlimKV figures are 7.65 and 13.78 seconds. The abstract summarizes this as up to 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K length.

Peak memory behaves differently and is easy to misread. SlimKV and the reconstruction variant have nearly identical peak allocated memory, because reconstruction changes only the computation flow, not the stored cache size; what lowers memory is the compression ratio, from 25.72 GB at 16x to 23.41 GB at 32x on the 32K setting. Time-to-first-token improves only modestly over the reconstruction variant, mainly by skipping key-side RoPE computation and full-key materialization for beacon states.

Where SlimKV fits and where it does not

The paper's own comparison with MLA sharpens the trade-off. MLA also uses low-rank KV representations to avoid full-dimensional reconstruction, but it keeps token granularity and a separate positional key branch, and it is tightly coupled to the model architecture, requiring costly base-model pretraining. SlimKV instead jointly compresses tokens into beacon states and their K/V features, acquires beacon positional cues during RoPE-aware prefill without a separate positional-key cache, and retrofits pretrained GQA-RoPE backbones by training only beacon projections.

That retrofit path is also the scope limit. SlimKV is mainly evaluated on RoPE-based language models and text-only long-context tasks; behavior under other positional encoding schemes and extension to multimodal KV caches remain unvalidated. The priority the evidence supports is therefore specific: when a RoPE-based backbone needs aggressive compression and retrieval-style long-context work dominates, SlimKV's reconstruction-free beacon design is the setting where its accuracy and latency advantages coincide. For order-sensitive code or formatting-heavy tasks, the paper's own results show the gain is the least reliable.

Sources used in this answer