Still: Breaking the KV Cache Bottleneck with Amortized Synthesis

TL;DR

The KV cache is the "memory wall" of modern LLMs. Still introduces a breakthrough by replacing traditional token selection with amortized synthesis. Instead of just picking which tokens to keep, Still uses a tiny, trained Perceiver module to "summarize" the entire KV cache into a compact form in a single forward pass. It beats the strongest baselines by 8-22 points on RULER and maintains performance at extreme 200x compression.

The Core Conflict: Selection vs. Synthesis

Deploying LLMs for long-horizon tasks (like coding agents or multi-day chats) is restricted by the KV cache's linear growth. Until now, researchers had to choose between:

1. Selection (Heuristic): Fast but "dumb." You keep some tokens and throw away the rest. If the information you need was in the discarded 90%, the model fails.
2. Synthesis (Optimization): Smart but slow. Methods like "Attention Matching" compress information better but require solving optimization problems for every single request, which is too slow for production.

Still enters the "underexplored quadrant": Amortized Synthesis. It learns how to compress context during a one-time training phase, making inference a lightning-fast forward pass while retaining the high-fidelity representation of synthesis.

Methodology: The Per-Layer Perceiver

Still places one Perceiver-style module per Transformer layer.

• Position-Free Frame: One of the paper's key insights is handling Rotary Position Embeddings (RoPE). Rotating keys before blending them creates "phase noise." Still un-rotates the keys, performs synthesis in a position-free space, and then re-applies RoPE to the compact output.
• Latent Queries: A bank of learned queries cross-attends to the full KV cache. It doesn't just pick tokens; it blends them.

Figure 1: The Still per-layer compactor. Learned latents Z cross-attend the full KV cache to produce compact keys and values.

Performance: Redefining the Pareto Frontier

The most impressive result is how Still behaves as context grows. In selection-based methods, accuracy typically falls off a cliff once the compression ratio gets too high. Still maintains a much flatter curve.

Figure 2: Speed–quality trade-off. Still remains in the "goldilocks zone"—high accuracy and high speed—even as context expands to 64k tokens.

Key Experimental Highlights:

• Scale Transferability: A compactor trained for a 4B model successfully transfers to 32B and MoE (Mixture-of-Experts) variants without retraining the base weights.
• Iterative Compaction: Because Still is just a forward pass, it can be called repeatedly as new tokens arrive. This allows for "rolling compression" in long-horizon agent trajectories.
• Summarization: On HELMET, Still recovered up to 95% of the full-context performance while using only a fraction of the memory.

Deep Insight: Why Why Synthesis Wins

The paper proves that "values carry the content, but keys define the routing." By learning both jointly, Still doesn't just store information; it organizes it so the frozen LLM knows exactly how to retrieve it later. In their ablation studies, they found that replacing learned keys with simple "high-importance" selection caused performance to tank, proving that how you index memory is just as important as what you store.

Conclusion & Future Outlook

Still demonstrates that we don't need to save every token to maintain "perfect" memory. By treating the KV cache as a learnable latent space, we can achieve 100x-200x compression with minimal loss in reasoning capability.

Limitations: It's not yet "lossless" for exact-string retrieval (like Needle-in-a-Haystack), and the training horizon still limits extrapolation. However, for semantic tasks and summarization, it is the new SOTA for efficient long-context deployment.