HOLA: Breakthrough Hippocampal Memory for Linear Attention

TL;DR

Complementary Learning Systems (CLS) theory suggests that the brain uses the neocortex for slow generalization and the hippocampus for fast, one-shot memory. HOLA (Hippocampal Linear Attention) applies this to AI: it augments Gated DeltaNet with a bounded exact KV cache that stores "surprising" tokens—those the recurrent state fails to predict correctly. This architectural shift allows a 340M model to outperform full-attention Transformers in perplexity while maintaining O(1) memory Scaling.

The "Losing Memory" Problem in Efficient Models

Modern efficient LMs (like Mamba or DeltaNet) are masterful at compressing context into a fixed-size state. However, this compression is intrinsically lossy. As the sequence grows, new associations overwrite old ones. This manifests as a "forgetting" behavior in tasks like "Needle in a Haystack" (NIAH), where the model fails to retrieve a specific fact buried deep in the prefix.

The author's core insight is that we shouldn't try to make the compressor perfect; we should give it a "hippocampus"—a small, exact buffer for what the compressor forgets.

Methodology: The Semiparametric "Neocortex + Hippocampus"

HOLA formalizes memory as test-time memory regression. The system consists of two parts:

1. Parametric State ($S_t$): The standard recurrent state (the "neocortex") that handles linearly compressible structure.
2. Non-parametric Cache ($A_t$): A bounded set of exact KV pairs (the "hippocampus").

1. What to Store? The "Surprise" Signal

Unlike traditional sliding windows that keep the most recent tokens, HOLA keeps the most important tokens. It uses the delta-rule write magnitude ($\beta \cdot \Vert e\Vert$) as an eviction score.

• If the state predicts a value poorly (large residual $e$), the token is "surprising."
• If the model commits this change strongly (large $\beta$), it is an essential support point for memory.

2. How to Read? Decoupled Retrieval

Standard linear attention often suffers from "soft averaging," where retrieval is too blurry to pick out one specific token. HOLA introduces a decoupled RMSNorm-$\gamma$ path. By increasing the logit scale during the cache read (and keeping the state update stable with unit-norm keys), the model achieves sharp, near-argmax retrieval.

Figure 1: HOLA Architecture - showing the dual-path memory system.

Empirical Results: Surpassing the Transformer Ceiling

The performance gains are striking. In Wikitext-103 perplexity, HOLA (22.92) outperformed the full-attention Transformer++ (26.88).

Length Robustness

On the RULER benchmark, which tests retrieval up to 32k tokens (16x the training length), HOLA remains robust while standard recurrent models (GDN) collapse.

Figure 2: (a) Perplexity reduction across models. (b) Needle recall robustness at 32k context.

Ablation: Why "Surprise" Matters

The authors compared their "surprise" eviction to a standard position-based "recency" cache. At 32k tokens, the recency cache only achieved 0.24 recall, while HOLA's surprise-based eviction reached 0.58. This proves that for a bounded memory, what you remember is more critical than how recent it is.

Critical Insight & Conclusion

HOLA represents a paradigm shift in how we view efficient attention. Instead of treating the recurrence and the cache as competing methods, it treats them as complementary estimators. By using the model's own delta-rule innovation as a selection signal, HOLA identifies the exact gaps in its own compression and fills them with a high-fidelity buffer.

Limitations: While HOLA significantly narrows the gap, it does not yet fully close the distance to full-attention models on extremely dense extraction tasks (like FDA). However, for practical long-context deployment, HOLA offers the best balance of SOTA performance and linear efficiency available today.