[Projecting into Peace] NESS: Exploiting Small Singular Values to Solve Catastrophic Forgetting

TL;DR

Catastrophic forgetting occurs because new task updates "overwrite" the weight directions critical for old tasks. NESS (Null-space Estimated from Small Singular values) solves this by identifying the directions previous tasks didn't use—the approximate null space—and confining all new updates to that "empty" space. By using a LoRA-style parameterization with a frozen basis derived from small singular values, NESS achieves virtually zero forgetting (and sometimes even positive knowledge transfer) on standard benchmarks.

Problem & Motivation: The Gradient Projection Headache

In the quest for models that learn forever without forgetting, Orthogonality has become a golden rule. If a new update is orthogonal to the space used by previous data, it shouldn't interfere with old knowledge.

Current SOTA methods like GPM (Gradient Projection Memory) implement this by projecting gradients during the optimization step. While effective, this is an "active" constraint—you have to modify the optimizer's behavior at every step. The authors of NESS asked a clever question: Why restrict the gradient when we can restrict the weights themselves?

Methodology: Finding the "Quiet" Directions

The core intuition of NESS lies in Singular Value Decomposition (SVD). When we look at the input features ($X$) of a layer:

1. Large Singular Values: Represent directions where the data has high energy—these are the features the model has already learned.
2. Small Singular Values: Represent the "Null Space"—directions that are nearly orthogonal to everything the model has seen so far.

The NESS Workflow

1. Collect Inputs: For each layer, collect a small set of representative inputs from previous tasks.
2. SVD Decomposition: Compute the SVD to find the directions ($U$) and their energy ($\Sigma$).
3. Construct the Basis: Select the singular vectors corresponding to the smallest singular values ($U_t$). This defines your "safe zone."
4. Adapter Learning: Instead of updating the main weights $W$, we learn a small matrix $V$ such that the update $\Delta W=U_{t}V$.

Because $U_t$ is fixed and represents the null space, any update we make through $V$ is mathematically guaranteed to have minimal impact on previous task outputs.

Figure 1: The NESS architecture. Updates are constrained to a fixed orthogonal basis derived from the quietest parts of the input manifold.

Experiments & Stability SOTA

The researchers tested NESS against heavyweights like TRGP, GPM, and DFGP across CIFAR-100, 5-datasets, and MiniImageNet.

The results in the Backward Transfer (BWT) metric are where NESS truly shines. While most models show negative BWT (meaning they forget), NESS often shows positive or zero BWT. This means the model essentially learns new tasks without hurting the old ones at all.

Table 1: NESS achieves near-zero BWT (highlighted in yellow), indicating superior stability compared to baselines.

Visualizing the Forgetting

The "Forgetting Map" in the paper shows a stark contrast. In the baseline models (like TRGP), the grid shows varying degrees of performance degradation as more tasks are added. NESS, however, maintains a remarkably clean performance profile across the sequential tasks.

Figure 2: Forgetting rate across tasks. NESS (a) shows significantly lower interference compared to the DFGP baseline (d).

Critical Insight: Parameter Efficiency

Because NESS only trains the $V$ matrix, and the dimensionality of $V$ is determined by the size of the null space, it effectively acts as a dynamic Parameter-Efficient Fine-Tuning (PEFT) method. By adjusting the threshold for what constitutes a "small" singular value (${ϵ}_1$), we can trade off between the model's capacity to learn (plasticity) and its ability to remember (stability).

Limitations & Future Work

Despite its success, NESS has a few "open problems":

• Threshold Tuning: Choosing the right ${ϵ}_1$ is critical. Too high, and you forget; too low, and you can't learn the new task.
• Memory-Free SVD: Currently, the method requires a forward pass on some data to compute the covariance. Moving toward a purely data-free null-space estimation would be the "Holy Grail."

Conclusion

NESS proves that the "empty" spaces in a neural network's representations are not useless. They are the ideal canvases for new knowledge. By shifting from gradient projection to weight-space adaptation, NESS provides a more stable, efficient, and theoretically grounded path for lifelong learning agents.