Can self-supervised models separate neural plasticity from recording instability?

Why chronic recordings blur plasticity and drift

Longitudinal recordings are meant to show how task-relevant population activity emerges and reorganizes across days. The obstacle is that electrode drift, signal degradation, and changes in which neurons are recorded can alter the observed activity even when the underlying computation is unchanged. The paper's stated problem is therefore not just decoding stability but attribution: current tools cannot easily distinguish representational plasticity from recording instability.

Existing approaches address this through neural manifold alignment, recalibration-based test-time adaptation, and transformer-based in-context learning. The paper argues that many of these methods are tailored to specific decoding tasks, or compensate for distribution shifts in observed activity without explicitly separating task-relevant structure from recording variability. That gap motivates a general-purpose representation that can accommodate changing populations while preserving biological change.

Predicting latents instead of reconstructing spikes

NeuroLens is built on JEPA, which predicts masked or future representations in latent space instead of reproducing every detail of the recorded activity. The paper's rationale is that probe drift can change which neurons are recorded even when population dynamics remain stable; a reconstruction-based model must account for those observational changes even if they are unrelated to the computation of interest. Predicting in latent space emphasizes temporally predictable structure and reduces sensitivity to transient, recording-specific variability.

The architecture has two parts. A day-adaptive encoder maps each day's variable-size neural population into a common latent space, and a causal transformer predictor shared across days predicts future latent embeddings from past ones. Training combines prediction mean squared error with sketched isotropic Gaussian regularization to prevent representation collapse. Because each time bin is encoded independently, temporal modeling is left to the predictor rather than leaking into the encoder.

Calibration signatures let the encoder transfer to new populations

Cross-attention is invariant to the order of neuron tokens, so spike counts alone do not tell the model which neuron is which. The paper solves this with a small set of daily calibration trials: each neuron is summarized by firing-rate statistics, autocorrelograms, and population coupling, and a network maps that signature to a neuron embedding added to the spike token. Using physiological signatures instead of fixed neuron-ID embeddings is what allows transfer to unseen neurons from few calibration trials; an ablation on three held-out sessions found calibration-based identity embeddings improved both movement and choice decoding.

Human Utah array recordings required a different route. There, channels capture mixtures of nearby neurons rather than spike-sorted units, so neuron signatures could obscure channel identity and induce representation collapse. The model instead maps channel features through an MLP encoder and uses a hypernetwork conditioned on calibration statistics to generate day-specific FiLM parameters for the encoder layers. An ablation on five held-out T12 days found channel-drift adaptation gave a slight phoneme improvement and a clear sentence-retrieval improvement.

Better decoding of choice and sentence meaning

On mouse Neuropixels recordings from the IBL task, where animals reported left or right stimulus location by turning a wheel, NeuroLens achieved lower latent effective rank, better choice decoding, and comparable movement decoding relative to causal NDT across 14 test sessions. Both models outperformed decoding from raw neural activity. The paper also projects latent trajectories for left- and right-choice trials onto the first three principal components and reports that separation between the two choices becomes clearer over April to June, with clearer separation in NeuroLens than in causal NDT, quantified by linear discriminant analysis balanced accuracy.

On human Utah array data from two participants with ALS-induced anarthria attempting to speak displayed sentences, NeuroLens improved sentence-embedding decoding over causal NDT with comparable phoneme decoding. Lower phoneme error rate and higher top-5 sentence retrieval accuracy indicate better decoding. Top-5 accuracy is averaged across held-out days, and each day has a candidate pool of 288 to 448 sentences, keeping chance level at 1 to 2 percent. These are different measures: phoneme error rate scores a 41-class phoneme sequence, while sentence retrieval ranks predicted embeddings against candidate sentences.

Pretraining pays off most for movement decoding

A key comparison asks whether multi-day pretraining beats training directly on the target session. The paper held the target-session data budget fixed: a single-session model was trained from scratch on the same 20 percent adaptation split used to adapt the pretrained model, with identical architecture, optimizer, schedule, and epoch budget, differing only in exposure to 39 prior sessions.

Across five randomly selected held-out sessions, pretraining improved movement-related behavior decoding on every session by 0.046 to 0.094 R², mean 0.076, which the paper describes as large relative to the spread across baselines. For choice decoding, the pretrained model was ahead on four of the five sessions. The exception is visible in the table: on 2023-05-24 the single-day model reached 0.83 balanced accuracy against 0.61 for the pretrained model, so the benefit is not uniform across targets.

Few-shot adaptation updates only read-in layers

For a new day, gradient-based few-shot adaptation updates only the read-in layers of the encoder, including the spike projector, identity network, and fusion projector, plus an attached decoder head for decoding. The cross-attention layers, region embeddings, and autoregressive predictor stay frozen to preserve the representations and dynamics learned during pretraining. When sufficient data from a new session are available, the paper instead fine-tunes the entire model, including the decoder head, for tasks such as speech decoding where few-shot adaptation may not perform well.

This split matters for interpreting the generalization claim. The stable part of the model is the shared predictor and the latent space; what adapts is the mapping from a new population into that space. The paper's claim of rapid few-shot adaptation to unseen neural populations therefore rests on calibration-based read-in changes, not on relearning the temporal structure.

Drift measured without a shared basis

To characterize representational drift, the paper uses Representational Similarity Analysis, which compares second-order structure rather than aligning representations point by point. For each day it builds a temporal representational dissimilarity matrix from Pearson correlations between time bins, then compares two days by one minus the Spearman correlation between their matrices. The measure is zero when two days have identical temporal geometry and increases as they diverge.

Because this comparison depends on pairwise relationships rather than a coordinate system, it can be applied across models or recording days without a shared basis. The paper corrects for within-day sampling variability by subtracting the average within-day split-half drift from each day pair, and visualizes the corrected dissimilarity as a function of lag between days. This gives a model-free description of drift, separate from decoding performance.

What the separation does not yet resolve

The paper is explicit that it does not disentangle changes due to learning and behavioral change from intrinsic neural changes, and it does not model cross-area interactions or region-specific representation drift. So the central framing, reducing sensitivity to recording instability while preserving biological change, is supported by improved decoding and more stable cross-day performance, not by a direct decomposition of the two sources. A reader should treat the latent space as a representation that is more robust to recording variability, not as a proven measurement of plasticity alone.

The practical priority follows from the comparisons rather than from the framing. Pretraining on 39 prior days helped movement decoding on all five tested sessions and choice decoding on four of five, while few-shot adaptation touched only read-in layers. The most defensible reading is that NeuroLens is most clearly valuable when a stable shared latent space can be reused across days and a new population needs only a light calibration-based mapping, with the caveat that one choice session reversed the pretraining advantage.

Sources used in this answer