Two views of one brain state, aligned
Swap-VAE is a generative model built on a variational autoencoder, a network that compresses a neural state into a small set of latent variables and then reconstructs the input from them. The paper's addition is an instance-specific alignment loss: two augmented views of the same brain state are pushed toward similar representations in one part of the latent space. The augmentations are spatial, dropping neurons from the input with probability 0.6, and temporal, selecting a nearby time point within a window of plus or minus 5 samples as a positive example.
The intuition given in the paper is that a representation surviving neuron dropout and small time shifts should be invariant to which specific neurons encode the state while preserving temporal consistency. The latent space is 128-dimensional and is split into two 64-dimensional blocks, called content and style. A decoder reconstructs neural activity from both blocks, so the style block is not free to collapse; it must still help explain the data.
BlockSwap strengthens the content alignment
The alignment loss alone could be satisfied by letting the two views differ in the style block. To prevent that, the paper introduces BlockSwap: before decoding, the content variables of the two augmented views are exchanged. The decoder must then reconstruct each view from the other view's content plus its own style, which pushes shared, behavior-relevant structure into the content block and leaves view-specific variation in style.
The ablation on the Chewie-1 dataset supports this design. Removing BlockSwap drops reach-direction accuracy from 72.81% to 63.17%, and removing the alignment term while keeping swapping gives 68.36%. A vanilla VAE reaches 63.79% and a beta-VAE 64.34% on the same task. The paper reports that removing the alignment loss while keeping BlockSwap changes performance only minimally, so the swapping operation carries much of the benefit.
Reach direction and timing land in different spaces
The central evidence for disentanglement is a double dissociation between the two latent blocks. On the Chewie-1 dataset, the content space decodes reach direction well while the style space has little predictive power over reach direction. For temporal decoding the pattern reverses: the content space is weak, and the style space organizes the data along an axis where reach direction has collapsed but time is ordered. The paper reports this comparison for Chewie-1 and notes similar trends in the remaining datasets.
The t-SNE visualizations tell the same story in a different form. In the content space, targets appear in a circular arrangement matching the eight reach directions, a structure the loss does not explicitly encourage. In the style space, reach direction is collapsed while time is organized. The paper treats this as evidence that semantic structure can be separated without labels while the overall behavioral structure is preserved.
Reach decoding gains are real but uneven
Across four reaching datasets, Swap-VAE's full space reaches 72.81%, 68.97%, 64.26%, and 66.12% accuracy on reach-direction classification for Chewie-1, Chewie-2, Mihi-1, and Mihi-2 respectively. The corresponding delta-accuracies, which the paper uses as a second measure, are 85.04%, 83.66%, 82.18%, and 82.74%. These are compared against a supervised reach decoder, a supervised pi-VAE, a beta-VAE, and two self-supervised methods, BYOL and MYOW.
The comparison is not a uniform win. Swap-VAE is on par with or better than all benchmarks on reach decoding, but MYOW reaches 70.48% on Mihi-1 against Swap-VAE's 64.26%, and MYOW's delta-accuracy there is 83.24% against 82.18%. On Chewie-1, MYOW's plain accuracy is 70.41% against 72.81%. The paper's own reading is that both self-supervised approaches can outperform supervised decoders on reach classification because of the regularization from their alignment losses.
The temporal decoding results are where the two-space design matters most. The paper reports that Swap-VAE performs comparably with the beta-VAE and the supervised time decoder on temporal decoding, while other self-supervised models fail to capture the temporal structure they built invariance to. The authors attribute this to the extra style variables, which carry information that a single aligned representation would have discarded.
Generated samples add modest supervised gains
Because Swap-VAE is generative, the paper also tests whether its samples are useful. New neural activity is generated and mixed into the training set of a supervised classifier at 50%, 100%, and 200% of the original data. On one dataset from each animal, Chewie-1 and Mihi-2, accuracy improves by roughly 5% and 3.5% over the supervised baseline respectively. The paper notes that these augmented supervised models still do not surpass Swap-VAE itself.
This is a bounded result: it covers two datasets, three mixing ratios, and a single classifier setup. It shows the generator produces something useful for a downstream decoder, not that generated data substitutes for real recordings.
A waiting period may sharpen the split
The two animals were trained differently. Chewie reaches immediately after the target appears, while Mihi waits 500 to 1500 ms after an auditory go cue before moving. The paper finds that disentanglement is more pronounced in Mihi than in Chewie, both in decoding accuracy across content and style spaces and in the multi-task covariance scores used to quantify separation between reach direction and movement dynamics.
The authors offer two readings and do not choose between them. One is a weakness of the method, possibly fixable with hyperparameters, a larger network, or new augmentations. The other is that preparation itself leads to more disentangled neural activity, which would make the score a lens on the role of preparation in movement. The paper explicitly says this needs to be studied across more individuals, so the observation is a hypothesis, not a finding about preparation.
The claim depends on the chosen augmentations
The paper states directly that, like other self-supervised methods, performance relies on the choice of augmentations, which may depend on the downstream task of interest. Only neuron dropout and localized temporal shifts are used here; the authors suggest that nonlocal view mining or sequential encoders could build further invariance. The datasets are spike-sorted recordings from primary motor cortex in two rhesus macaques, binned at 100 ms, with roughly one hundred neurons and about 1.3k data points per dataset, split 80% train and 20% test.
The practical priority that follows is specific to this design. If the goal is a single representation that supports both reach-direction and timing readouts, the two-block split with BlockSwap is the part worth keeping, since the ablation shows swapping contributes more than the alignment term alone. If the goal is only reach decoding, the margin over MYOW is not consistent across animals, and the choice between them should be made per dataset rather than assumed from the aggregate comparison.
Sources used in this answer
Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
Swap-VAE learns an unsupervised split of neural activity into a content space carrying reach direction and a style space carrying movement timing, with reach decoding on par with or better than benchmarks on four macaque datasets.