Can sparse-view 3D animal behavior representations improve neural encoding?

Why 2D behavior features leave information unresolved

Behavior is a primary lens for studying the brain, and large-scale video now makes detailed behavioral measurement possible. The paper frames two common ways of turning that video into features: pose estimation, which tracks predefined keypoints, and nonlinear video embeddings, which capture richer information but are hard to interpret. Pose tracking is interpretable but discards everything beyond the chosen keypoints; nonlinear embeddings keep more detail but resist scientific reading.

Both approaches also typically operate on 2D projections of an animal that moves in three dimensions. The paper argues that capturing behavior in 3D is necessary to resolve viewpoint ambiguities and recover fine-grained movements that cannot be inferred from 2D observations alone. The target, then, is a representation that is simultaneously dense, interpretable, and three-dimensional.

Two camera views are the hard case for 3D recovery

Two-view imaging is a common configuration in animal behavior experiments and is cheaper, more compact, and easier to deploy than larger multi-camera rigs. It is also the minimal setup for recovering 3D kinematics through stereo reconstruction. But the two views in this data have limited overlap, mainly covering the nose, forepaws, and wheel.

Existing 3D foundation models were trained on datasets with dense visual overlap, often ten views or more, so they struggle here. The paper reports specific failure modes on the two-camera IBL data: Depth Anything 3 produces plausible monocular point clouds but fails to infer camera poses for cross-view fusion; VGGT applied zero-shot collapses to a degenerate single-view 2D shortcut; E-RayZer shows a similar failure and, even after fine-tuning, produces overlapping, flattened point clouds. This is the gap SABLE is built to close.

How SABLE forces genuine 3D structure

SABLE is a self-supervised framework. A first multi-view transformer predicts camera intrinsics and extrinsics from the input views and converts them into ray tokens. Those ray tokens are fused with image tokens from a DINOv3 encoder, masked at the patch level, and passed to a second multi-view transformer that produces 3D-structured embeddings. A Gaussian-splatting decoder turns those embeddings into per-pixel 3D Gaussian primitives, which a differentiable renderer projects back into 2D images.

The critical addition is geometric regularization. Because reconstruction from sparse, minimally overlapping views is ill-posed, reconstruction loss alone often collapses to trivial 2D solutions. SABLE therefore supervises its predicted point clouds with pseudo-point clouds built from monocular depth estimates, aligned across views using animal pose keypoints as correspondences and merged with a rigid transformation estimated by the Kabsch algorithm. A geometric loss then pulls the predicted point cloud toward this pseudo target. The paper's ablation is direct: without the geometric loss, reconstruction degenerates.

SABLE also differs from E-RayZer in its masking strategy. E-RayZer masks entire views, but in a two-view setting removing one view leaves only a single view, which provides insufficient geometric constraints and leads to 2D collapse. SABLE instead applies patch-level masking across both views to preserve cross-view context.

What the neural benchmarks actually measure

The evaluation uses two datasets. On IBL, the analysis covers 2,619 units from 6 probe insertions across 3 mice, spanning 37 anatomical regions. Neural activity is sampled from 400 trials per held-out session across three sessions, giving 1,200 one-second trials; spike trains are binned at roughly 16.7 ms (60 bins per trial, matching the 60 Hz video frame rate), and neurons with mean firing rates below 5 Hz are excluded. Behavioral measurements include wheel movements and video-based pose estimates from synchronized left and right cameras.

Encoding and decoding are distinct measures and should not be conflated. Neural encoding is reported in bits per spike, asking how well the behavioral representation predicts neural activity. Decoding is reported as PSNR and SSIM, asking how well neural activity can be used to reconstruct behavior. A model can do well on one and not the other, so the paper's claim rests on both.

Cheese3D provides a second test: six synchronized views of head-fixed mice capturing whole-face movements at 100 Hz and 1280×1024 resolution, downsampled to 60 Hz. Ten sessions without neural recordings (about 5,500 synchronized frame pairs) are used for training, and the single session with neural recordings is used for encoding evaluation. To mimic the IBL challenge, the authors select the top-left and top-right views, which have limited overlap.

Where SABLE wins and why model size is not the explanation

Across IBL and Cheese3D, the paper reports that SABLE learns 3D representations that match or exceed prior state-of-the-art performance in neural encoding and decoding. On reconstruction, SABLE recovers consistent cross-view geometry, separates foreground from background in depth, and avoids the degenerate solutions seen in the baselines. On Cheese3D, DA3 misregisters the two point clouds around the nose, and VGGT leaves background geometry intermingled with the mouse.

A model-size ablation addresses an obvious alternative explanation. The authors compare against a keypoint-based model, a ResNet-AE-18 autoencoder, and a ViT masked autoencoder (BEAST), then test larger variants: ResNet-152 and a ViT-Large BEAST. Increasing model size does not improve performance for either baseline, and SABLE outperforms both the best ResNet-AE and BEAST variants across the encoding and decoding metrics. The paper concludes that the advantage is not simply explained by scaling the backbone, pointing instead to the representation-learning architecture.

Zero-shot transfer works, but only within a narrow animal scope

Once pretrained across animals, SABLE serves as an off-the-shelf model that generalizes zero-shot to unseen animals without animal-specific calibration or retraining. This is a meaningful practical property: a lab does not need to collect labeled 3D data or retrain per animal to obtain 3D behavior embeddings.

The scope is narrower than that sentence alone suggests. The paper states that SABLE has only been applied to head-fixed mouse datasets; extending it to freely moving animals and other species is left as future work. The zero-shot claim therefore applies to unseen mice in this head-fixed setting, not to new species or unconstrained behavior. The framework is also not inherently limited to two views: the appendix reports single-view reconstruction on IBL after fine-tuning, and 3-of-6-view reconstruction on Cheese3D, though the six-view case is noted as straightforward because of large camera overlap, with E-RayZer already performing well there.

The priority is geometric supervision, not a bigger backbone

The paper's most decision-relevant comparison is not SABLE versus a larger model but SABLE with versus without geometric regularization. The ablation shows that removing the geometric loss causes reconstruction to degenerate, and the model-size ablation shows that scaling ResNet-AE or BEAST does not close the gap. For a reader deciding what to change in a sparse-view behavior pipeline, the evidence points to the geometric inductive bias, built from monocular depth and pose-keypoint correspondences, as the component that makes 3D recovery possible at all.

That priority comes with a boundary the paper itself sets. The geometric signal is derived from external depth and pose models used as frozen preprocessing, and the demonstrated benefit is on head-fixed mouse data with two minimally overlapping views. Whether the same geometric supervision transfers to freely moving animals or other species is untested here, so the case for adopting it rests on the sparse-view, head-fixed regime the paper actually evaluates.

Sources used in this answer