Can a JEPA learn invariant and variant latent structure without reconstruction?

Why reconstruction does not guarantee a useful split

A representation of a dynamical world can summarize observations and still hide how the latent state is organized. The paper's motivating case is a robot pushing a cube to a goal: when the camera shifts or the lights dim, nothing in the scene has moved, so the same action should be taken. A useful representation should therefore keep the stable, task-relevant content and separately express the factors that vary between related observations.

The paper argues that existing approaches to this decomposition usually obtain it through reconstruction. An encoder maps observations into latent variables, and a decoder must be able to reproduce the observations from them. That adds modeling and computational cost, and it can push representation capacity toward noise and observation details that downstream tasks do not need. More fundamentally, the paper states that reconstruction encourages a representation to preserve what is needed to reproduce observations, but does not by itself specify how that information should be partitioned into invariant and variant structure. An invertible nonlinear reparameterization can preserve the full observation distribution while mixing the underlying latent factors.

Joint embedding predictive architectures (JEPAs) offer the alternative the paper builds on: they model the latent state directly and never reconstruct observations. The gap the paper names is that no existing result recovers the invariant and variant parts of the state a JEPA learns. SplitJEPA is proposed to close that gap.

What SplitJEPA actually identifies

SplitJEPA does not claim to recover the true latent coordinates themselves. The guarantee is block-level: under stationary Gaussian predictive dynamics and a full-rank variation condition, the method identifies the invariant and variant subspaces up to independent block-wise isometries. In plain terms, the split between the two parts is pinned down, while each part may still be rotated or reflected internally.

That distinction matters for reading the results. A downstream user can rely on which information sits in the invariant block and which sits in the variant block, but not on the individual coordinates of either block matching some ground-truth factor. Because the guarantee requires no observation decoder, the paper frames the contribution as extending reconstruction-free latent recovery to invariant-variant block identification.

The proof conditions are part of the claim, not incidental setup. The result is stated for stationary Gaussian predictive dynamics and depends on a full-rank variation condition, meaning the matched observations must supply enough variation for the split to be determined. The paper's sensitivity experiment shows what happens when the underlying geometry is only approximately met: increasing nonlinear coupling strength progressively degrades the global identification stage.

How prediction and consistency produce two blocks

The method translates the theoretical conditions into representation-space losses evaluated on observed pairs. A predictive alignment loss makes the representation of one observation close to the representation of its paired observation, so the evolving state is preserved in latent space. An invariant consistency loss applies only to the designated invariant block, encouraging it to agree across matched pairs. A variance-preserving regularizer discourages collapse.

The exact analysis additionally assumes a normalization constraint: under the predictive marginal, the full representation has zero mean and identity covariance. Finite-weight variance regularization only encourages this geometry; it does not impose exact whitening. The paper is explicit that full whitening includes zero cross-covariance between the designated blocks, and that once predictive optimality yields a linear solution, normalization makes the transform orthogonal, which is what eliminates cross-block components.

The robotic implementation follows the same decomposition rather than a separate recipe. An ImageNet-initialized ResNet-50 encoder processes 128×128 RGB observations and outputs a partitioned representation. An action-conditioned predictor maps the current representation and action to a stop-gradient target of the next representation, while the invariance loss is applied only to the invariant block. Variance and covariance regularization prevent collapse. Each optimization step combines a batch of 32 adjacent temporal transitions with an independently shuffled batch of 32 matched pairs.

The synthetic test shows where identification breaks

The synthetic study is the clearest place to see the identification claim under stress. In the default configuration, SplitJEPA reaches a global R² of 0.9770±0.0003, an orthogonality error of 0.0056±0.0007, and cross-block leakage terms of 0.0187±0.0037 and 0.99995±0.00002. The comparison methods sit far lower on global R²: LeJEPA at 0.6778±0.0360 and shuffled pairs at 0.6598±0.0555, with correspondingly larger orthogonality and leakage errors.

The sensitivity table is the more informative result for judging scope. As nonlinear coupling strength rises from 0.5 to 1.0 to 2.0, global R² falls from 0.9979 to 0.9565 to 0.8765, orthogonality error rises from 0.0199 to 0.0586 to 0.1507, and the two cross-block leakage terms grow as well. The paper reads this as departures from the required geometry under finite optimization appearing jointly as increased orthogonality error and cross-block leakage.

This is a seed-0 diagnostic under the default synthetic configuration, not a broad sweep, and the paper presents it that way. The practical reading is that the guarantee is a statement about an idealized solution class, and the measured degradation tracks how far finite optimization drifts from it.

The blocks separate under viewpoint and trajectory change

The robotic experiments test whether the recovered blocks behave as intended, using a representation-distance measure normalized by the distance between randomly paired observations for each pair type. On PickCube, the normalized response to a viewpoint change is 0.18 in the invariant block and 1.51 in the variant block, making the variant response 8.4× larger. Across matched trajectories the corresponding difference is 2.1×.

PushCube shows the same pattern more strongly: viewpoint and matched-trajectory responses are 14.0× and 3.1× larger in the variant block, respectively. The paper describes this as a representation validation distinct from downstream success rate. The invariant block stays comparatively stable when viewpoint and execution factors change, while the variant block responds more strongly to the same interventions.

Temporal transferability points the same way. Using state-matched asynchronous pairs drawn from different validation trajectories at different normalized timesteps, the paper groups pairs into small, medium, and large timestep gaps and measures each block's representation change relative to its random-pair distance. Across both tasks and all gaps, the invariant block changes less than the variant block, and both change more as the gap grows. The paper interprets the relative stability of the invariant block as evidence that it captures information invariant across these observations, while the variant block reflects execution-specific factors.

Why the split matters for robustness

The controlled intervention isolates the robustness implication. The paper compares two representation interfaces while holding environment state and all other policy inputs fixed and corrupting only the variant block. The invariant-only policy retains approximately 100% of its clean performance, whereas the policy exposed to both blocks becomes progressively more sensitive as corruption magnitude increases.

The numbers make the gap concrete. On PushCube, zeroing the variant block leaves the both-blocks interface at 93.7±3.4% and the invariant-only interface at 99.9±0.2%, a 6.1 percentage-point advantage. At noise of 2.0σ the both-blocks interface falls to 44.1±4.0% while the invariant-only interface stays at 99.9±0.2%, a 55.8-point gap. On PickCube the same 2.0σ condition gives 29.6±0.8% versus 100.0±0.0%, a 70.4-point gap.

The paper's explanation is that exposing the variant block adds a pathway through which perturbations confined to it can affect downstream decisions. This supports the stated implication that once the invariant block is separated from variation, predictions based only on it are substantially more robust to perturbations of the variant component. The result is about perturbations injected into the variant block, not about every possible distribution shift.

The invariant block is not always the better input

The ablation table shows that the components are doing different work. Removing the invariance loss is the most damaging change to out-of-distribution performance: on PushCube with the invariant block, camera out-of-distribution success falls to 19.7±9.5% and visual out-of-distribution to 15.7±11.0%, against 71.5±10.4% and 64.5±12.7% for the full invariant-block model. Removing prediction also hurts but less sharply, and removing diversity leaves out-of-distribution numbers close to the full model.

The sample-efficiency results complicate any simple preference for the invariant block. On PushCube, both interfaces already exceed 90% in-distribution SR@10cm with only 10% of the demonstrations, and their camera and visual out-of-distribution performance improves as the training budget grows. On PickCube, baseline in-distribution performance rises with data, but under out-of-distribution conditions most baselines do not improve with more data and instead plateau. The paper reports that neither of its representation inputs dominates uniformly: the invariant-only input is preferable in several low-data or camera-shift settings, while the combined input becomes competitive or stronger when additional execution-specific information is useful.

The paper's stated limitation is scale. The experiments cover controlled nonlinear systems and two robotic manipulation tasks, and the authors name scaling SplitJEPA to large-scale complex real-world scenarios as the next step. The identification guarantee also carries its own conditions, so the block split should be treated as established within the tested regimes rather than assumed everywhere.

When to expose only the invariant block

The decision the paper supports is not whether SplitJEPA works but which block to hand downstream. If the concern is perturbation or nuisance variation that lives in the variant block, the intervention results favor exposing only the invariant block: it held near 100% clean performance under every tested corruption level, while the combined interface lost up to 70.4 percentage points on PickCube at 2.0σ noise.

If the task benefits from execution-specific detail, the combined interface is the better choice, and the paper reports it as competitive or stronger in those settings. The block-validation and temporal-transfer results give a way to check which situation applies before committing: measure each block's normalized response to the variation you care about, and prefer the interface whose block is stable under that variation.

The priority that emerges is therefore conditional rather than absolute. The paper's strongest evidence is for the invariant block as a robust prediction interface under variant-block corruption, with the combined representation retained when variation carries useful information. Both readings depend on the block split holding, which the paper establishes under its stated conditions and tests at the scale of synthetic systems and two manipulation tasks.

Sources used in this answer