NOVA: Rethinking World Models Through Weight-Space Rendering

TL;DR

The dominant paradigm of world modeling—encoding pixels into latent vectors and decoding them back—is hitting a wall of interpretability and scalability. NOVA (Neural Ontology for Visual Abstraction) flips this script. Instead of predicting pixels, it predicts the weights of a tiny neural network that represents the image. This "Render, Don’t Decode" approach yields a world model that is resolution-independent, naturally disentangled, and remarkably stable over long horizons.

Problem & Motivation: The Decoder Bottleneck

In current SOTA world models like Dreamer or Genie, the spatial decoder is often the "heavy lifter," sometimes accounting for over 87% of the model's parameters. These decoders are:

1. Resolution-Locked: You cannot query a 64x64 model for a 1024x1024 image without retraining.
2. Opaque: The latent space is a "black box" where identity and motion are hopelessly entangled.
3. Inefficient: They waste capacity on high-frequency noise rather than structural physics.

The authors of NOVA argue that Implicit Neural Representations (INRs)—which map (x, y) coordinates to RGB values via an MLP—are the ideal "structured" latent space. By treating the weights of these INRs as the state, we treat the data point as a function, not a grid.

Methodology: The "Weight-Space" Ontology

NOVA organizes the visual world into a three-level hierarchy using simple additive logic:

• The Anchor ($\bar{z}$): A shared "base" weight set that absorbs the static background.
• The Content ($z_t$): Per-frame offsets that represent object identity.
• The Motion ($u_t$): Latent actions that dictate how $z_t$ evolves.

Architectural Flow

During training, an encoder maps a frame to a weight offset $z_t$. An Inverse Dynamics Model (IDM) looks at $z_t$ and $z_{t+1}$ to "guess" the action $u_t$. The Forward Dynamics Model (FDM) then predicts the next state using an additive operation: $z_{t+1}=A(z_t)+B(u_t)$.

Figure 1: The NOVA workflow. Note the parameter-free analytical renderer that replaces the traditional spatial decoder.

Crucially, because rendering is an analytical function of the coordinates, you can sample the coordinate grid as densely as you like at inference time, achieving Zero-Shot Super-Resolution.

Experiments: Disentanglement and Physics

One of the most striking results is NOVA's ability to perform Motion Retargeting. Because $z_t$ (content) and $u_t$ (motion) are additively decoupled, you can take the "identity" of a digit '5' and force it to follow the "trajectory" of a digit '1' simply by swapping the action vectors at runtime.

Long-Horizon Stability

While pixel-based models like LAPO collapse into blurry messes over 1,000 steps, NOVA maintains structural integrity. By operating in the weight space of a coordinate network, the model is forced to maintain a "sharp" functional representation of the objects.

Figure 2: Comparing long-term forecasting. NOVA stays crisp while baselines fade or collapse.

Zero-Shot Super-Resolution

On the WeatherBench dataset, NOVA was trained on a 32x64 grid. At test time, by querying a 1024x2048 grid and applying a Nyquist-Shannon frequency mask to prevent aliasing, it produced high-fidelity weather fronts that outperformed standard bilinear interpolation.

Figure 3: Meteorological forecasting at 32x scale. NOVA preserves sharp gradients without artifacting.

Critical Analysis & Conclusion

NOVA proves that weight-space learning isn't just a mathematical curiosity—it's a viable path for scalable world models. Its core strength lies in latent orthogonality: the ability to edit a scene's content without breaking its physics.

Limitations:

• The model sometimes encodes redundant info (like brightness) in the action space.
• It current relies on a fixed INR architecture; scaling to extremely complex "in-the-wild" videos may require more expressive, perhaps hierarchical, INRs.

Final Takeaway: By moving away from pixel-reconstruction and toward functional-rendering, NOVA provides a blueprint for world models that are as interpretable as they are expressive. For digital artists and roboticists alike, this separation of "what an object is" from "how it moves" is the holy grail of controllable simulation.