[Meta AI & KAUST] Neural Computers: When the Model Becomes the Operating System
Neural Computers
This paper introduces Neural Computers (NCs), a novel machine form that unifies computation, memory, and I/O within a learned latent runtime state. By instantiating NCs as video models (NCCLIGen and NCGUIWorld), the authors demonstrate the first steps toward a Completely Neural Computer (CNC) capable of rendering interfaces and responding to actions without relying on a conventional external operating system.
TL;DR
Meta AI and KAUST have proposed a paradigm shift in computing: Neural Computers (NCs). Instead of using AI to control a computer, they’ve built a model that is the computer. By unifying computation, memory, and I/O into a single "learned runtime state," these models (instantiated as video-to-video generators) can execute CLI commands and GUI actions with high fidelity, rendering the screen in real-time while maintaining internal execution context.
Positioning: This is not just a better "AI Agent"—it is the first serious attempt at a Completely Neural Computer (CNC), aiming to replace modular hardware/software stacks with a "neural latent stack."
The Problem: The Gap Between Model and Machine
Currently, AI exists as a "layer" on top of traditional systems. Whether it's an agent using a tool or a world model predicting the next frame, the executable state still lives outside the model (in the actual OS, the CPU, or the simulator).
The authors argue that this separation is the root of "symbolic brittleness." Conventional computers are precise but fail under noise and require rigid programming. NCs, by contrast, utilize distributed numerical semantics, offering robustness and the ability to "learn" the programming language of user interaction directly from I/O traces.
Methodology: Building a Latent Runtime
The researchers built two prototypes based on the Wan2.1 diffusion model:
- NCCLIGen: A terminal-based NC that learns from millions of
asciinematraces. It renders terminal physics (scrolling, prompt wrapping) directly from text prompts. - NCGUIWorld: A desktop-based NC that takes mouse/keyboard actions and rolls out desktop frames (Ubuntu/XFCE environment).
The Neural Latent Stack
The core innovation lies in how actions are injected. Rather than just feeding actions as text, the team experimented with four modes: External, Contextual, Residual, and Internal.
Figure: The four modes of action injection. The 'Internal' mode, which adds cross-attention sub-layers inside transformer blocks, proved superior for fine-grained GUI control.
To solve the problem of "cursor drift" (where the mouse pointer disappears or jitters), they introduced SVG Mask/Reference Conditioning. By providing the model with an explicit visual "anchor" for the cursor, accuracy skyrocketed from a measly 8.7% to 98.7%.
Experiments & Results: Symbolic reasoning vs. Rendering
The paper reveals a fascinating nuance in AI "reasoning." While initial models failed basic math (arithmetic probes), the authors found that reprompting (providing better context/logic) allowed the same stagnant weights to improve from 4% to 83% accuracy.
Figure: Reprompting dramatically boosts symbolic performance, suggesting the model is a strong "renderer" of conditioned logic rather than a native calculator.
Key Findings:
- Data over Scale: For GUIs, 110 hours of high-quality "goal-directed" data outperformed 1,400 hours of random exploration (FVD score 14.72 vs 48.17).
- Character Precision: The NCCLIGen can render readable 13px fonts with high character accuracy (0.54), proving that video latents can indeed carry precise text-based interface state.
The Road to CNC (Completely Neural Computer)
The paper doesn't claim to have finished the quest. A "Complete" Neural Computer must satisfy four criteria:
- Turing Completeness: Ability to express general computation.
- Universal Programmability: Inputs can "install" new routines that remain callable.
- Behavior Consistency: It must not "forget" or drift unless explicitly updated.
- Machine-Native Semantics: Leveraging tensor-to-tensor transformations as primitives.
Figure: The evolution from conventional computers to Neural Computers, unifying the functions of agents and world models.
Critical Insight & Conclusion
The true value of this work is the realization that interaction logs are programs. If every movement on a screen is a trace that can be learned, we no longer need to write drivers or OS kernels. We simply "show" the Neural Computer how a system should behave.
However, the "Reasoning Gap" remains. While NCs are masterful at rendering the look and feel of a computer, internal logic—like stable execution over long horizons and precise symbolic math—still requires architectural breakthroughs beyond today’s video diffusion models. As the authors conclude, we are moving toward a world where the computer is not a box of chips, but a persistent, learned state in a massive neural network.