[CVPR 2025] CompACT: Planning in 8 Tokens — Drastically Accelerating World Models via Extreme Compression

Executive Summary

TL;DR: CompACT is a breakthrough in world modeling that challenges the status quo of high-fidelity latent representations. By compressing environment observations into as few as 8 discrete tokens (approx. 128 bits), it achieves a 40x speedup in planning latency without sacrificing accuracy. This is accomplished by prioritizing "planning-critical" semantic information over photorealistic reconstruction.

Background Positioning: While the field has trended toward larger, more detailed latent spaces (like SD-VAE with 784 tokens), CompACT shifts the focus toward extreme semantic abstraction. It positions itself as a practical solution for real-time robotic control, bridging the gap between slow, high-quality generative simulators and the need for nimble, real-time Model Predictive Control (MPC).

Problem & Motivation: The Quadratic Wall

The core bottleneck in modern world models is the token count. Most state-of-the-art models extract hundreds of tokens per image frame. Since most world models utilize attention-based architectures (like Transformers or DiTs), the computational cost grows quadratically with the number of tokens.

In real-world deployment (e.g., navigation or robot arm control), waiting 3 minutes for a model to "think" through a trajectory is unacceptable. The authors identify a "perceptual trap": existing tokenizers prioritize textures and lighting—details that are vital for human-eye photorealism but largely irrelevant for calculating where a robot should move next.

Methodology: Semantic Abstraction & Generative Decoding

The authors introduce CompACT, which operates on a simple yet powerful insight: Planning requires semantics, not pixels.

1. Semantic Encoding via Frozen Backbones

Instead of training an encoder from scratch, CompACT uses a frozen DINOv3 model. Because DINOv3 is already trained to understand object-level semantics and spatial layouts, it acts as a filter that discards high-frequency "noise" (like shadows or textures). A cross-attention-based resampler then distills these features into a tiny set of 8-16 tokens.

2. Generative Decoding

To handle the "ill-posed" problem of reconstructing an image from only 16 tokens, the authors don't attempt direct pixel regression. Instead, they use a masked generative decoder. This decoder predicts tokens of a target tokenizer (like VQGAN) conditioned on the 16 compact tokens. In essence, it "imagines" plausible fine-grained details that are consistent with the high-level semantic layout.

Figure 1: The CompACT pipeline: (a) Tokenization, (b) World Model training in latent space, and (c) Decision-time planning via MPC.

Experiments: Order-of-Magnitude Gains

The efficiency gains of CompACT are staggering when applied to practical tasks:

• Navigation (RECON/SCAND): Compared to the SD-VAE baseline (784 tokens) used in state-of-the-art navigation world models, CompACT reached similar Absolute Trajectory Error (ATE) while reducing planning latency from 178.78 seconds down to 5.78 seconds (at 16 tokens).
• Action-Conditioned Video Prediction: On the RoboNet manipulation dataset, CompACT not only generated videos 5.2x faster but also preserved "action consistency" much better than standard discrete tokenizers.

Key Evidence: Modular Attention

Visualizations show that CompACT tokens are not just random numbers; they learn to attend to semantically meaningful objects. In a robot scene, one token might focus on the "end-effector," while another tracks the "target object."

Figure 2: Each latent token learns to attend to specific, modular scene elements (e.g., animals in ImageNet or robot arms in RoboNet).

Critical Analysis & Conclusion

Takeaway

CompACT proves that for downstream control, less is more. By leaning on the "mental abstractions" provided by vision foundation models, we can bypass the expensive requirement of photorealism. This moves world models away from being "offline simulators" and closer to "real-time brains" for robots.

Limitations & Future Work

• Context Dependency: While CompACT excels at high-level navigation, the authors note that 8 tokens might struggle with extremely cluttered scenes where every small object matters for contact-rich manipulation.
• Beyond Visuals: The next frontier is applying this extreme compression to multi-modal inputs (tactile, audio) to see if a similarly tiny token set can represent a "global" state.

CompACT sets a new benchmark for efficiency, suggesting that the future of AI agents lies in their ability to discard the irrelevant and focus on the 8 tokens that truly matter.