[CVPR 2026] CaTok: Unified 1D Causal Tokenization for Vision and Language

TL;DR

CaTok is a 1D causal image tokenizer that enables images to be treated like text in Large Language Models (LLMs). By combining a MeanFlow decoder with interval-based token binding, it solves the "imbalance problem" in previous diffusion autoencoders. It achieves SOTA reconstruction on ImageNet (0.75 rFID) and supports lightning-fast one-step generation while maintaining the classic "next-token prediction" paradigm.

Background: The Causal Gap in Vision

The success of LLMs is built on 1D causal sequences—predicting the next word based on the previous ones. Vision, however, is inherently 2D.

1. Traditional VQGANs flatten 2D grids into 1D, but the spatial relationship is often non-causal.
2. VAR Models use multi-scale 2D tokens, which work well but break the standard "next-token" simplicity of LLMs.
3. Diffusion Autoencoders (like Semanticist or FlowMo) often either treat tokens as a global condition (no causality) or use "nested dropout" which makes the model over-rely on the first few tokens, causing an "imbalance" that hurts generation quality.

Methodology: Taming the Flow

CaTok introduces two critical innovations to solve these issues: MeanFlow Interval Binding and REPA-A Regularization.

1. MeanFlow and Time Interval Selecting

Instead of predicting the instantaneous velocity at every point, CaTok uses the MeanFlow objective. It selects a segment of 1D tokens $V_{r:t}$ and binds them to a specific time interval $[r,t]$ in the diffusion process.

• Why it works: By modeling the average velocity field over a subpath, the model learns a balanced distribution of information across the sequence. Unlike previous methods where earlier tokens are "seen" more often, CaTok ensures every token segment contributes meaningfully to the flow.

Figure 1: The overall architecture of CaTok featuring the causal ViT encoder and MeanFlow DiT decoder.

2. REPA-A: Aligning with the Giants

To speed up training, the authors propose REPA-A. It forces the encoder’s internal features to align with a Vision Foundation Model (like DINOv2). This "shortcut" provides strong semantic priors from the start, allowing the 1D tokens to capture rich visual concepts without needing millions of extra iterations.

Experiments & SOTA Results

The performance of CaTok is measured against two benchmarks: Reconstruction (Fidelity) and AR Generation (Quality).

Reconstruction Excellence

CaTok-L-256 (using 256 tokens) achieves a 22.53 PSNR, significantly higher than Semanticist and FlexTok. More importantly, it demonstrates a fine-to-coarse trend: as you reduce the number of tokens from 256 to 16, the image progressively loses detail but keeps its global structure. This is definitive proof that the causality is working.

Figure 2: Visualizing causality. Note how reconstruction quality degrades gracefully as tokens are reduced (Cols 3-7) and how different segments represent different concepts (Cols 7-10).

AR Generation

When paired with a Llama-style generator, CaTok achieves a competitive 2.95 gFID on ImageNet-1K. It proves that a continuous-valued 1D tokenizer can be just as effective as discrete VQ-based tokenizers, but with much higher flexibility for one-step sampling.

Depth Insight: Why Does This Matter?

The most significant contribution of CaTok is logical consistency. In the past, vision researchers had to choose between "Diffusion models" (good quality, bad at scaling like LLMs) and "Autoregressive models" (good scaling, hard to define causal order).

CaTok provides a bridge:

• It uses Diffusion (MeanFlow) for high-fidelity decoding.
• It uses Causal 1D sequences for LLM compatibility.

Limitations

While CaTok performs exceptionally well on ImageNet, the authors note that training visual AR models is still computationally expensive compared to pure diffusion. Furthermore, the reliance on a frozen VAE (KL-16 MAR-VAE) to reduce computation means the method is currently a "tokenizer for a latent space," not quite "pixels-to-tokens" yet.

Conclusion

CaTok represents a significant step towards the unification of Vision and Language modeling. By shifting from heuristic spatial orders to a flow-based causal 1D representation, it proves that "next-token prediction" is a viable and powerful paradigm for high-fidelity image synthesis.

Senior Editor's Note: Keep an eye on the REPA-A alignment. It suggests that the future of tokenization isn't just about reconstruction, but about inheriting the "wisdom" of frozen vision foundation models.