[CVPR 2026] CAT: Steering T2I Models via Conditioned Activation Transport

TL;DR

Conditioned Activation Transport (CAT) is a new framework that fixes the "broken" safety-utility trade-off in Text-to-Image models. By replacing simple linear steering with nonlinear MLP transport maps and geometry-aware gating, researchers can now suppress toxic imagery (NSFW, violence, hate) without destroying the artistic quality of safe requests.

Background Positioning: This work moves beyond post-hoc pixel filtering (which can be bypassed) and architectural retraining (which is expensive) by intervening directly in the model's latent manifold during inference.

The Problem: The "Linearity Trap" in Safety Steering

Previous methods like ActAdd or Linear-ACT assume that "safety" is a simple direction in vector space (e.g., "Safe = Unsafe + Vector V"). However, the authors argue that the manifold of human toxicity is far more complex—it is non-convex, multi-modal, and context-dependent.

Existing methods suffer from two main flaws:

1. Indiscrimate Steering: They apply the same shift to every image, causing "benign" prompts (like a lady in a bikini at a beach) to become distorted or abstract because the model is confused by global "nudity-removal" vectors.
2. Topological Failure: Linear shifts cannot "unbend" a non-convex crescent of toxic activations into a tight cluster of safe ones.

Methodology: Nonlinearity and Precision Gating

The CAT framework introduces two key innovations to navigate the latent space more intelligently.

1. Nonlinear MLP Transport Maps ($T_{h}eta$)

Instead of a single translation vector, CAT uses a multi-layer perceptron (MLP) to learn a local vector field. This allows the model to handle "Context Dependency" (e.g., a "white power" sign is hate speech, but a "love thy neighbor" sign is safe, even if both prompts use the word "sign").

Figure 1: Synthetic validation showing how MLP Transport (bottom row) correctly morphs complex topologies compared to linear baselines (middle rows).

2. Geometry-Aware Conditioning

CAT uses a Mahalanobis distance metric to create an elliptical decision boundary around the "unsafe" manifold.

• If the activation falls inside this toxic region: Steering is applied.
• If it falls outside (Benign): The model is left untouched, preserving 100% of the original image quality.

SafeSteerDataset: The Foundation

To train these maps, the authors created SafeSteerDataset, containing 2,300 pairs of prompts that are semantically identical except for the safety violation (e.g., "A person with a tattoo" vs. "A person with a swastika tattoo"). This allows the transport map to isolate exactly what needs to change while leaving the rest of the image context intact.

Results: Safety Without Sacrifice

The researchers tested CAT on the Z-Image (Diffusion Transformer) and Infinity (AutoRegressive) backbones.

Linear methods achieved "safety" by effectively breaking the model's ability to generate coherent images (the CLIP score plummeted). In contrast, CAT successfully neutralized the threat while keeping the CLIP score nearly identical to the original unaligned model.

Figure 2: Qualitative comparison showing CAT successfully removing toxic symbols and violence while maintaining semantic structure.

Critical Insight: Multimodal Intervention is Key

The paper reveals a fascinating find: Steering only the text encoder is insufficient. If you steer only text, the vision backbone can still "hallucinate" toxic features based on its prior training. CAT performs best when applied to both the text and vision components of the model, creating a synergistic defense that neutralizes the request and rectifies the generation trajectory simultaneously.

Conclusion and Future Outlook

CAT represents a significant shift from "blunt force" safety filters to "surgical" latent interventions. By acknowledging the complex geometry of unsafe manifolds, this work paves the way for generative models that are safe by design, yet remain creatively uninhibited.

Limitations: While powerful, inference-time steering doesn't "delete" the knowledge from the weights; determined attackers might still find "adversarial cracks" in the conditioning boundary. Future work will likely focus on making these geometric boundaries even more robust against adaptive prompting.