[CVPR 2026] DACO: Safeguarding MLLMs via Dictionary-Aligned Sparse Autoencoders
Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs
The paper introduces Dictionary-Aligned Concept Control (DACO), a framework for safeguarding Multimodal Large Language Models (MLLMs) via inference-time activation steering. Utilizing a curated dictionary of 15,000 multimodal concepts (DACO-400K) and Sparse Autoencoders (SAEs), DACO achieves state-of-the-art safety performance across benchmarks like MM-SafetyBench and JailBreakV while preserving general-purpose utility.
TL;DR
Researchers from UPenn and Amazon have developed DACO (Dictionary-Aligned Concept Control), a surgical framework to prevent Multimodal Large Language Models (MLLMs) from generating harmful content. By combining a curated 15,000-concept dictionary with Sparse Autoencoders (SAEs), DACO can identify and "mute" harmful activation patterns during inference without affecting the model's general intelligence.
Problem & Motivation: The "Fragile Guardrails" of Multimodal AI
While MLLMs like LLaVA or Qwen-VL are impressive, their safety mechanisms are surprisingly fragile. Adversaries can bypass safety filters using multimodal jailbreaks—for instance, embedding harmful instructions within an image that looks benign to a text filter.
Current defenses fall into three (mostly flawed) camps:
- Prompting: Easy to implement but easily bypassed by clever attackers.
- Response Filtering: Requires running the model twice, doubling the cost.
- Fine-tuning: Computationally expensive and often leads to "catastrophic forgetting" or "over-refusal" (where the model becomes too scared to answer even safe questions).
The authors' insight? Don't change the weights; steer the activations. By treating the model's internal representation as a mix of individual "concepts," we can selectively suppress the toxic ones.
Methodology: The DACO Framework
The core of DACO is a massive effort in "Representative Reading." The team curated DACO-400K, a dataset linking 15,000 WordNet concepts to 400,000 image-caption stimuli.
1. Curating the Dictionary
By running stimuli through a frozen MLLM, they extracted "concept vectors"—mathematical directions in the latent space that correspond to specific ideas like "violence," "fraud," or "helpfulness."
2. SAE Training & Alignment
Instead of training a Sparse Autoencoder (SAE) from scratch with random noise, DACO pre-loads the SAE decoder with these concept vectors. This "Dictionary Alignment" ensures that the learned dimensions (atoms) of the SAE are grounded in human-understandable semantics from day one.
Figure 1: The DACO pipeline—from stimulus retrieval to inference-time compositional steering.
3. Inference-Time Intervention
During generation, DACO decomposes each token's activation into SAE atoms. If a "harmful" atom (like #13331: aggressive) peaks, DACO zero-purity it out. Conversely, it can amplify "helpful" atoms to steer the model back toward a safe, compliant response.
Experiments: Safety Without the Sacrifice
The framework was tested on multiple backbones (Qwen2.5-VL, LLaVA1.6, InternVL3.5) across several safety benchmarks.
Performance Gains
DACO consistently hit the highest safety scores. For Qwen2.5-VL, it improved the defense success rate from 0.442 to 0.990 on MM-SafetyBench (MS-R). Unlike previous steering methods, the impact on general utility (MMMU) was negligible.
Table 1: Comparison of DACO against prompting and other steering baselines. Note the high safety scores alongside stable utility metrics.
Efficiency and Over-refusal
One of DACO's standout features is its speed. It only adds 14.62% latency per token, making it viable for production environments. Furthermore, it avoids the "nanny-state" problem: in tests like MOSSBench (designed to trigger over-sensitive models), DACO remained constructive where other models simply refused to speak.
Figure 2: DACO successfully avoids "over-refusal" on benign prompts involving "cigarettes" by identifying the context as safe.
Conclusion: A New Paradigm for AI Alignment
DACO proves that we don't need to choose between a smart model and a safe one. By combining the interpretability of concept dictionaries with the decomposing power of SAEs, we can control multimodal models at the "thought level."
Future Outlook: The authors plan to expand the dictionary to more specialized domains and adapt the technique for video and audio models. This work suggests that the future of AI safety lies in transparent, granular intervention rather than opaque, blanket filters.
Disclaimer: This blog post is based on the paper "Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs". Image credits to the original authors.