The Multimodal Achilles' Heel: A Comprehensive Survey of Attacks on LVLMs

TL;DR

As Large Vision-Language Models (LVLMs) like LLaVA and GPT-4V become ubiquitous, their security landscape has grown exponentially complex. This paper provides the first systematic roadmap of how these models can be broken, manipulated, or poisoned. By bridging the gap between vision-based adversarial noise and text-based jailbreaking, the authors reveal that the very multimodal integration that makes these models powerful is also their greatest security liability.

The Motivation: Why Multimodal is Harder to Secure

Traditional security research has treated Large Language Models (LLMs) and Vision Encoders as separate silos. However, LVLMs create a new "attack surface" where:

1. Complexity is Amplified: Vulnerabilities in the vision encoder (e.g., CLIP) can be leveraged to bypass the safety alignment of the LLM backbone (e.g., Vicuna).
2. Cross-Modal Interaction: A seemingly benign image containing hidden "typographic" text can override a system's safety instructions—a feat impossible in unimodal systems.

Methodology: Breaking the Black Box

The survey breaks down attacks into four distinct methodologies, each targeting a different stage of the model lifecycle:

1. Adversarial Attacks (The Signal Noise)

These involve using gradient-based optimization (like PGD or FGSM) to add imperceptible noise to images. The goal is to cause "Cognitive Bias" where the model misidentifies objects or generates specific malicious strings.

• Insight: Adversarial images can act as "hijacks," forcing the model to invoke unauthorized API calls or external tools.

2. Jailbreak Attacks (The Alignment Breaker)

Unlike LLM jailbreaks that rely on clever role-playing, LVLM jailbreaks often use the visual modality.

• Technique: "Typography attacks" render harmful questions as text within an image. Since many models prioritize visual "truth" over system prompts, they end up answering forbidden questions.

3. Prompt Injection (Instruction Manipulation)

Attackers inject malicious instructions directly into the prompt space. In the multimodal context, this can be Unimodal (just text or just image) or Multimodal (coordinated attacks across both).

• Example: Scaling a malicious "invisible" overlay on a screenshot to control an autonomous web agent.

4. Data Poisoning (The Silent Killer)

By introducing "Backdoors" during the instruction-tuning or RLHF phase, attackers can ensure a model behaves normally until it sees a specific "trigger" (e.g., a specific red patch in an image), at which point it executes a malicious command.

Key Results and Comparison

The paper provides an invaluable comparison of existing literature. One of the most striking findings is the transferability of these attacks. An adversarial image generated for a white-box model like LLaVA can often deceive closed-source giants like GPT-4V or Gemini Pro Vision.

Deep Insight: Future Trends

The authors argue that the next frontier of LVLM security lies in Attack Practicality. Most current attacks require gradients (White-box). Future threats will likely focus on:

• Black-box Hard-label Attacks: Guessing gradients solely by querying the model.
• Cross-Modal Dependencies: Designing noise that "resonates" across both text and image embeddings to evade detection.
• Data Bias Amplification: Subtly influencing model decision-making by exploiting inherent training data biases.

Conclusion

This survey is a call to action for the AI community. As we move toward autonomous multimodal agents, the "plug-and-play" nature of vision and language components must be replaced with a unified, security-first architecture. The repository maintained by the authors (Awesome-LVLM-Attack) will serve as a critical resource for developers aiming to build the next generation of robust AI.

Senior Editor's Note: This work effectively maps out the "Wild West" of multimodal security. The most critical takeaway is that multimodality is a double-edged sword: it provides the context needed for intelligence, but provides the noise needed for deception.