What new capabilities create new risks?
Unified models that both understand and generate content are more powerful than separate systems, but that power cuts both ways. TokenFlow [1] and Janus [2] both show that a single model can handle understanding (e.g., answering questions about an image) and generation (e.g., creating a new image from text) with state-of-the-art performance. This means an attacker could use the same model to create a fake image and then to interpret it in a way that supports a false narrative—something that would require two separate tools before. The risk is amplified because these models are designed to be flexible, as Janus notes, allowing each component to choose its own encoding method, which could be exploited to bypass safety filters.
The ability to generate realistic images is particularly concerning. TokenFlow reports a strong FID score of 0.63 at 384×384 resolution, meaning its generated images are nearly indistinguishable from real photos to automated metrics. Combined with the understanding capability, a malicious actor could generate a photorealistic image of a person doing something they never did, then use the model's understanding to caption it convincingly, creating a deepfake that is hard to detect. The same applies to video: UniVL [4] extends this to video and language, so the risk scales to moving images and spoken or written descriptions.
How can attackers poison the input?
Because these models learn from vast amounts of paired data (images with text, video with audio), they are vulnerable to data poisoning. If an attacker can inject malicious examples into the training set, they might be able to make the model behave badly in specific situations—for instance, causing it to misclassify a certain object or to generate harmful content when triggered by a specific phrase. The paper on cross-modal generalization [5] highlights that models often assume different modalities are perfectly aligned, which is unrealistic; this misalignment can be exploited. For example, an attacker could pair an innocuous image with a harmful caption in the training data, teaching the model to associate the two, so that later the image alone triggers the harmful output.
Adversarial examples are another input-side risk. Since the model processes multiple modalities, an attacker might craft a tiny perturbation to an image that is invisible to humans but causes the model to misclassify it or to generate a wrong description. The decoupling of visual encoding in Janus [2] and TokenFlow [1] means that the understanding and generation pathways are separate, but they share a unified transformer, so an attack on one pathway could potentially affect the other. This is a new attack surface that didn't exist when understanding and generation were separate models.
What are the output-side harms to plan for?
The most obvious output-side harm is the generation of misleading or harmful content at scale. With a unified model, you can generate text, images, and even video (as UniVL [4] does) that are coherent and contextually relevant. This could be used to create fake news articles with accompanying fake images, or to generate realistic video of a public figure saying something they never said. The risk is not just in the content itself but in the ability to automate it, making it cheap to produce disinformation campaigns.
Another output-side risk is the amplification of biases. The models are trained on large internet datasets that contain societal biases. If a model is used to generate images from text prompts, it might produce stereotypical representations (e.g., always showing a nurse as female). The paper on visual-semantic embeddings [3] shows that the learned embedding space captures regularities, such as 'image of a blue car' - 'blue' + 'red' being near images of red cars. This kind of vector arithmetic could be exploited to manipulate the model's output in unintended ways, for instance, by using biased prompts to generate discriminatory content. The key is that these models don't just retrieve content; they generate new content, so the bias can be amplified in novel ways.
About These Sources
This answer is built on 5 studies (3 peer-reviewed, 2 preprints) — published from 2022 to 2025, 2 from 2024 or later, collectively cited 1,562 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 36 papers retrieved from a database of over 500 million.
Sources used in this answer
TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
TokenFlow introduces a dual-codebook image tokenizer that achieves state-of-the-art performance in both understanding (7.2% improvement over LLaVA-1.5 13B) and generation (GenEval score 0.55), demonstrating that unified models can be highly capable—and thus more dangerous if misused.
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Janus decouples visual encoding for understanding and generation, showing that separate pathways improve performance and flexibility, but this also means that attacks could target each pathway independently, increasing the attack surface.
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
The visual-semantic embedding model shows that vector arithmetic in the embedding space can capture multimodal regularities, which could be exploited to manipulate outputs in unintended ways, such as generating biased or misleading content.
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
UniVL extends unified modeling to video and language, achieving state-of-the-art results on five downstream tasks, which means the misuse potential scales to video content, including deepfake videos and automated misinformation.
Achieving Cross Modal Generalization with Multimodal Unified Representation
Uni-Code addresses cross-modal generalization, showing that models can learn unified representations from paired data, but also highlighting that assumptions of perfect alignment are unrealistic—a vulnerability that attackers could exploit by creating misaligned training data.