What’s changed: text-only metrics are no longer enough for multimodal quality
For years, evaluating the quality of multimodal outputs—like translations that include an image—relied on automatic metrics designed for text alone. But recent research shows this approach is flawed: these metrics often fail to capture errors that depend on visual context. In a 2026 study on multimodal machine translation, both n-gram-based and embedding-based metrics struggled on a contrastive evaluation set designed to catch context-dependent errors, meaning they couldn't tell when a translation was wrong because it ignored the image [1][3]. This overturns the assumption that text-only evaluation can adequately judge multimodal systems.
The same study found that all metrics could distinguish image-aware from image-agnostic systems on general test sets, but that's not enough—real-world errors often hinge on subtle visual cues. So, the new picture is clear: to evaluate subjective quality of flattened multimodal representations, you need metrics that explicitly incorporate visual information, not just text.
How to actually evaluate: use image descriptions and human ratings
The most promising approach is to feed automatically generated image descriptions into supervised evaluation metrics. In the 2026 study, incorporating such visual information into supervised metrics yielded better alignment with human judgment compared to text-only metrics [1][3]. This means that if you're evaluating a multimodal system, you should generate a textual description of the image and include it in your metric's input—this helps the metric 'see' the visual context.
Human judgment is also crucial, but it's not static. The study observed that when human evaluators were given the opportunity to see the visual context, their ratings were often substantially revised [1][3]. This suggests that subjective quality is not a fixed property—it depends on the context provided to the evaluator. So, for a fair evaluation, always present the multimodal input (image + text) to human raters, not just the text output.
When does this work? It depends on the task and the metric
The effectiveness of visual context in evaluation varies by task. In the 2026 study, all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, but they failed on contrastive sets designed to capture context-dependent errors [1][3]. This means that if your evaluation set is too easy (e.g., only general sentences), even text-only metrics might look fine, but they'll miss the hard cases where the image matters.
For other multimodal tasks like sentiment analysis in videos, the challenge is similar: you need to jointly model spoken words and visual gestures, as shown in the MOSI dataset [5]. That dataset provides per-frame visual and per-millisecond audio annotations, which are essential for training and evaluating models that capture cross-modal interactions. So, the lesson is: tailor your evaluation to the specific multimodal task, and include contrastive examples that test whether the model actually uses the visual information.
About These Sources
This answer is built on 5 studies (2 peer-reviewed, 3 preprints) — published from 2022 to 2026, 4 from 2024 or later, 1 in Q1–Q2 journals, collectively cited 348 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 43 papers retrieved from a database of over 500 million.
Sources used in this answer
Visual Semantics in MT Evaluation: Do Image Descriptions Help with Assessment of Multimodal MT Quality?
In a 2026 study on multimodal machine translation, incorporating automatically generated image descriptions into supervised evaluation metrics improved alignment with human judgment, while n-gram and embedding metrics failed on contrastive sets designed to capture context-dependent errors.
Perceptual Visual Quality Assessment: Principles, Methods, and Future Directions
A 2025 overview of perceptual visual quality assessment (PVQA) outlines both subjective methods (human ratings) and objective methods (algorithmic prediction) for evaluating quality across multimedia types, including images, video, VR, and generative AI content.
Visual Semantics in MT Evaluation: Does Image Descriptions Help Assessing Multimodal MT Quality?
A duplicate of [1] (same authors and findings) confirms that visual context in evaluation metrics improves human alignment and that human ratings are often revised when visual context is provided.
FOAA: Flattened Outer Arithmetic Attention for Multimodal Tumor Classification
FOAA, a flattened outer arithmetic attention mechanism for multimodal fusion, achieves state-of-the-art results on two tumor classification datasets, demonstrating that attention-based fusion can improve feature quality for multimodal tasks.
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
The MOSI dataset provides the first opinion-level annotated corpus for sentiment and subjectivity in online videos, with per-frame visual and per-millisecond audio annotations, enabling joint modeling of spoken words and visual gestures.