Why accuracy metrics alone can't measure subjective quality
Subjective quality is about whether a model's output feels right to a human, not just whether it matches a label. A 2023 study probed multimodal LLMs by systematically changing visual prompts, text prompts, and adding extra knowledge, then testing on VQA (visual question answering) data. The result: existing models didn't truly understand the input—they mostly fit the training data distribution, so their performance collapsed when prompts changed in ways that required real understanding [1]. This means a model can score high on a standard benchmark yet fail in real-world use where prompts vary. So, evaluating subjective quality requires testing with diverse, unexpected prompts to see if the model's behavior aligns with human expectations, not just checking if it gets the 'right' answer on a fixed set.
Domain-specific benchmarks reveal quality gaps in generation
For generation tasks, subjective quality depends on how well the output matches human expectations in a specific domain. A 2025 study on chart understanding and generation created a high-quality instruction-tuning dataset using GPT-4, then trained ChartLlama. On standard benchmarks—ChartQA, chart-to-text, and chart-extraction—ChartLlama outperformed all prior methods [2]. This shows that when a model is trained on diverse, high-quality instructions, it can produce outputs that are more aligned with what humans expect in that domain. But the key is the evaluation: using benchmarks that cover a wide variety of chart types and tasks, not just one narrow test. So, for subjective quality in generation, you need benchmarks that reflect the full range of real-world inputs and outputs, and you need to compare against human judgments, not just automated scores.
Aligning with human perception: lessons from video quality assessment
In video quality assessment, subjective quality is defined by human ratings. A 2021 study tackled the challenge of evaluating in-the-wild videos (no reference, complex distortions) by training a unified model on multiple datasets simultaneously. They explicitly modeled human perception—content dependency and temporal-memory effects—and aligned predictions to subjective quality scores. The model outperformed state-of-the-art methods across four public datasets (LIVE-VQC, LIVE-Qualcomm, KoNViD-1k, CVD2014) [4]. This demonstrates that to evaluate subjective quality, you need a model that is trained to predict human ratings, not just objective distortion metrics. The same principle applies to multimodal understanding and generation: you need human-annotated quality scores or human preference data to train and evaluate your model, and you need to account for content and context effects.
Multimodal features matter for subjective understanding
Subjective quality in multimodal tasks often hinges on how well the model integrates visual and linguistic cues. A 2024 study on meme understanding argued that metaphors in memes require joint modeling of visual and linguistic features, not just text. They proposed a method that generates multimodal metaphorical features by deriving visual characteristics from linguistic attributes, using a text-conditioned generative adversarial network. On the MET-Meme benchmark, this approach significantly outperformed baselines on emotion classification and intention detection [3]. This shows that for subjective understanding—like grasping a meme's intended meaning—evaluation must test whether the model can fuse visual and textual information in a way that captures the underlying metaphor. If a model only uses text, it misses the visual context, leading to poor subjective quality.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2025, 2 from 2024 or later, 2 in Q1 journals, collectively cited 250 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 40 papers retrieved from a database of over 500 million.
Sources used in this answer
What is the limitation of multimodal LLMs? A deeper look into multimodal LLMs through prompt probing
Prompt probing on VQA data showed that existing multimodal LLMs do not truly understand input content but rather fit training data distribution, indicating that subjective understanding is lacking.
A Multimodal LLM for Chart Understanding and Generation
ChartLlama, trained on a GPT-4-generated instruction-tuning dataset, outperformed all prior methods on ChartQA, chart-to-text, and chart-extraction benchmarks, demonstrating that high-quality domain-specific training data improves generation quality.
Generating Multimodal Metaphorical Features for Meme Understanding
A method generating multimodal metaphorical features (MMMC) significantly outperformed baselines on emotion classification and intention detection on the MET-Meme benchmark, showing that joint visual-linguistic modeling improves subjective understanding.
Unified Quality Assessment of in-the-Wild Videos with Mixed Datasets Training
A unified video quality assessment model trained on mixed datasets with explicit perceptual alignment outperformed state-of-the-art models on four in-the-wild video datasets, showing that human-perception-aligned training improves subjective quality prediction.
LLM Multimodal Traffic Accident Forecasting
In traffic accident forecasting, using large multimodal models (LLaVA) with deep probabilistic reasoning enhanced real-time responsiveness, suggesting that multimodal integration can improve subjective decision-making in autonomous driving.