Why subjective evaluation alone is unreliable
When humans judge image quality, their personal experience and biases creep in. A 2023 study in mammography had five radiographers evaluate 1,000 mammograms, and the agreement among them was poor—kappa values of 0.165 for one view and 0.135 for another, where 1.0 means perfect agreement and 0 means none. Even the best pair of raters only reached moderate agreement (0.433). This means that if you ask two experts to rate the same image, they often disagree, so any single human's subjective score is shaky ground for quality assessment [1].
The same study suggested a fix: have two people evaluate, and if they disagree, bring in a third. But that's still human judgment. The authors also proposed using a computer program to measure geometric features like the angle and length of the pectoral muscle or symmetry—objective numbers that don't depend on who's looking. That's a key lesson for AI image detection: subjective scores need to be anchored to measurable, repeatable criteria [1].
Adding reasoning and objective features makes evaluation more robust
To move beyond pure subjectivity, you can extract concrete features from images and use them to explain or justify quality judgments. A 2023 paper on defect detection introduced an 'AI-Reasoner' that pulls out morphological characteristics (like shape and size) from images and uses decision trees to reason about them, then outputs charts and text explanations. This approach was tested on 366 defect images and successfully explained the AI model's predictions, making the reasoning behind quality assessments transparent and less dependent on a single human's gut feeling [5].
Similarly, in aerial image detection, a 2021 method used ground sample distance (GSD)—a physical measure of pixel size—to improve object detection. By combining GSD with the size of detected regions, the model could estimate physical object size, which served as a powerful prior for classification. This shows that incorporating objective, physical measurements into AI reasoning can improve accuracy and reduce ambiguity, which is exactly what you need when evaluating image quality for reasoning-based tasks [3].
What the evidence shows about agreement and conflict
The studies here agree on one central point: subjective human evaluation is inconsistent, and adding objective, quantitative measures helps. The mammography study [1] and the AI-Reasoner paper [5] both highlight the need for structured, repeatable evaluation methods, though they approach it from different angles—one in medical imaging, the other in industrial defect detection. This convergence strengthens the case for hybrid approaches that combine human oversight with automated feature extraction.
However, there's a tension: while objective measures improve consistency, they may miss the 'communicability' or emotional impact of an image, which is crucial in fields like advertising. A 2025 study on AI-generated images in visual communication introduced a database with 2,500 images spanning 14 topics and 8 emotion types, annotated for both preference and reasoning. They found that existing quality assessment methods, including large multimodal models, have strengths and weaknesses in predicting preference and reasoning—meaning that purely objective metrics may not capture subjective aspects like emotional resonance [2]. So, the best evaluation likely blends objective features with human judgment, but you must decide what 'quality' means for your specific use case.
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2021 to 2025, 1 from 2024 or later, 1 in Q1 journals, collectively cited 101 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 69 papers retrieved from a database of over 500 million.
Sources used in this answer
The impact of subjective image quality evaluation in mammography
In a study of 1,000 mammograms, five radiographers showed poor overall agreement (Fleiss' kappa 0.165 for CC and 0.135 for MLO projections), with the best pair reaching only moderate agreement (Cohen's kappa 0.433), demonstrating that subjectivity heavily impacts image quality evaluation.
AI-generated Image Quality Assessment in Visual Communication
The AIGI-VC database, containing 2,500 AI-generated images across 14 advertisement topics and 8 emotion types, provides coarse-grained preference annotations and fine-grained descriptions, and an empirical study of existing IQA methods and large multimodal models revealed their strengths and weaknesses in preference prediction, interpretation, and reasoning.
GSDet: Object Detection in Aerial Images Based on Scale Reasoning
The GSDet method for aerial image detection incorporates ground sample distance (GSD) as a prior, converting GSD regression into probability estimation and combining it with RoI sizes to estimate physical object size, improving detection over existing two-stage methods on the DOTA dataset.
Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art
In an annotation experiment, using initial images as prompts alongside text improved subject representation in AI-generated art across all subject types, with the most significant improvement for concrete singular subjects, and icons and photos produced high-quality generations of different aesthetics.
Morphological Image Analysis and Feature Extraction for Reasoning with AI-Based Defect Detection and Classification Models
The AI-Reasoner extracts morphological characteristics (DefChars) from defect images and uses decision trees to reason with them, exporting visualizations and textual explanations; tested on 366 images, it effectively explained the IE Mask R-CNN model's predictions, improving transparency in AI-based defect detection.