How can subjective quality be evaluated for compact document conversion models?

Why human ratings are the gold standard for subjective quality

Subjective quality is, by definition, what people perceive. The most direct way to evaluate it is to run a controlled human study where participants rate the output of your conversion model. This is exactly what researchers do for other visual content: a 2023 study on high dynamic range (HDR) videos gathered more than 20,000 human quality judgments from 310 videos to build a reliable quality database [4]. Similarly, a 2021 virtual reality study had 36–34 subjects rate foveated videos to capture perceptual quality [1]. For document conversion, you would follow the same principle: recruit a representative set of users, show them the converted documents, and collect their ratings on a scale (e.g., 1–5). This gives you ground truth for what 'good' means to real users.

However, human studies are expensive and time-consuming. That's why researchers also develop objective models that predict human ratings. The key is to validate these models against human data first. For example, the HDR study used their human judgments to improve an existing objective model (VMAF) and showed it performed better on both HDR and standard dynamic range content [4]. So, for your compact document conversion model, you should start with a small human study to establish a baseline, then use that data to tune or select an objective metric that you can run automatically on many documents.

Objective metrics can approximate human perception—if they're tuned for your content

Objective quality metrics are algorithms that try to predict how humans would rate quality. They are essential for large-scale evaluation because they are fast and cheap. But they only work well if they are designed for the specific type of content and distortion you care about. For example, a 2022 study on 3D point clouds found that existing objective models only achieved 'limited success' in predicting human ratings, so they proposed a new model that significantly outperformed them [2]. Similarly, a 2025 study on display content videos found that existing video quality metrics failed to capture display-specific issues like color distortion and motion artifacts, so they built a specialized model [3].

For document conversion, the distortions are different—blur, compression artifacts, OCR errors, layout shifts. A generic image quality metric may not capture what matters. The 2025 DeQA-Doc study addressed this by adapting a multi-modal large language model (MLLM) to assess document quality, and it 'significantly outperformed existing baselines' across diverse degradation types [5]. This suggests that modern AI-based metrics can be more flexible and accurate for document-specific quality. But remember: these metrics are only as good as their training data. You should always validate them against human ratings for your specific use case.

The gap between best-case and typical-case evidence

The strongest evidence for subjective quality evaluation comes from large, controlled human studies. For example, the 2023 mobile cloud gaming study collected 14,400 subjective ratings from 600 videos [6]. That's the kind of scale you need to trust a quality model. But such studies are rare and expensive. In contrast, many published models are tested on smaller or less diverse datasets, which limits their generalizability. For instance, the 2021 VR study used only 10 reference videos [1], and the 2022 point cloud study used 20 source point clouds [2]. So, when you read claims about a model's performance, check the size and diversity of the test set.

For compact document conversion models, you likely won't have the resources to run a 20,000-judgment study. But you can still do a mini version: collect ratings from 10–20 people on a representative set of documents, then compare those ratings to your objective metric. If the correlation is high, you can trust the metric for larger batches. If not, you need to adjust the metric or collect more human data. This pragmatic approach bridges the gap between the ideal and the feasible.

About These Sources

This answer is built on 6 peer-reviewed studies — published from 2021 to 2025, 2 from 2024 or later, 5 in Q1 journals, collectively cited 219 times — selected as the most relevant from 6 studies that passed quality screening, drawn from 57 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions