What does 'subjective quality' mean for retrieval-free knowledge internalization?
When you internalize a document into a model's parameters, you can't just check if the model can repeat the text. Subjective quality is about whether the model can use that knowledge in a natural, helpful way—answering questions accurately, staying coherent, and not losing its general abilities. The papers here define quality through two lenses: domain-specific question answering (QA) accuracy and general performance on standard benchmarks like IFEval, MMLU, and MSBench [3]. A model that nails domain questions but forgets how to follow instructions or reason generally is not high quality.
For dialogue models, subjective quality is often judged by how informative and natural the responses are. One study turned knowledge documents into simulated multi-turn dialogues and found that models trained on these dialogues outperformed other retrieval-free methods and matched retrieval-augmented systems [4]. This suggests that a good subjective test is to have a human or an automated judge rate the informativeness and coherence of responses in a conversation, not just whether the right fact is mentioned.
How do you actually measure subjective quality?
The most direct way is to run a QA test on the document collection without giving the model access to the source. The 2026 IAR study used exactly this: they measured domain QA accuracy and general performance on standard benchmarks [3]. They found that their staged approach—inject, align, recover—improved domain QA accuracy by an average of 3.6 percentage points over standard fine-tuning, while also improving general performance by 12.1 points. That means you can get better domain knowledge without sacrificing general smarts, which is a key part of subjective quality.
Another approach is to simulate real usage. The dialogue study created simulated multi-turn dialogues from documents and used them to train dialogue models. They then compared the models' responses to those from retrieval-augmented systems, finding comparable performance [4]. This shows that you can evaluate subjective quality by having the model engage in a conversation about the document and judging the responses for informativeness and coherence—either by humans or by automated metrics.
What are the trade-offs and caveats in evaluation?
There's a tension between domain knowledge and general ability. The IAR study explicitly addressed this by adding a 'recover' stage to merge the domain-adapted model with the base instruction model. They found that while some methods (like LoRA) could win on individual general metrics, only IAR achieved leading domain internalization while keeping a strong general profile [3]. So when evaluating subjective quality, you need to look at both axes—a model that only excels at domain trivia but fails at general reasoning is not high quality.
Another caveat is that evaluation methods can be biased by the training data. The dialogue study noted that retrieval-augmented approaches rely on finely annotated training data, which is costly. Their retrieval-free method avoided that, but the simulated dialogues might not capture all the nuances of real user queries [4]. Also, the IAR study used continued pretraining objectives like continuation and rewrite, which might not align perfectly with how users ask questions. So subjective quality evaluations should include a variety of question types and conversational contexts to be robust.
About These Sources
This answer is built on 4 studies (3 peer-reviewed, 1 preprint) — published from 2023 to 2026, 2 from 2024 or later, 1 in Q1–Q2 journals — selected as the most relevant from 4 studies that passed quality screening, drawn from 31 papers retrieved from a database of over 500 million.
Sources used in this answer
SAGE: A Framework of Precise Retrieval for RAG
SAGE, a RAG framework, improved QA quality by 61.25% over baselines and cut token costs by 49.41%, but it focuses on retrieval, not retrieval-free internalization.
Toward a Model to Evaluate Machine-Processing Quality in Scientific Documentation and Its Impact on Information Retrieval
A quality model for scientific documents, based on accessibility, content, and reproducibility, found that only 21% of 120 documents were high quality, showing that document structure affects retrieval success.
Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization
The IAR staged post-training framework improved domain QA accuracy by 3.6 percentage points and general performance by 12.1 points over vanilla SFT across multiple model families, showing that internalization can be done without losing general abilities.
Retrieval-free Knowledge Injection through Multi-Document Traversal for Dialogue Models
KiDG, a retrieval-free method that converts documents into simulated dialogues, outperformed other retrieval-free methods and matched retrieval-augmented performance in dialogue models, while being cheaper for domain transfer.