Can ontology concept overlap serve as a reward for clinical question answering?

Clinical answers break both standard reward designs

Reinforcement learning post-training usually picks one of two reward designs: human preferences, as in RLHF and DPO, or binary verifiers, as in RLVR. The paper argues clinical question answering fits neither. Near-correct and correct clinical responses can differ by a single substituted entity, so random-negative preference pairs do not capture the contrast that matters. At the same time, no executable check decides whether "ceftriaxone" is the right empirical antibiotic for meningitis, so a binary verifier has nothing to run.

The paper's motivating example is a drug-class answer versus a specific compound: naming the right class but the wrong compound is a clinical error, yet cross-entropy loss assigns the same penalty to "consider an anticoagulant" as to "warfarin" when the reference answer is "heparin." The proposed fix is a soft verifier instantiated from a maintained controlled vocabulary rather than from human labels or an executable check.

UMLS CUIs turn entity overlap into a graded reward

The Unified Medical Language System assigns Concept Unique Identifiers that collapse synonymous strings such as "heart attack," "myocardial infarction," and "MI" onto one identifier, while keeping clinically distinct entities such as "heparin" and "warfarin" apart. The paper computes CUI overlap between a generated response and the reference response via scispaCy as set-level F1. That yields a graded, externally specified reward computed without a model in the loop.

This is the key contrast with ontology-aware work that scores diagnostic reasoning by traversing UMLS knowledge graph paths. Concept overlap needs no traversal: it checks only that the right concepts appear, regardless of the path between them. The paper states the trade-off directly — the approach loses the reasoning-chain signal but gains robustness on the noisy generations early-training policies produce, where graph paths are often unreachable.

Three reward terms cover different failure modes

The composite reward is R = α·R_judge + β·R_CUI + γ·R_cons, with α = 0.6, β = 0.3, and γ = 0.1. The ontology term supplies the graded entity-level signal; an entropy-normalised LLM judge covers the safety and evidence axes that overlap cannot see; and a small consistency penalty on padding and repetition keeps early-training samples scorable. The judge and the reference model stay frozen throughout training, and the policy samples G = 4 responses per question with nucleus sampling.

The division of labour matters because concept overlap counts entities, not the order they appear in or the inferential structure connecting them. A response that names every right entity in a nonsensical order still scores well. The paper's stated general pattern is to pair an ontology signal, which is structured and externally specified, with a model signal covering reasoning, safety, and evidence, so each catches what the other misses.

Token-F1 is the primary metric, not Exact Match

Evaluation uses greedy decoding on 500 held-out MedQA test questions and 500 PubMedQA questions, with Exact Match, Token-F1, and Sequence Match as metrics. The paper treats Token-F1 as primary because EM scores a response zero unless it reproduces the gold answer verbatim, discarding answers that name the right entity inside a longer response. Because n = 500 is not large, the authors also report a paired bootstrap 95% CI over 10,000 resamples and a McNemar test on the key EM deltas alongside the cross-seed standard deviation.

On Phi-3-mini (3.8B) over MedQA, the composite reward reaches EM 0.700 ± 0.005, Token-F1 0.202 ± 0.004, and SeqM 0.238 ± 0.005 over 3 seeds, beating SFT on all three. The EM gain is small in absolute terms — 0.700 versus 0.680, +2.9%, with bootstrap 95% CI [−0.03, 0.07] and McNemar p = 0.41, not significant. The Token-F1 gain is large: 0.145 to 0.202, +39%, CI [0.03, 0.09]. On Llama-3.2-3B the pattern reproduces at +35% Token-F1 and +14% EM. Cross-seed σ stays below 0.005 on every cell.

The ontology term adds three EM points

A reward ablation on Phi-3-mini varies the two learned reward terms while holding the consistency weight fixed at γ = 0.1, over the same 3 seeds as the main table. Judge-only (α = 0.9, β = 0) settles at EM 0.667 and Token-F1 0.162. UMLS-only (α = 0, β = 0.9) is noisier early, because CUI extraction on incoherent generations returns empty sets and gives no gradient, and converges to EM 0.592 and Token-F1 0.143. The full mixture scores highest on every metric at EM 0.700 and Token-F1 0.202.

The judge-to-composite gap is 3 EM points, which the paper attributes to the ontology term catching entity substitutions the judge cannot. The authors do not lean on those 3 points in isolation: at n = 500 the paired test does not clear significance, with McNemar p = 0.18. A separate calibration on 500 held-out zero-shot outputs from the training split found Spearman ρ of 0.31 for the judge reward, 0.38 for the CUI reward, and 0.44 for the composite at the chosen split, with the ordering holding on Qwen2.5-7B and Llama-3.2-3B.

Transfer to PubMedQA holds without retuning

Training on the PubMedQA train set with the same composite reward, the same weights (α = 0.6, β = 0.3, γ = 0.1), and the same prompt template as MedQA, then evaluating on 500 held-out PubMedQA test questions, improves Token-F1 over SFT by 22% on Phi-3-mini (0.183 versus 0.150) and 17% on Llama-3.2-3B (0.142 versus 0.121) without retuning the reward mixture or the judge prompt.

The transfer gain is smaller than the MedQA in-domain gain of 39% and 35%, but consistently positive. The paper reads this as evidence that the composite reward generalises beyond MedQA and is not an artifact of the LLM judge prompt or the CUI overlap statistic. The transfer test holds language and ontology constant; cross-ontology transfer, such as UMLS to MeSH or ChEBI, is named as the natural next test and has not been run.

Three negative findings bound the method

DPO under random negatives underperforms SFT for strong-prior models but helps the weakest-prior one. The rejected responses are sampled uniformly from MedQA's three incorrect options, which the paper calls weakly contrastive: rejected responses are arbitrary wrong answers rather than near-miss confusables. On Phi-3-mini DPO reaches EM 0.600 versus SFT 0.680, and on Qwen2.5-7B it reaches 0.630 versus 0.775, while on Llama-3.2-3B it reaches 0.640 versus SFT 0.570.

PPO under a sparse neural reward diverges, and GRPO with KL-in-loss collapses at 7B across all configurations. Qwen2.5-7B SFT reaches EM 0.775, higher than any GRPO result reported here, because supervised fine-tuning directly amplifies its strong medical priors. The paper therefore scopes its contribution to the 3B–3.8B sizes where GRPO is stable, and states that at 7B plain SFT on a stronger base is the better choice on EM and the ontology reward offers no benefit it can deliver.

The limitations are stated alongside these results: three seeds across two model families is an initial signal rather than a definitive variance estimate; the reward ablation is on Phi-3-mini only, with cross-model replication deferred; the single-GPU constraint kept the study below 8B parameters, and the 7B collapse does not predict behaviour at larger scales. scispaCy's EntityLinker has precision 0.78 and recall 0.67 on a labelled clinical sample, and the calibration already reflects that linker noise on model generations, which bounds the reward's validity.

Where the overlap reward is worth using

The paper's contribution is the form of the reward signal, not the specific ontology: a reward of the form "set-level overlap of canonical identifiers extracted from generated and reference text, against a maintained vocabulary" applies wherever such a vocabulary exists. Legal QA via statute and case-citation identifiers and scientific writing via MeSH or ChEBI terms are named as plausible near-term targets, but the authors state they have not tested any of these and treat the cross-domain claim as an unevaluated hypothesis.

The practical decision threshold the evidence supports is a scale split, not a universal upgrade. At 3B–3.8B, the composite reward is the better choice on Token-F1, which is the metric that credits partially correct clinical content; at 7B, SFT on a stronger base already wins on EM and the method offers no advantage. Within the composite, the ontology term is worth keeping because it catches entity substitutions the judge cannot, but its 3 EM points on Phi-3-mini do not clear significance at n = 500, so the case for it rests on the Token-F1 and ablation pattern rather than on the EM delta alone.

Sources used in this answer