Can multimodal reward models be trained without human preference labels?

A few thousand parameters on a frozen backbone

The reward model here is not a fine-tuned vision-language model. It is a bias-free linear head placed on mean-pooled response-token states from layer N−k of a frozen Qwen3-VL backbone, so the 2B version trains 2,048 parameters and the backbone trains none. For an image, a prompt and a response, the head turns those pooled hidden states into a single scalar score.

Training uses in-batch Bradley–Terry: each chosen response in a micro-batch of eight is compared against every rejected response, so each positive meets eight negatives per step rather than one. Supervision enters only through how those negatives are produced, which is why the paper treats the negative sampler, not the optimizer, as the thing that defines the reward.

The cost claim follows from the frozen backbone: the recipe runs at roughly a thousandth of the compute of full backbone fine-tuning. The paper also reports that a head trained on mined negatives matches full fine-tuning on the same 15k mined pairs, whereas unfreezing the backbone on 15k curated RLAIF-V pairs recovers 11 points overall to .613 — real pairs need the unfrozen model, perturbations do not.

Negatives that edit one fact, not the style

The rejected response is generated by perturbing the positive itself. For captions, operators substitute editable spans such as colour, number, size or spatial words, splice in another caption's tail, shuffle clauses, or return the caption of a different image; the recipe uses a 0.15 weight on the cross-image operator and one edited span. A caption reading "two brown horses beside a red fence" becomes "two brown horses beside a blue fence": one colour word moves, the token count is unchanged, and the pair differs only in a fact about the image.

The paper argues this matters because the usual label-free alternative — having a model write the rejected response — changes author, length and style along with the fact, and the reward then learns all of those differences. In the authors' own test, a head trained on human captions against the policy's captions for the same images scores .36 on Multimodal RewardBench, below chance, and picks the more hallucinated caption in every Best-of-N cell.

These negatives are cheap: the caption operators are string edits with no model in the loop, and the 22k-pair bank builds in 10 seconds on one CPU core. The image arm instead perturbs the prompt and re-renders both sides with a fixed seed, which takes 5 GPU-hours for 1,212 pairs.

The probe layer is worth 5.8 points

The best probe layer is never the top of the network: it is N−4 for 2B, N−6 for 4B, and N−12 for both 8B and 32B, and also N−12 for GLM-4.1V-9B with its 40 layers. The gap over the last layer is 3 to 7 points in the layer sweep and 5.8 trainable points on the real 4B run. The depth is set by the backbone rather than by its layer count, and each peak is a plateau several layers wide.

This is the one place labels enter. The layer is chosen on Multimodal RewardBench trainable accuracy, so the paper treats that benchmark as development and never as transfer evidence; VL-RewardBench and MMRB2, which no recipe choice has seen, carry the transfer claim.

By contrast, every training-side change the authors tried leaves the benchmark within .014: changing the loss form or negative composition, training on four times the pairs or twice the steps, or ordering negatives easy-to-hard or hard-to-easy. A 540-configuration grid over head learning rate, β and weight decay all reached .995 or better in distribution, and selecting among them out of distribution moved the benchmark by .0003.

Perturbations beat curated pairs at equal budget

At 15k pairs, the mined-negative head is 3 to 4 trainable points ahead of curated RLAIF-V pairs on Multimodal RewardBench and 2 to 5 on VL-RewardBench, including the hallucination split RLAIF-V was built for. The abstract summarizes this as 3 to 5 points at an equal pair budget.

The comparison is not uniform across negative types. Cross-image negatives alone reach 99% training accuracy yet fall below the untrained base on the benchmark, because they amplify a length and style shortcut; that head scores .224 on the length-sensitive safety/bias column against .451. For images, the cross-image replica transfers nothing to MMRB2 text-to-image (52.7 against a label prior of 53.2), while the same-seed planted edit accounts for all of the gain (56.0).

One untuned recipe reaches GPT-4o at 32B

The 32B probe is at or above GPT-4o on all three benchmarks and separably above it on one. On VL-RewardBench it scores .724 overall against GPT-4o's published .658, a 6.6-point gap at 3.6 standard errors of the difference. On Multimodal RewardBench and MMRB2 reasoning the benchmark cannot resolve the margin: separating either at two standard errors would take 13 and 3.5 times the released pairs.

The stronger reference is the backbone itself used as a generative judge. On MMRB2, the one benchmark the authors ran it on, that judge leads the probe by 4.7 points (64.6 against 59.9 on the four-task average). The probe's compensation is that it decodes nothing: it needs one forward pass per response, where the judge generates a judgment in both presentation orders.

The recipe is not specific to Qwen3-VL. On GLM-4.1V-9B, a second family with 40 layers, the same recipe at N−12 beats its last layer and its untrained judge on both benchmarks. Training data shares no prompt with the benchmarks and 0.1% of images with Multimodal RewardBench.

Two heads tie on the benchmark, differ by 6.4 points

Because the backbone is frozen, the objective is convex under a fixed quadratic penalty and has a unique minimizer characterized as an average of the sampler's difference vectors, each weighted by how badly the head still misorders that pair. The practical consequence is that the pair law, not the benchmark, sets the direction of the reward.

The demonstration: two probes trained on the two extreme quartiles of the same mined-negative data, ranked by the drop in CLIP image–text cosine from the real to the perturbed caption, 15k pairs each at 2B. They reach similar Multimodal RewardBench accuracy, .597 against .603 trainable, then separate on VL-RewardBench hallucination (72.0 against 78.4) and reasoning (63.4 against 56.8). As Best-of-N scorers, the CLIP-visible head beats probe-8B on CHAIRs by 3.8 points while losing 2.9 on QA.

The paper also reports that benchmark rank predicts Best-of-N rank in neither direction, and that where the probe does beat majority vote by 7 to 11 points, a training-free log-likelihood judge matches it. A benchmark score, on this evidence, does not identify which reward a head learned.

Verifier labels help deployment and hurt the benchmark

The analysis leaves one sampler untried: the policy's own errors. The authors mint labels with cheap verifiers on the policy's Best-of-N candidates — exact match for QA, CHAIR for captions — and retrain the same head on those pairs. This loop uses verifier labels, not annotators.

The resulting head cuts caption hallucination by 5.1 CHAIR points below the base Qwen3-VL-2B and 8B generators and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark than the probe it replaced. One DPO turn on the policy's own pairs also cuts hallucination by 3 to 6 CHAIR points at no QA cost.

This is the paper's sharpest tension: the supervision that moved deployment is drawn from the policy's own verified errors, and it is exactly the supervision that a benchmark-ranked selection procedure would reject.

Test a reward model on your own policy's samples

The paper's closing priority is not benchmark rank but Best-of-N behavior on the samples the reward will actually score. Two heads with near-identical Multimodal RewardBench accuracy differed by 6.4 points in hallucination judging and split in opposite directions on CHAIRs versus QA, so a single preference-accuracy number cannot tell a deployer which specialization they are buying.

The concrete checks the evidence supports are narrow. First, ask which pairs the head was trained on, since the sampler specifies the reward and the benchmark does not identify it. Second, evaluate on the target policy's own candidates rather than only on released preference pairs, because the verified-label head improved CHAIRs and image-generation pass rate while losing on every benchmark. Third, treat the layer choice as a real hyperparameter: it was worth 5.8 points over the last layer, more than any training-side change the authors tried.

The scope conditions are equally specific. The layer was selected on Multimodal RewardBench, so that benchmark is development data here, not transfer evidence. The 32B margin over GPT-4o is resolvable only on VL-RewardBench; on the other two benchmarks the comparison is a tie within the released pairs. And the backbone's own generative judge still leads the probe by 4.7 points on MMRB2, so the probe's case rests on cost and on not decoding, not on being the strongest judge available.

Sources used in this answer