Do Reaction Feasibility Verifiers Agree with Chemists Across Different Candidate Sources?

FREA labels feasibility, not experimental success

FREA is a benchmark of 751 reactions, each labeled by expert chemists under a written criterion rather than by whether a database records it or a model can recover it. A reaction counts as feasible if it lists all substantive reactants and admits reasonable conditions that yield the target products in isolable form, in a single step or a credible one-pot operation.

That definition matters because feasibility is used loosely in synthesis planning. A reaction may be called feasible because it is recorded, because a forward model recovers its product, because its reaction center is precedented, or because it works in a laboratory. These senses agree on routine transformations but diverge on the cases that matter most for evaluation: an unprecedented but sound disconnection may be executable, while a precedented transformation applied to an incompatible substrate is not. FREA measures the validity of a retrosynthesis proposal as stated, not experimental feasibility, which would additionally require identified conditions and acceptable yield.

The criterion is structural, not stoichiometric. Annotators ask where each product fragment comes from rather than whether element counts balance, because counts can balance while a substantive partner is missing and can fail to balance when nothing is missing. A positive label therefore requires more than a recognizable transformation, and a negative label denotes a defect in the proposal as stated, not proof that the transformation is impossible under every conceivable condition.

Four candidate sources test different failure modes

The 751 reactions come from four sources: proposals of eight retrosynthesis models, zero-yield experimental records, edits of recorded reactions by large language models, and candidates from five negative candidate generation methods. Annotators label each reaction without seeing its source, so the labels are independent of how verifiers are built.

The sources are not interchangeable. Published reaction data favor successful outcomes, and automatically generated negative candidates need not be infeasible, so neither recorded nor generated data alone gives an unbiased basis for judging a verifier. Zero-yield records enter as candidates and are labeled under the same criterion as every other source: a reported failure records an unsuccessful attempt without settling whether other reasonable conditions could work.

Because the four sources are pooled with an unweighted mean, the benchmark's aggregate number is a cross-source average rather than a deployment-weighted estimate. The source proportions were chosen for evaluation, not estimated from any particular planning workflow, so the four-source mean need not reflect performance in a given pipeline.

No verifier leads across all four sources

Comparing LLMs, forward models, and verifiers trained on generated negatives, the paper reports that no verifier leads across all sources. Criterion-prompted LLMs are competitive with dedicated verifiers, and Claude Opus 5 achieves the highest mean balanced accuracy. Balanced accuracy is the mean of the acceptance rate for feasible reactions and the rejection rate for infeasible reactions, so it covers systems such as the LLM and top-10 decisions for which no continuous score is extracted.

Forward models are strongest on retrosynthesis proposals, yet at the evaluated operating points they reject most feasible edits of recorded reactions. That split is the central comparison: the same systems that look reliable on model proposals behave very differently on LLM-edited recorded reactions, so a single aggregate score hides which screening task a verifier is actually suited to.

The LLM results describe two models under one shared prompt with no examples from FREA and no parameter updates, so they may change with the prompt or inference settings. Continuous-score thresholds were fixed on held-out generated pairs before being applied to FREA, which means a low acceptance rate at these thresholds does not establish that a model cannot achieve a different precision-recall tradeoff.

Forward models fall below chance on alternative disconnections

Looking within the generated source reveals a sharper failure than the aggregate scores suggest. Both forward models score below chance when separating infeasible alternative product disconnections from the same feasible reference reactions, with AUROCs of 26.2 and 32.5. AUROC is the primary metric here, and a value below 50 means the ranking is worse than random on that comparison.

This is a within-source result, not a cross-source average, and it concerns a specific construction: infeasible alternative disconnections paired against feasible generated candidates. The paper's framing is that the strongest verifiers of model proposals can rank infeasible candidates above feasible ones, which is exactly the error that matters when a verifier gates proposals entering synthesis planning.

Adding generated negatives to forward training improves this comparison and gives the highest mean AUROC across sources, but it reduces performance on other constructions. Those selective gains leave open whether negative supervision improves verification beyond particular constructions, especially on model proposals.

Negative supervision raises mean AUROC but not on proposals

To test whether negative supervision addresses these weaknesses, the authors use an accompanying corpus of over 14 million recorded reactions and generated negative candidates, with no benchmark reaction in the training data. In matched training comparisons, adding a mixture of generated negatives to forward training raises mean AUROC across sources, but these gains do not extend to retrosynthesis proposals.

Varying the generation method shows a further tension: the largest gain on generated candidates coincides with worse proposal screening. In the discriminative verifier family, training on a single negative method produces very different AUROC profiles by source, and the mixture settings do not dominate every source. For example, at the reported 20,000-step budget, the RR-only model reaches 74.5 AUROC on retrosynthesis proposals while the mixture with 7.10M pairs reaches 67.8, even though the mixture scores higher on zero-yield records.

The paper reads these results as motivation to design negatives for transfer to model proposals rather than to maximize average performance on generated candidates. Because the comparisons are single-run and the benchmark has limited coverage, the authors present the improvement directions as hypotheses rather than established recipes.

Thresholds and source composition shape the verdict

Decision thresholds are frozen on 10,000 held-out generated pairs and then applied unchanged to FREA. Scores are log-likelihoods for the forward models, raw logits for the verifier models, and probabilities for the shallow baseline. Thresholds calibrated on generated pairs may transfer poorly to other sources, so a low acceptance rate at these thresholds is a statement about one operating point rather than about a model's ceiling.

Source composition also shapes the verdict. All four sources derive from ORDerly's patent and literature records, and screening removes parsing failures and selected structural defects before annotation, so the benchmark does not represent the full range of chemistry or unfiltered planner outputs. The generated source also uses the same generation methods as the negative training corpus, so gains on that source do not by themselves demonstrate transfer to model proposals or experimental records.

The retrosynthesis ranking analysis covers eight models on their sampled proposals at the evaluated operating points, with bootstrap intervals over proposals within each model. It does not establish a general ordering of planners or verifiers.

Expert labels agree at 77.9 percent before adjudication

Even with a written criterion, chemists may differ on whether conditions are reasonable, a target is isolable, or an omitted component serves a permitted auxiliary role. The labels rely on two chemists whose independent agreement was 77.9 percent with Cohen's kappa of 0.55 before adjudication. Joint review produces a shared reference but does not eliminate possible annotation errors or establish agreement with other experts, and the final binary labels do not express the uncertainty of borderline judgments.

Several comparisons also depend on small minority classes, and the training comparisons are single-run. The authors state that hypotheses about label reliability and transfer therefore require tests across seeds, architectures, and corpora.

The benchmark evaluates whether individual proposals admit reasonable conditions under the stated criterion, without specifying those conditions or testing the reactions experimentally. A feasible label therefore does not establish a practical procedure, an acceptable yield, or the quality of a complete synthesis route, where compatibility between steps and purification also matter.

Screen verifiers on the source they will judge

The paper's practical priority follows from the source split rather than from any single average. Because no verifier leads across all four sources, and because strong aggregate performance can conceal below-chance behavior on alternative disconnections, a verifier should be selected and calibrated on data representative of the screening task it will actually perform.

The most decision-relevant comparison in FREA is therefore not the cross-source mean but the gap between retrosynthesis proposals and LLM edits of recorded reactions: forward models lead on the former and reject most feasible examples of the latter at the evaluated operating points. For anyone choosing a gatekeeper for a planning pipeline, that gap, together with the below-chance AUROCs of 26.2 and 32.5 on alternative disconnections, is the result to check against their own candidate mix before trusting an aggregate score.

The accompanying corpus of over 14 million recorded reactions and generated negatives supports that check, but the matched training results show why it is not sufficient on its own: adding generated negatives raised mean AUROC across sources without improving performance on model proposals, and the largest gain on generated candidates coincided with worse proposal screening. Performance on model proposals is the test the paper puts at the center of negative-data design.

Sources used in this answer