GAUGE: Auditing LLM-as-a-Judge Gates That Rank Broadly but Mis-Anchored on Satisfaction

TL;DR

Bodhwani, Tran, and Wei introduce GAUGE, a reusable offline audit protocol that asks whether the common release gate for task-oriented agents, a persona-driven user simulator plus an LLM-as-a-Judge scorer, ranks candidate agents the same way a verifiable non-LLM reward would. Across 25 agents from six providers on tau2-bench and SimulatorArena, the gate's aggregate ranking is high, Spearman correlation 0.94, but its satisfaction anchor is almost uninformative about success: 57.5 percent of conversations rated satisfied by a blind human panel failed the verifiable task, matching the sample's 57.3 percent base failure rate. The same gate loses resolution precisely where release decisions are made, promoting the lower-reward agent on 31 percent of near-equal pairs. The proposed remedy is a calibrate-then-trust cadence: use a zero-cost completion bit for truncation regressions, reserve the paid judge for semantic and coarse ranking checks, and re-audit when the evaluation configuration changes.

Background positioning

This work is best read as a measurement-validity audit, not a new scoring model. Its contribution is to place the routine simulator-plus-judge release gate under the same conceptual lens as psychometric validation: does the instrument measure what the decision depends on? The paper explicitly separates two properties that industry practice conflates: ranking validity, whether the gate orders agents like a grounded reward, and construct validity, whether the construct it optimizes, satisfaction, actually stands for task success. Table 1 positions GAUGE against prior evaluation work by criterion: it is the only listed audit that tests ranking validity against a non-LLM reward, operates at the release-decision unit, quantifies the satisfaction-success gap, scales across providers, and grounds claims in human ratings.

The timing matters. Agent evaluation increasingly moves from static task sets to interactive simulations, but the simulator and judge are not neutral observers; they co-produce the transcript and the score. GAUGE treats the composite evaluator as an instrument to be validated before it is trusted. That is a different stance from most LLM-as-a-Judge papers, which validate judges against human preference or satisfaction. Here the human-validated signal can still be the wrong construct.

Problem and motivation

The de facto release loop for customer-service, tool-use, and tutoring agents is cheap: drive each candidate variant against persona-conditioned user simulators, score the transcripts with an LLM judge, and promote the higher-scoring variant. The loop rests on an unmeasured assumption: that the judge's score ranks variants the way a real, outcome-grounded evaluation would. Known judge biases, such as self-preference and position bias, are not the central problem here. The deeper failure is that satisfaction itself may be the wrong anchor, and the judge may inherit that misalignment while still correlating strongly with human satisfaction.

GAUGE isolates a specific failure mode with quantitative force. A blind human panel rated satisfaction on a 150-transcript tau2 sample, and 57.5 percent of conversations rated satisfied, at least 5 on a 7-point scale, failed the customer's verifiable task. The paper reports that this rate is statistically indistinguishable from the stratified sample's own 57.3 percent base failure rate, so satisfaction is not merely weak; it is essentially uninformative at the transcript level, with area under the curve 0.44. The panel's satisfaction score, however, is human-validated: two independent LLM judges reproduce human satisfaction at correlations 0.846 and 0.827. A gate can therefore pass a human-preference test and still certify the wrong outcome.

The failure is not only construct misalignment; it is also resolution collapse in the regime that matters for release decisions. Broad ladder ranking looks excellent, Spearman correlation 0.94, because the six-provider capability span is wide. But among near-equal strong agents, where teams actually compare frontier candidates, the gate misselects the lower-reward agent far more often. The paper's headline decision-disagreement number is 31 percent on close pairs versus under 1 percent on wide pairs. That means aggregate correlation can be high while the gate is unreliable for the exact comparisons it is deployed to make.

Core: GAUGE turns the release gate into a measured instrument

Formalizing the gate, the reward, and two validities

GAUGE evaluates a set of candidate agents under persona strata and task domains. For each candidate agent, persona, and domain, the agent and user simulator produce a transcript. The audit attaches a per-transcript verifiable label from a non-LLM oracle, and separately scores the same transcript with subjective signals such as the judge gate, a satisfaction proxy, or a human panel. The release-decision unit is the agent variant, not a single transcript, because per-cell scores are too noisy to act on directly. The agent-level gate score and verifiable reward are therefore expectations over personas and domains:

Here a is a candidate agent, p is a persona stratum, d is a domain, tau is the resulting transcript, gate is the scalar score assigned by the judge signal, and r is the binary verifiable success label produced by the non-LLM oracle. The expectation symbol means an average over the evaluated persona and domain cells, so R is the agent's task success rate under the simulation protocol, while G is its aggregate gate score. The formula is deliberately coarse: it defines the object that a release engineer actually compares across variants. If no oracle reward is available, or if aggregation is done too finely, the audit cannot certify the gate at the release-decision unit.

The first validity question is whether the gate's ordering matches the oracle's ordering:

In this expression A is the set of evaluated agents, G is the agent-level judge score, R is the agent-level verifiable reward, and rho is the Spearman rank correlation between the two vectors. This formula captures what most evaluation papers report, but GAUGE's point is narrower: high rho can be a property of breadth rather than resolution. If the candidate pool spans very weak to very strong agents, a satisfaction-driven judge can still order them well because capability dominates the score. The correlation coefficient hides exactly where it fails, namely the close comparisons that determine whether one near-peer agent replaces another.

The second validity question is whether satisfaction predicts success within the transcripts it approves:

In this definition s is any subjective satisfaction signal, such as a judge gate, an LLM proxy, or a human panel rating, and s_sat is the satisfaction threshold used by the paper, at least 5 on a 7-point scale for human and Anthropic-scale ratings, or at least 8 on a 10-point scale for SimulatorArena. The probability is the share of satisfied transcripts that the non-LLM oracle scored as task failures. If satisfaction were construct-valid as a success proxy, this conditional failure rate should be substantially below the base failure rate. Instead, for the human panel the paper reports 57.5 percent satisfied-but-failed against a 57.3 percent base failure rate, and transcript-level discrimination AUC 0.44. That is the formal definition of the satisfaction-success gap.

Figure 1 shows the audit end to end: an agent and persona simulator converse on a substrate, and four evaluators score each transcript. Three signals are subjective satisfaction signals, the LLM-as-a-Judge gate, the LLM human-proxy, and a blind human panel. One signal is the objective non-LLM reward. This architecture is important because GAUGE does not claim the simulator is the sole culprit. Every central claim is anchored on the simulator-independent verifiable reward and on human panel results; the simulator is simply part of the gate being audited.

Measurement design: disjoint judges, verifiable oracles, and controlled degradation

GAUGE operates on two substrates. The primary substrate is tau2-bench retail and airline, where the non-LLM oracle includes database-state and action checks; the airline reward is fully deterministic, while retail is predominantly deterministic with a deterministic database check gating an LLM-scored natural-language assertion. The replication substrate is SimulatorArena math tutoring, which uses human satisfaction ratings and human-graded correctness. The headline cross-provider ladder covers 25 scored model-temperature configurations from 14 base models across six providers, about 3,700 transcripts in total. This wide capability span makes broad ranking validity measurable. The complementary controlled-degradation set covers 12 configurations of a single model, crossed with 6 persona strata and 2 domains for 720 transcripts, and supplies a known broken-versus-working axis.

The rubric design is a key control. The policy-aware gate is an operations-supervisor rubric that reads the full transcript, including tool calls and task goal, and scores service quality with policy adherence and task resolution first. The satisfaction proxy is process-blind: it reads only user-visible turns, with tool calls and task stripped, and rates how the conversation felt. The paper's rationale for disjointness is explicit: with different roles, evidence, and vocabulary, the two signals cannot agree by a shared-rubric artifact. That matters because many LLM-judge evaluations risk tautology when the judge and proxy see the same text and use similar wording. Here, the proxy and gate separate the constructs: the gate assesses whether the task resolved; the proxy assesses whether it felt served.

Human ratings ground the subjective axis. A blind three-person panel rates a stratified 150-transcript tau2 sample in a fully crossed design, with Krippendorff alpha 0.79 and a human-human ceiling correlation of 0.85. The panel ratings are blind to model variant and outcome, and instructions tell raters to judge the feeling of being served, not the mechanics of task resolution. That design choice strengthens the central inversion: even when humans are asked only about experience, their satisfaction judgments do not track success.

The controlled-degradation set is a positive control. A single model, Claude Sonnet-4.5, is degraded only through inference-time flags such as output-token caps, step caps, and error tolerance. No prompt or code is edited. By construction, degraded configurations should rank below good and medium configurations, and the verifiable reward confirms a clean separation: degraded-tier mean reward is 0.05 versus about 0.65 for the good and medium tiers. This design answers the question of what happens when an agent is genuinely broken but superficially helpful. Without it, the broad six-provider grid cannot provide a known bad case, because frontier models are broadly competent.

To move from rank correlation to decision risk, GAUGE defines a pair-level error rate. Let a set of agent pairs, P, be near-equal when their verifiable rewards differ by less than 0.1:

This decision-disagreement rate counts how often the gate promotes the agent with the lower verifiable reward. The symbols G and R are agent-level scores, the pair condition uses one gate score exceeding another while the corresponding reward is strictly lower, and P is a selected set of pairs. Its role is to translate a global correlation into a release-relevant error: if two candidates are statistically or practically close, how often does the cheap gate choose the wrong one? The threshold of 0.1 is not arbitrary in the paper's framing: about 30 percent of all pairs and 55 percent of top-half pairs differ by less than 0.1, and the paper treats this as near the oracle's resolution under per-agent sampling noise. If a signal fails only on wide pairs, it is a poor detector. If it passes wide pairs but fails close pairs, it is valid only outside the regime where release decisions live.

Attribution: human validation and mis-anchoring coexist

The most striking result is that the gate can be both human-validated and mis-anchored. Human-panel satisfaction and verifiable success are essentially uncorrelated, with Spearman rho minus 0.147, and the satisfied-but-failed rate matches the base failure rate. The policy-aware gate behaves differently because it sees whether the task resolved: among conversations the gate accepted, the failure rate is 20.0 percent against a 40.2 percent base rate on the natural task mix, yielding relative risk 0.50 and AUC 0.73. The process-blind proxy sits between these two interpretations of satisfaction: its satisfied-but-failed rate is 32.7 percent versus a 40.2 percent base rate, but its AUC is only 0.49, close to chance. The paper's reading is that the distinction is not between LLM and human evaluators; it is between evaluators with outcome-grounded evidence and evaluators relying on subjective experience alone.

The gap is not limited to the word satisfaction. The human panel rated five subjective dimensions: satisfaction, respect or tone, clarity, perceived helpfulness, and would-return. Appendix B reports that all five are decorrelated from verifiable success, with absolute Spearman correlations no larger than 0.17, and each inverts at comparable high-rated-fail rates. The dimensions are highly collinear, so they are better interpreted as one latent "this interaction felt good" factor measured several ways. That factor is orthogonal to task success. This generalization matters: the problem is not a poorly specified satisfaction rubric, but the construct itself when used as a success proxy.

Ranking validity then explains why the gate is not useless. The Opus-4.8 gate recovers the verifiable reward ordering at rho 0.94 across the 25-agent ladder, and the ranking survives domain splits, independent judge checks, and an independent provider simulator swap. GPT-5.4 user-simulation preserves the ordering at rho 0.93 against 0.94 under the original Sonnet-4.5 simulator, weakening the claim that same-family simulator affinity drives the result. But resolution collapses among close pairs: the primary Opus-4.8 gate promotes the lower-reward agent on 31.0 percent of near-equal pairs, versus 0.9 percent on wide pairs. Appendix F reports that the close-pair disagreement base-model cluster-bootstrap 95 percent interval runs from 11.6 to 50.0, with the lower bound still far above the wide-pair rate. The threshold is also not knife-edge: Opus-4.8 disagreement is 45 percent, 31 percent, and 26 percent at absolute reward gaps below 0.05, 0.10, and 0.15 respectively.

Judge self-preference is present but does not explain the broad failure. The two-provider design isolates same-family inflation: the Opus-4.8 judge inflates Claude agents relative to the GPT-5.5 judge by a difference-in-differences of plus 0.75 on a 7-point scale, and the deployed Sonnet-4.5 judge shows a similar plus 0.67 effect against GPT-5.4. However, the frontier judges' agent-level rankings agree at rho 0.92, and all four judges recover the broad reward ordering between rho 0.84 and 0.94. The self-preference effect is therefore best read as a caveat on absolute accept-reject thresholds, not as the cause of the broad ranking validity. This is a subtle but important distinction: a gate can be rank-valid while still being biased in ways that flip a fixed score cut.

Experiments and evidence

Table 2 reports the central construct-gap numbers for the human panel, the process-blind proxy, and the policy-aware gate. It compares the failure rate among satisfied conversations with the pool's own base failure rate, then adds relative risk and transcript-level discrimination.

The interpretation is not simply that satisfaction is "noisy." The human panel's satisfied-but-failed rate is effectively equal to the base rate, so knowing that a panel rated a transcript satisfied does not reduce the probability that the task failed. The relative risk of 1.00 makes this precise. The policy-aware gate does reduce risk, roughly halving it against the natural mix base rate, and its AUC rises to 0.73. The proxy, which never sees tools or task state, behaves like satisfaction: its AUC is 0.49, near chance. These numbers are the empirical backbone of the separation between ranking validity and construct validity.

Figure 2 reinforces this by showing satisfaction versus verifiable task success across raters and substrates. On the left, the share of satisfied conversations that failed is concordant across populations, with rates from roughly 47.6 to 59.5 percent on tau2, and SimulatorArena human ratings also invert at 38.7 percent for conversations rated at least 8 out of 10. On the right, empirical success probability does not rise monotonically with satisfaction for any rater. The plot rules out the intuition that human satisfaction is merely a weak success predictor; in the audited data, it is flat.

Table 9 from Appendix F converts the broad ranking result into decision risk. It reports how often each signal promotes the lower-reward agent when the pair's verifiable rewards are close, with absolute reward difference below 0.1, versus when the pair's rewards are far apart.

This table is where the practical warning lives. The close-pair set contains 87 agent pairs, the wide-pair set contains 211, and two exact-tie pairs are excluded from the 300 total unordered agent pairs. The primary gate is nearly perfect on wide pairs but fails on nearly a third of close pairs. The effect is not unique to one judge or one signal: proxy satisfaction also misranks 29.9 percent of close pairs. The authors further show that the 31 percent number is not merely reference-ranking noise: among 60 close pairs flipped by at least one signal, only 4 are flipped by all five, and 22 are flipped by exactly one. Thus the instability is signal-specific rather than all signals sharing the same error pattern, although they remain highly correlated overall.

Figure 3 presents the positive control. Across known quality tiers, verifiable reward collapses from 0.66 to 0.05, while satisfaction signals stay nearly flat and the policy-aware gate is only partially sensitive, moving from 0.54 to 0.21. The sharpest case appears in the degradation table: configuration D7 is step-starved, receives verifiable reward 0.00, yet earns human satisfaction 4.64 out of 7, the highest among degraded configurations. This is the concrete failure mechanism: a trajectory can be cut off before the verifiable action while still reading as helpful turn by turn.

Table 3 summarizes the cost-performance division of labor. The controlled-degradation set has 12 variants, and the reported correlations are with the verifiable reward on that broken-versus-working axis.

The table supports a narrow but useful mitigation: a judge-free completion bit is nearly as effective as the paid gate for detecting severe truncation-style regressions, correlating 0.87 with reward versus 0.80 for the full gate on the degradation set. But its scope is limited by the natural failure distribution. Of 1,485 grid failures, 96.5 percent terminate normally as semantic failures, and only 3.5 percent truncate. The bit's recall is therefore 1.0 on truncation but only 0.035 overall. On the six-provider grid it collapses to correlation 0.51 versus the gate's 0.94. The correct conclusion is not that the completion bit replaces the judge; it is that different signals serve different operating regimes.

The negative results further sharpen the guidance. The paper tests natural repair attempts and finds they do not transfer out of sample. A human-anchored persona reweighting lifts in-sample validity from rho 0.71 to 0.80, but leave-one-stratum-out drops it to 0.68. Combining four judges gives no lift over the best single judge and is worse under leave-one-agent-out. Across 21 candidate signals, averaging all four judges reproduces the best single judge's 31 percent close-pair error exactly, 27 of 87 pairs, because the judges are highly correlated and flip the same close pairs. Only calibrated abstention helps: when the gate's score gap falls inside its bootstrap sampling noise, the protocol declines to rank the pair, and on the pairs it does rank, close-pair error falls to 14.8 percent. This supports the calibrate-then-trust recipe: learn the trusted region, then operate inside it, rather than assume a static gate can be fixed once and trusted everywhere.

Evidence quality should be judged by the triangulation rather than by any single experiment. External validity comes from the broad six-provider ladder and independent judge and simulator checks. Internal validity comes from the flag-only degradation set, where the broken-versus-working ordering is known by construction. Human validity comes from the blind panel and the second substrate. The limitation is not a lack of controls, but the boundaries of the audited substrates. The human panel is modest, three annotators rating 150 transcripts, though leave-one-annotator-out keeps the headline inversion within 56.4 to 60.5 percent. The degradation set captures truncation-style failures, not every semantic failure mode. tau2 instructions are synthetic, and the harness runs on Bedrock with a tolerant parser. The paper argues the parser is lossless for verdicts and preserves deterministic database and action checks, and the gap also appears on the fully deterministic airline substrate, but generalization to live customer interactions remains untested.

Deep insights and conclusion

GAUGE's central lesson is that a release gate can be locally reasonable and globally misleading. It can rank broad capability differences well, track human satisfaction when humans are asked to rate satisfaction, and still fail to predict whether the customer's task was completed. The formal separation of ranking validity and construct validity is the work's durable contribution. It exposes a class of evaluation failures that is easy to hide: a high Spearman correlation across a wide agent pool does not imply the instrument can make the near-peer decisions for which it was built.

The engineering implication is a change in cadence, not necessarily a change in metric. A team should run a periodic verifiable audit to learn where the cheap gate is trustworthy, then use the cheap gate in continuous integration within that calibrated region. The zero-cost completion bit is a tripwire for configuration regressions that truncate conversations. The paid judge remains useful for coarse ranking and semantic checks. Satisfaction should be retained as an experience signal, but it should not be the sole release anchor, especially once it becomes an optimization target for prompts, policies, models, or tools. When judge and candidate share a provider, thresholds should be calibrated against an out-of-family judge because the measured plus 0.75 self-inflation can flip an accept-reject decision even if the broad ranking remains stable.

The limitations are concrete and testable. The audit is grounded in synthetic benchmark substrates with deterministic or human-graded correctness checks, not in noisy production logs. The close-pair threshold depends on the oracle's effective resolution, and the paper itself links part of the difficulty to agent near-equality bounded by oracle resolution. The degradation set covers budget-style failures such as token caps, step caps, and error aborts, so it does not establish that the completion bit will catch every regression. The paper also does not report live online validation or production A/B outcomes, which means the practical cadence is well supported as an offline protocol but still needs field calibration.

Future work should extend GAUGE from binary verifiable rewards to partially observable, real-world outcomes: refund completion, retention, escalation avoidance, and post-conversation satisfaction under production logging. Another direction is formal decision policies around abstention, because calibrated abstention halves error on the pairs it ranks but buys selectivity rather than resolution. The more interesting question is not whether an LLM judge can be made human-like, but whether the human-like signal is the right target at all. GAUGE answers that with data: for task-oriented agents, a human-validated satisfaction gate is still a mis-anchored gate unless it is calibrated against what the customer actually needed to get done.