ReSA steals the reward, not the prompt
Existing red-teaming attacks split into two families. Prompt manipulation attacks craft adversarial prompts, but need intensive query-level optimization or expensive offline training to find universal suffixes. Contrastive decoding attacks modify token logits using contrasts between strictly matched safe and unsafe model pairs, so they require identical tokenizers and vocabularies. The paper frames both as limited by either high computational cost or strict model pairing.
ReSA targets a different object: the latent safety reward the authors argue underlies alignment. The claim is that RLHF and RLVR reshape the LLM's output distribution in a way that can be viewed as regulated by an underlying reward, never explicitly exposed but consistently governing behavior across prompts and models. If that reward can be recovered, it becomes a universal adversarial signal that needs neither per-prompt optimization nor a matched model pair.
Reward extraction treats the aligned model as an expert
Stage one frames reward recovery as an inverse reinforcement learning problem. The aligned policy is treated as the expert, and trajectories are collected by sampling its responses to harmful prompts. The prompt distribution is explicitly restricted to harmful prompts so that optimization isolates the conditional distribution within the safety domain.
Because many reward models can explain identical expert trajectories, the recovery problem is ambiguous. The paper adopts the Principle of Maximum Entropy, selecting the highest-entropy trajectory distribution among those matching expert feature expectations. Proposition 1 states this constrained problem has a unique closed-form Boltzmann solution, letting the proxy reward be written as a linear function of trajectory features with a frozen feature extractor and learnable parameters.
Optimization uses the standard feature-matching gradient, pushing the model's feature expectations toward the expert's. Since the model expectation is intractable, a pre-trained generative LLM acts as a tractable sampler, trained with GRPO to maximize the proxy reward under a KL penalty to a frozen reference policy. The paper describes this alternating update as a GAN-like dynamic, with the reward model as discriminator and the actor as generator.
Reversing the reward turns safety into an attack prior
Stage two derives an adversarial policy that minimizes the proxy safety reward while staying close to the safe policy to preserve generation quality. Proposition 2 gives the optimal adversarial policy as an exponentially tilted distribution over the safe policy, with a scaling factor controlling the trade-off between adversarial strength and generative quality.
Direct sequence-level sampling is prohibitive because of the global partition function, so the paper uses an autoregressive approximation that applies negative reward guidance at each decoding step. A constant strength would risk degrading coherence on neutral tokens, so the method introduces an adaptive strength based on the marginal reward, the gap between a candidate token's reward and the context reward. For neutral tokens the adversarial strength vanishes; for tokens with large safety implications the full strength engages.
Active exploration beats passive linear probing
To justify the IRL framework, the paper compares against a supervised linear probing baseline trained via continuous trajectory resampling from a frozen actor. On Llama3.1-8B-Instruct, linear probing reaches 74.8 ASR on AdvBench and 76.0 on HarmBench, while ReSA reaches 85.2 and 86.0. On Qwen2.5-14B-Instruct the gap widens: 25.7 versus 48.5 on AdvBench, and 38.0 versus 60.5 on HarmBench.
The authors attribute the gap to how each method explores. Passive sampling fails to challenge the probe's blind spots, yielding a shallow decision boundary vulnerable to reward hacking during decoding. ReSA's iteratively updated actor acts as an automated hard negative miner, forcing the reward model to penalize borderline trajectories and tighten its safety boundary. This is the paper's explanation, not a directly measured mechanism.
Extraction survives without a fully unaligned actor
The framework does not strictly require an unaligned actor. In an Unalign-Free ReSA setting, an aligned model serves as the actor against a stricter expert policy, and the relative safety gap between the two provides enough signal to isolate genuine safety behaviors. On Llama3.1-8B-Instruct, Unalign-Free ReSA reaches 83.2 ASR on AdvBench and 86.5 on HarmBench, close to standard ReSA's 85.2 and 86.0.
On Qwen2.5-14B-Instruct the pattern differs: Unalign-Free ReSA reaches 56.2 ASR on AdvBench, above standard ReSA's 48.5, but 51.5 on HarmBench, below ReSA's 60.5. The paper concludes cross-family extraction is viable and highly effective without fully unaligned counterparts, though the two benchmarks do not move together on this target.
Attack strength and alignment method shift the trade-off
The scaling factor governs the trade-off between attack effectiveness and linguistic coherence. On Llama3.1-8B-Instruct, raising it from 0.5 to 2.0 lifts the harmful score on AdvBench from 1.08 to 3.79 while perplexity rises from 17.11 to 21.52. The paper reports that around 1.5 sits on the optimal Pareto frontier, achieving high harmful score while maintaining perplexity levels.
Robustness to alignment method is mixed. ReSA recovers the latent reward from both DPO- and RLVR-aligned LLMs despite structural differences. The DPO-recovered reward drives potent harmful generation but incurs higher perplexity costs, which the authors link to a fragmented probability landscape. The RLVR-recovered reward shows attenuated effectiveness, which the authors attribute to a sparser signal from discrete constraint-based alignment that is harder to reconstruct via maximum entropy IRL.
Removing the adaptive strength weakens attack effectiveness without improving coherence. The paper describes the adaptive term as a filter that suppresses adversarial pressure on neutral tokens while focusing the attack budget on safety-critical decision boundaries.
The recovered reward separates safe from harmful trajectories
The paper validates that ReSA reconstructs the latent safety reward rather than memorizing surface patterns. As training proceeds, the gradient norm declines in parallel with the actor LLM's harmful score, which the authors read as the actor progressively mastering expert behavior through proxy reward updates.
A safety-landscape visualization shows that from step 0 to 200 the region of suppressed attack success rate widens and deepens, so the final model maintains low ASR despite parameter noise. The density plot of the recovered reward is bimodal, assigning consistently high scores to safe expert trajectories and low scores to harmful ones with a substantial decision margin. The authors treat this separability as evidence the proxy reward provides a precise guidance signal when reversed.
The transferable claim rests on one recovered reward
The paper's central priority is transferability: a single recovered reward generalizes across prompts and diverse models, which is what distinguishes ReSA from per-prompt optimization and from matched-pair contrastive decoding. The comparison that matters most is therefore not any single ASR number but whether one extraction transfers, and the reported cross-model and cross-family results are the evidence for that.
Two boundaries qualify that claim. First, the reward is recovered from a specific expert policy, and the paper's own sensitivity results show that the alignment method behind that policy changes both effectiveness and perplexity cost, with RLVR-derived rewards notably weaker. Second, the adaptive strength is not optional: the ablation shows constant strength weakens effectiveness without buying coherence. For readers weighing this attack, the practical question is which alignment method produced the target model and how much adversarial strength the coherence budget allows, since those two variables move the result more than the choice of benchmark.
Sources used in this answer
Reward Stealing Attack on Large Language Models
ReSA recovers a proxy safety reward from an aligned LLM via maximum entropy IRL and reverses it during decoding, reporting higher attack success and cross-model transferability than prompt manipulation or contrastive decoding baselines.