What the audit measures and why near-zero is misleading
Fine-tuning-based prompt injection defenses train a model to follow the user's instruction and ignore instructions embedded in external data. Because these defenses lack formal guarantees, they are judged empirically: developers run the strongest realistic attack available at release and report how often it succeeds. The paper argues this practice can overstate robustness, since a defense that resists today's attack may still be brittle against a stronger search.
The central quantity is attack success rate (ASR), the share of evaluated samples on which an adversarial suffix makes the model produce the attacker's target output. The authors audit StruQ and SecAlign, two fine-tuning-based defenses, on Llama-3-8B-Instruct, Mistral-7B-Instruct and Qwen2-1.5B-Instruct. Prompt-specific attacks are scored over 50 randomly selected AlpacaFarm samples, with 95% Wilson score intervals reported because several conditions sit at or near 0% or 100% ASR.
Checkpoints act as stepping stones for the attack
The method, Checkpoint-GCG, exploits an artifact of training itself. Fine-tuning produces a sequence of intermediate checkpoints on the way to the final defended model. The attacker runs GCG, a gradient-based search that appends a suffix to the prompt to elicit a target string, at each checkpoint in turn. The suffix that works at one checkpoint initializes the search at the next, so optimization climbs a ladder toward the final model instead of starting cold against it.
The authors also adapt a universal suffix, a single suffix optimized to attack several training prompts at once, and carry it forward across checkpoints. They report that selecting checkpoints by gradient norm gives the best trade-off between attack effectiveness and computational cost. The same framework is instantiated with BEAST, a different optimization-based attack, which reaches 80% ASR on 20 AlpacaFarm samples against SecAlign-defended Llama-3-8B-Instruct where BEAST against the final model alone reaches 0%.
Standard GCG plateaus where Checkpoint-GCG keeps climbing
Against undefended models, all methods saturate: standard GCG, standard GCG given the same total step budget as Checkpoint-GCG, and Checkpoint-GCG all reach 98–100% ASR. The difference appears once a defense is applied. Standard GCG with 500 steps falls to 6% ASR against SecAlign-defended Llama-3-8B-Instruct, and giving it the Checkpoint-GCG step budget only raises that to 12%. Checkpoint-GCG reaches 88% on the same setting.
The pattern holds across models and defenses. Against StruQ, standard GCG ranges from 26% to 88% depending on the model, while Checkpoint-GCG reaches 90–100%. Against SecAlign, Checkpoint-GCG reaches 84–96% across the three models. On 20 SEP samples with distinct injected instructions and target phrases, Checkpoint-GCG reaches 80% ASR against SecAlign-defended Llama-3-8B-Instruct while standard GCG with the Checkpoint-GCG budget reaches 15%. The authors read this as evidence that evaluating only the final model with a known attack can give a false sense of security.
Universal suffixes generalize beyond the training prompts
A universal suffix is optimized on 10 AlpacaFarm training samples and then tested unchanged on 198 held-out AlpacaFarm samples and on 500 SEP samples. Against SecAlign-defended Llama-3-8B-Instruct, Checkpoint-GCG reaches success@1 of 75.3% in-distribution, meaning a single attempt succeeds on about three quarters of held-out samples, while standard GCG reaches 0% with either 500 steps or the Checkpoint-GCG budget. With five attempts, Checkpoint-GCG approaches near-perfect in-distribution success.
Out-of-distribution performance drops, as expected when an attack must generalize to unseen inputs, but the ranking holds: Checkpoint-GCG reaches 78.3% on SEP against SecAlign-defended Llama, 88.9% against StruQ-defended Llama, and 87.9–99.0% against StruQ-defended Mistral and Qwen. The authors also ablate suffix length against SecAlign-defended Llama: training ASR stays at 100% for 20, 25 and 30 tokens, while held-out ASR falls from 75.3% to 64.1% and 45.5%, so longer suffixes fit the training prompts more tightly without generalizing better.
Weaker attackers can reconstruct comparable stepping stones
Intermediate checkpoints are not available to real attackers, so the paper tests whether the exposed weaknesses survive weaker threat models. Given an open-weights base model and its defended counterpart, an attacker can linearly interpolate between the two sets of weights to build a synthetic trajectory. Interpolating between Llama-3-8B-Instruct and its SecAlign-defended version yields 51 intermediate states; running Checkpoint-GCG over them reaches 90% ASR on 20 samples, with a 95% Wilson interval of 69.9–97.2%, at an average of 1,205 total steps, comparable to using actual fine-tuning checkpoints.
Transfer is the second route. A universal suffix found on a checkpointed surrogate transfers to a target with a related base model and defense. Using SecAlign-defended Llama-3-8B-Instruct as surrogate, the suffix reaches 63.9% ASR on Meta-SecAlign-8B in a black-box setting and 78.3% on 198 held-out samples after only 5 GCG steps in a white-box setting, where standard GCG transfers at 0% even after 5,000 steps. Transfer across substantially different model families is limited: Mistral and Qwen surrogates yield 0%.
Where the audit applies and where it stops
The evidence covers two fine-tuning-based prompt injection defenses, StruQ and SecAlign, on three instruction-tuned models, plus a jailbreak defense replication mentioned in the appendix. The authors note that they reused released code and hyperparameters, changing GPU counts and precision to fit their compute, and that a dedicated hyperparameter search could strengthen the defenses but was outside the paper's scope. The reported ASRs therefore describe these replicated defenses, not every possible configuration of them.
The threat model matters for reading the headline number. The 96% figure comes from an auditor who can attack intermediate checkpoints, an advantage real attackers do not have. The interpolation and transfer experiments are what connect that strong audit to realistic adversaries, and they show substantial but lower success: 90% via interpolation on 20 samples, and 63.9% black-box transfer within a related model family. Cross-family transfer at 0% marks the boundary of that claim.
Audit before release rather than trust the best attack of the day
The paper's practical priority follows from the gap between the two evaluation regimes. A defense that scores near zero against standard GCG on the final model can score 84–96% against Checkpoint-GCG, and the checkpoints needed for that audit already exist during training. The authors argue developers should therefore stress-test defenses with strong adversaries before release, shortening the feedback loop rather than waiting for the next state-of-the-art attack to appear.
The most decision-relevant comparison is not the absolute 96% but the contrast between what a final-model evaluation reports and what a checkpoint-based audit reveals on the same defense and model. For SecAlign-defended Llama-3-8B-Instruct, that contrast runs from 6% with standard GCG at 500 steps to 88% with Checkpoint-GCG. The paper also reports that suffixes optimized on intermediate checkpoints progressively shift the final model's attention from the user instruction toward the injected content, which is the mechanism the authors offer for why the stepping-stone search works.
Sources used in this answer
Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defenses
Using intermediate fine-tuning checkpoints as stepping stones for GCG and BEAST raises prompt injection attack success rates against state-of-the-art defenses from near zero to as high as 96%, and related adversaries can exploit comparable weaknesses via weight interpolation and transfer.