Payload understanding is split into four measurable layers
The paper argues that earlier work treats payload analysis as a single-layer classification task, which cannot show how deeply a model understands a payload. PayloadSemBench instead operationalizes understanding as four tasks ordered from relatively direct to more interpretive: L1 type identification, L2 intent understanding, L3 obfuscation assessment, and L4 severity assessment.
Each layer has its own metric and denominator, so they are not interchangeable. L1 is a six-class task (SQLi, XSS, CMD, PT, SSRF, or benign) scored by accuracy and Macro-F1 over n=232 after excluding 8 entries in the C3 negative pool. L2 is a natural-language explanation scored by a judge model on a three-dimensional rubric: intent accuracy weighted 0.5, reasoning completeness 0.3, and context sensitivity 0.2, for a maximum of 5. L3 asks whether a payload is obfuscated and which technique was used, with detection and technique-identification rates computed over 92 obfuscated attack payloads and the base-payload false-positive rate over 100 base attack payloads. L4 assigns a five-level severity rating, measured by mean absolute error, exact-match rate, and linearly weighted Kappa across all n=240 payloads.
The benchmark mixes real payloads, generated variants, and benign requests
PayloadSemBench contains 240 payloads in Experiment I: 100 base attack payloads drawn from the public PayloadsAllTheThings and SecLists repositories and manually screened for testbed compatibility, 100 obfuscated variants produced by two LLM generators, and 40 manually constructed benign requests covering 23 business scenarios rather than collected from production traffic. The split is 140 base and 100 obfuscated payloads, distributed as 40 SQLi, 40 XSS, 37 CMD, 38 PT, 37 SSRF, and 40 benign.
Every payload went through testbed validation on DVWA and WebGoat and received one of five labels: EFFECTIVE (151), MECHANISM_VERIFIED (23), INEFFECTIVE (9), UNVERIFIABLE (17), or BENIGN_OK (40). The paper is explicit that not all payloads were functionally validated: it retains and discloses both the UNVERIFIABLE entries, whose effects could not be determined in the current environment, and the INEFFECTIVE entries that failed validation. Ground truth was set by consensus among three security-domain annotators after two rounds of anchor calibration and then re-verified by a fourth independent expert.
Type identification and intent understanding hold up, severity does not
Across the four models, L1 accuracy ranged from 0.931 to 0.948 and Macro-F1 from 0.929 to 0.947. DeepSeek V4 Pro and GLM 5.2 tied for the highest accuracy at 0.948; Kimi K3 was slightly lower at 0.931. Mean L2 intent-understanding scores ranged from 4.574 to 4.749 out of 5.
Severity assessment was the principal weakness. After normalizing L1, L2, and L4 onto a [0,1] interval, the L1−L2 gap was only −0.005 to 0.035, while the L2−L4 gap was 0.388 to 0.573. The paired L2−L4 difference was significant for all four models (p<0.001), and under paired bootstrap the confidence intervals excluded 0. The paper's conclusion C1 is therefore that the four-layer capability profile is non-uniform rather than monotonically declining: models can interpret payloads and describe intent accurately, but their severity ratings are substantially less reliable.
Obfuscation degrades fine-grained parsing, not every layer equally
Obfuscation did not act as a uniform bottleneck. L1 accuracy on obfuscated payloads (0.967–0.978) actually exceeded accuracy on base payloads (0.907–0.929). Human review indicated that the encoding and nested structures introduced into the obfuscated variants made the category-discriminating signals more salient in this benchmark, so obfuscation was not a bottleneck for type identification.
The damage appeared at finer granularity. L3 obfuscation detection rates were high, but complete identification of the specific technique was much lower: on the main dataset, detection was 0.989 for all four models while complete technique identification ranged from 0.489 to 0.587. The paper's conclusion C2 is that obfuscation challenges fine-grained semantic parsing rather than degrading all four layers synchronously, with intent explanation and reconstruction of specific obfuscation techniques most susceptible.
Correct understanding does not guarantee a correct detection decision
Experiment II mapped semantic understanding onto practical detection. It evaluated four LLMs and four conventional security tools on the DVWA and WebGoat testbeds over 249 cases: 174 attacks, including 117 exploitable cases with GT=positive and 57 attack-like but non-exploitable cases with GT=negative, plus 75 benign cases. The attack detection rate used the 117 positive cases as its denominator and the false-positive rate used the 75 benign cases; the 57 negative attack-like cases were reported separately and excluded from detection-performance statistics.
Of the 8 deduplicated missed-detection cases, 4 triggered no semantic-failure rule for any model: the reruns showed correct L1 type decisions and L2 total scores of 4.0–5.0, yet the models still classified these cases as not exploitable during detection. The attributable failures came mainly from insufficient identification of obfuscation techniques (rule R1, 4 model-case instances) and insufficient intent understanding (rule R3, 5 model-case instances). Across the four models, only 12.5% to 50% of missed detections could be attributed to these semantic-understanding failures, supporting conclusion C3 that semantic understanding and detection decisions are partially decoupled.
An independent WAF-alert dataset reproduces the same profile
To test generalization, the authors built a 180-record external validation set: 100 base attack payloads from production WAF alerts, 50 variants (18 native WAF-log records and 32 systematically constructed records), and 30 benign requests from crawler traffic collected across 11 real-world projects. No non-empty payload string appeared in both the validation and main datasets. The type distribution was 47 SQLi, 41 XSS, 35 CMD, 5 PT, and 22 SSRF records plus 30 benign requests.
The validation set reproduced the non-uniform four-layer structure. L1 accuracy ranged from 0.939 to 0.967 and mean L2 scores from 4.070 to 4.383, both still high, while the L2−L4 mean gap ranged from 0.127 to 0.418 and was positive for all four models, with paired differences significant for DeepSeek, Kimi, and Qwen. The layer-specific obfuscation pattern also held: L3 detection rates reached 0.897–0.928, whereas complete technique identification was only 0.423–0.485. The paper notes that one GLM rep1 judge result could not be parsed, so its rep1 L2 mean and two layer-gap measures were computed with n=179.
The evidence covers general web payloads, not vision endpoints
The study evaluates model behavior using general web payloads and the DVWA/WebGoat testbeds rather than samples collected specifically from computer vision service endpoints, even though the motivation is the security of vision services exposed through web interfaces and APIs. L2 relies on scores assigned by a judge model; human review and cross-validation with a second judge support reliability, but judge leniency and evaluator–subject overlap may still affect absolute scores. The validation set contains only 5 PT records, and some request-structure information was omitted from the payload-only prompt, which limits interpretation of type-specific differences and structural effects.
Watch severity ratings and technique reconstruction, not type labels
The most decision-relevant split in this paper is not between models but between layers. Type identification and intent explanation were consistently strong across both datasets, so a high L1 or L2 score carries little information about whether a model can be trusted on the harder tasks. The two places where the evidence shows real weakness are L4 severity assessment, where the L2−L4 gap stayed positive for all four models on both datasets, and L3 complete technique identification, where rates fell to 0.489–0.587 on the main dataset and 0.423–0.485 on validation even as obfuscation detection stayed near 0.9.
The second split is between understanding and acting. Because only 12.5% to 50% of missed detections were attributable to semantic-understanding failures, and 4 of 8 deduplicated missed cases showed correct type decisions and L2 scores of 4.0–5.0 yet were still judged not exploitable, improving a model's payload explanation is not by itself a reliable route to better detection decisions. Readers weighing these models for payload triage should treat severity calibration and obfuscation-technique reconstruction as the priority checks, and should not assume that strong intent explanations transfer into correct exploitability judgments.
Sources used in this answer
An Evaluation of the Semantic Understanding Capabilities of Large Language Models for Web Attack Payloads
Across a 240-payload benchmark and an independent 180-record validation set, four LLMs identified payload types and explained attack intent strongly but rated severity far less reliably, obfuscation degraded fine-grained technique reconstruction rather than all layers, and only 12.5% to 50% of missed detections were attributable to semantic-understanding failures.