SEMANTICQA: Why LLMs Still Trip Over the "Pain in the Neck" of Phrasal Semantics

TL;DR

Despite their prowess in solving math problems and writing code, Large Language Models (LLMs) still struggle with the subtle nuances of Multiword Expressions (MWEs)—phrases like "kick the bucket" or "baby oil" where the whole is different from the sum of its parts. This paper introduces SEMANTICQA, a diagnostic benchmark that proves models often "hallucinate" semantic understanding: they can paraphrase an idiom fluently but fail to identify where it starts and ends in a sentence.

Problem: The Hidden Fragility of Phrasal Semantics

For decades, MWEs have been called a "pain in the neck" for NLP. The challenge lies in compositionality:

• Idioms: "Silver lining" isn't about precious metals.
• Noun Compounds: "Baby oil" is for babies, but "olive oil" is made from olives.

Current benchmarks largely ignore these sub-sentential units or test them in isolation. If a model passes a multiple-choice test on idioms, does it actually understand the phrase, or did it just memorize a common pattern? The authors argue that we need to disentangle these operations to see the truth.

Methodology: Operation-Aligned Evaluation

The core innovation of SEMANTICQA is its Operation-Aligned structure. Instead of one big score, it breaks down phrasal processing into three specific "Atomic Operations":

1. Extraction: Can the model find the phrase in a sentence? (Structural Identification)
2. Classification: Can the model categorize the relationship (e.g., is "heavy rain" an intensifier?)? (Semantic Categorization)
3. Interpretation: Can the model explain what it means in context? (Generative Reconstruction)

Figure 1: The SEMANTICQA framework aligning lexical phenomena with atomic task operations.

By testing the same phrase across these three dimensions, the authors can measure cross-operation consistency. If a model can interpret a phrase but can't extract it, its "understanding" is computationally brittle.

Experiments & Results: The "Fluent but Blind" Phenomenon

The authors evaluated a "Who's Who" of AI, including GPT-5, OpenAI o3, DeepSeek-R1, and Claude-Sonnet-4.5.

1. The Bottleneck of Extraction

The results show a startling gap. Models often achieve high BERTScore or METEOR values in interpretation (meaning they talk a good game), but their Exact Match (ACCs) in extraction is surprisingly low. Essentially, models can explain a phrase they can't even "see" properly in the text.

2. The Failure of Sequential Reasoning

In "Sequential Task Compositions" (e.g., Extract → Interpret), performance tanks. If a model makes a tiny error in pinpointing a phrase, the downstream interpretation usually fails. This highlights that LLMs lack workflow robustness in linguistics.

Table 1: Performance variation across Different LLMs and Tasks. Notice the disparity between detection (IED) and extraction (IEE).

3. The Power of the "Oracle Schema"

One bright spot: providing models with an Oracle Schema—explicit definitions and types of expressions—significantly boosts performance. This suggests that models have the underlying capacity but lack the "instructional grounding" to trigger the right semantic knowledge.

Critical Analysis: What This Means for the Future

The takeaway is sobering: high fluency does not equal high comprehension.

SEMANTICQA reveals that our current evaluation methods might be overestimating LLMs. By using flexible metrics like BERTScore, we reward models for being "close enough" in meaning, while ignoring their failure to grasp the strict linguistic structures (the "syntax-semantics interface") that humans use naturally.

Key Insights:

• Metric Sensitivity: Similarity-based metrics (BERTScore) mask structural errors.
• ICL Limits: In-context learning helps with "how to talk" (interpretation) but struggles with "what to pick" (extraction) as the number of categories scales.
• The Grounding Gap: There is a persistent divide between a model's ability to generate text and its ability to ground that text in specific linguistic units.

Conclusion

SEMANTICQA isn't just another leaderboard; it's a microscope. It shows that for LLMs to truly master human language, they must move beyond surface-level statistics and achieve structurally grounded semantic reasoning. Until then, MWEs will remain the "pain in the neck" that prevents AI from reaching human-level linguistic precision.

Figure 2: Scaling effects of category size on classification accuracy—LLMs struggle as the semantic granularity increases.