You Can’t Learn What’s Not There: Why BERT and RoBERTa Struggle with Baker's Paradox

TL;DR

Despite the massive scale of modern self-supervised learning, certain "hidden" rules of language remain unlearnable from text alone. This paper puts BERT and RoBERTa to the test using the Locative Alternation—a classic linguistic puzzle. The verdict? These models rely on shallow statistical correlations (like whether a verb is near the word "water") rather than the deep semantic structures required for true linguistic fluency.

The Ghost in the Machine: The Poverty of the Stimulus

For decades, linguists led by Noam Chomsky have argued for the Poverty of the Stimulus (POS): the idea that children learn things about language that simply aren't present in the data they hear.

A prime example is Baker’s Paradox. Consider these sentences:

1. Hal loaded hay into the wagon. (Grammatical)
2. Hal loaded the wagon with hay. (Grammatical)
3. Amy poured water into the glass. (Grammatical)
4. *Amy poured the glass with water. (Ungrammatical)

Why is "load" allowed to alternate but "pour" is not? There is no "negative evidence" in a gold-standard text corpus telling a model that sentence #4 is wrong. If a model only learns from what is there, can it ever truly know what isn't allowed?

Methodology: Testing the "Blind Spots"

The researchers focused on the Locative Alternation, creating a dataset that distinguishes between:

• Alternating verbs: e.g., load, spray.
• With-only verbs: e.g., coat, cover.
• Into/Onto-only verbs: e.g., pour, drip.

They used BERT and RoBERTa, fine-tuned on the CoLA (Corpus of Linguistic Acceptability) task, to see if these models would flag ungrammatical alternations as "unacceptable."

The Illusion of Success

While RoBERTa showed some ability to identify ungrammatical "Into/Onto" sentences (like the "pour" example above), the researchers suspected a "clever Hans" effect.

By analyzing the Contextualized Embeddings using PCA, they found that the models clustered verbs based on superficial features—specifically, whether the verb typically interacts with liquids.

As shown in the embedding space above, verbs like pour, dribble, and slosh cluster together because they share the "liquid" context. However, the models failed to distinguish between alternating and non-alternating verbs within those clusters. To a Transformer, "pour" and "spray" look semantically identical because they both involve moving liquids, even though their "linking rules" to syntax are fundamentally different.

Performance Breakdown

The quantitative results were sobering for those hoping that "more data" solves everything:

• BERT's correlation with human judgment on "Into/Onto" violations was a abysmal 0.05.
• RoBERTa performed better (0.40), but largely due to the aforementioned liquid-based heuristics.

Critical Analysis & Conclusion

The takeaway is a powerful warning for AI researchers: Distributional Semantics (learning by context) is not the same as Conceptual Semantics.

The Limitation

The models lack a Grammatically Relevant Subsystem (GRS). In human cognition, we understand that "pouring" focuses on the motion of the mass (gravity-fed), while "filling" focuses on the change of state of the container. Transformers, lacking a world model of physics or intent, cannot "see" these constraints because they are rarely explained in raw text.

Future Outlook

If we want models that truly understand language, we may need to move beyond pure self-supervision on text. Potential paths include:

• Multi-modal grounding: Learning how verbs work by watching video or simulating physics.
• Inductive Biases: Incorporating structural linguistic constraints directly into the architecture.

Until then, LLMs will continue to have "blind spots" in their understanding of the very foundations of human grammar.