2D-RoPE: Why Frontier LLMs Fail at Simple Copying and How to Fix It

TL;DR

Even the most advanced AI models, including GPT-5.5 and Gemini 3.1 Pro, fail at a task a toddler could do: exactly copying a long string. The culprit is the 1D nature of standard positional encodings, which confuses the model when patterns repeat. This paper introduces 2D-RoPE, a revolutionary way to encode positions by treating text as a 2D grid (rows and columns), enabling models to generalize their copying ability to lengths thousands of times longer than seen during training.

The "Copying" Crisis in Frontier Models

We expect LLMs to solve Olympiad-level math, yet they stumble when asked to format a simple Python list or repeat a binary string. The authors found that as soon as a string contains repeated substrings (like periodic sensor data), LLM accuracy plummets.

As shown in the following chart, even "Pro" models drop below 50% accuracy on relatively short sequences:

Why do they fail?

Standard Transformers use RoPE (Rotary Positional Embedding) in 1D. This creates a "shortcut" where the model looks for a similar local context (e.g., "what came after '011' last time?") rather than tracking the absolute position. When '011' appears 50 times in a sequence, the model gets "lost" in the repetitions.

The Solution: Thinking in 2D

The core insight of this paper is that copying is naturally a 2D task. Imagine writing the input on one line and your copy on the line below. Suddenly, the token you need to copy is always directly above your current position, regardless of how long the line is.

2D-RoPE Architecture

Instead of one number for position, 2D-RoPE gives every token two:

1. Row ID: Increments every time a (newline) appears.
2. Column ID: Resets at every newline.

By splitting the attention head's dimensions—half for the row and half for the column—the model gains a permanent "visual" alignment. Copying becomes a simple operation: "Look at the previous row, same column."

Experimental Breakthroughs

The results are staggering. While standard RoPE models fail immediately when test sequences are longer than training ones, 2D-RoPE models show near-perfect length generalization.

Key Findings:

• 1000x Generalization: A model trained on length-100 sequences could perfectly copy length-100,000 sequences.
• Pretraining Success: On the DCLM 1.4B scale, 2D-RoPE achieved 92.6% accuracy on complex 4K copy tasks, whereas standard RoPE hit 0%.
• No Reasoning Penalty: The model's performance on common-sense reasoning (MMLU, ARC) remained stable, proving that 2D-RoPE doesn't damage the model's "intelligence."

Adaptive Layouts: Auto-2D-RoPE

What if there are no newlines? The authors developed Auto-2D-RoPE, which uses a learnable affine transformation to discover the 2D structure of text. It decides dynamically when to "start a new row" based on the data, making the 2D benefit available for all types of text formats.

Critical Perspective: The Future of Positional Bias

This work highlights a fundamental flaw in the "Attention is All You Need" dogma: Inductive Bias matters. By simply changing how we represent the "where," we drastically change the model's ability to learn the "what."

Limitations

• Separator Reliance: The basic 2D-RoPE relies heavily on newline tokens. While Auto-2D-RoPE mitigates this, scaling it to 70B+ parameter models remains a future engineering challenge.
• Complexity: Splitting dimensions for 2D rotations adds a layer of implementation complexity for optimized kernels like FlashAttention.

Conclusion

By viewing text in 2D, we bridge the gap between simple associative recall and robust algorithmic copying. This research paves the way for LLMs that can reliably handle massive data logs, code structures, and repetitive formatting tasks without breaking—a prerequisite for the next generation of AI agents.