Why can't long-context models just use more memory?
The biggest practical barrier to long-context LLMs is that the key-value (KV) cache—the intermediate data needed for attention—grows linearly with context length and quickly exhausts GPU memory. For instance, PQCache uses product quantization to compress the KV cache, achieving a 4.6% score improvement on InfiniteBench over existing methods, but still requires careful management to avoid latency spikes [2]. Hardware approaches like BlockPIM and LoL-PIM move computation into memory (Processing-in-Memory) to bypass bandwidth limits, with BlockPIM boosting throughput by 62% over prior PIM solutions [3], and LoL-PIM achieving up to 16x speedup over GPU-PIM systems [4]. However, these hardware solutions are not yet widely deployed, and even the best of them (e.g., InstAttention, which offloads attention to computational storage drives) only improves throughput 11.1x over SSD-based baselines for a 13B model—still far from real-time long-context inference [6].
The evidence here converges on a clear point: memory bandwidth, not compute, is the primary bottleneck. All four hardware-focused papers [2][3][4][6] identify the KV cache as the limiting factor, and each proposes a different workaround (quantization, PIM, offloading to storage). None claims to fully solve the problem—they each trade off between model quality, latency, and scalability. For example, PQCache maintains model quality but adds overhead [2], while LoL-PIM requires a custom compiler and multi-node setup [4]. This suggests that no single solution is yet ready for production at very long contexts (100K+ tokens).
Do current benchmarks actually measure long-context understanding?
No—existing benchmarks often conflate context length with task difficulty, and models that perform well on synthetic tasks fail on realistic long-context reasoning. The LongICLBench benchmark [7] tests models on extreme-label classification with up to 174 classes and 50K-token inputs, and finds that even top models like Gemini struggle: they show a bias toward labels presented later in the sequence and cannot reason over multiple pieces of information. Similarly, Ada-LEval [8] introduces length-adaptable tests up to 128K tokens and reports that both closed-source and open-source models 'demonstrate limitations... especially in ultra-long-context settings.' These two studies [7][8] are the only ones here that directly evaluate long-context reasoning, and they agree that current models are poor at it.
The gap is not just about memory—it's about reasoning. For example, the 'lost-in-the-middle' problem, where models ignore information in the middle of long inputs, is documented in medical QA [1] and persists even with retrieval-augmented generation (RAG). BriefContext [1] mitigates this by using a map-reduce strategy, but it doesn't eliminate the issue. Meanwhile, the most cited paper here [9] (569 citations) shows that models struggle to learn long-tail knowledge—facts that appear rarely in training data—and that retrieval augmentation helps but doesn't fully close the gap. This suggests that even if memory were infinite, models would still fail on tasks requiring integration of many facts across long contexts.
Do models actually learn long-range context, or just local patterns?
A subtle but critical evidence gap is that some models, especially those trained with overlapping tokens (like DNABERT), may not learn long-range context at all—they rely on local token identity instead. A study on DNA language models [10] found that DNABERT, trained with overlapping k-mers, 'struggles to learn larger sequence context' and that its embeddings 'largely represent token sequence.' This means that for tasks requiring long-range dependencies (e.g., predicting regulatory elements far from a gene), the model may be memorizing local patterns rather than understanding global structure. The same paper notes that good performance on genome-biology tasks can still be achieved, but only when 'larger sequence context is of less relevance.'
This finding is echoed in the long-context DNA model PlantCAD2 [5], which uses an 8,192bp context window and improves accessible chromatin prediction in maize from AUPRC 0.587 to 0.711—a 21% relative improvement—by leveraging long-range context. The contrast between DNABERT's failure and PlantCAD2's success suggests that architecture and training matter enormously: PlantCAD2 is a long-context model trained on 65 genomes, while DNABERT uses overlapping tokens. The evidence here is mixed but instructive: long-context models can learn long-range patterns, but only if they are designed to do so, and many existing models (including some general-purpose LLMs) may be learning local shortcuts instead.
About These Sources
This answer is built on 10 peer-reviewed studies — published from 2022 to 2026, 9 from 2024 or later, 1 in Q1 journals, collectively cited 633 times — selected as the most relevant from 13 studies that passed quality screening, drawn from 55 papers retrieved from a database of over 500 million.
Sources used in this answer
Leveraging long context in retrieval augmented language models for medical question answering
Proposes BriefContext, a map-reduce strategy to mitigate the 'lost-in-the-middle' problem in medical QA, improving robustness without modifying model weights.
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
Introduces PQCache, using product quantization to compress KV cache, achieving 4.6% score improvement on InfiniteBench over existing methods while maintaining low latency.
BlockPIM: Optimizing Memory Management for PIM-enabled Long-Context LLM Inference
Presents BlockPIM, a cross-channel block memory layout for PIM architectures, achieving 62% average throughput increase over prior PIM solutions for long-context LLM inference.
LoL-PIM: Long-Context LLM Decoding with Scalable DRAM-PIM System
Proposes LoL-PIM, a multi-node PIM architecture with pipeline parallelism, achieving up to 16x speedup over GPU-PIM systems for long-context LLM decoding.
PlantCAD2: A Long-Context DNA Language Model for Cross-Species Functional Annotation in Angiosperms.
Introduces PlantCAD2, a long-context DNA language model with 8,192bp context window, improving accessible chromatin prediction in maize from AUPRC 0.587 to 0.711 by leveraging long-range context.
InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
Proposes InstAttention, offloading attention computation to computational storage drives, improving throughput by up to 11.1x for a 13B model compared to SSD-based solutions like FlexGen.
Long-context LLMs Struggle with Long In-context Learning
Introduces LongICLBench, showing that long-context LLMs like Gemini struggle with extreme-label classification (up to 174 labels at 50K tokens), exhibiting bias toward later labels and poor multi-information reasoning.
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks
Introduces Ada-LEval, a length-adaptable benchmark up to 128K tokens, finding that both closed-source and open-source LLMs show limitations in ultra-long-context settings.
Large Language Models Struggle to Learn Long-Tail Knowledge
Shows that LLMs' ability to answer fact-based questions correlates strongly with the number of relevant documents in pre-training data, and that retrieval augmentation reduces this dependence (569 citations, the most cited here).
Distinguishing word identity and sequence context in DNA language models.
Demonstrates that DNABERT, trained with overlapping k-mers, struggles to learn larger sequence context and relies primarily on token identity, limiting its use for long-range genomic tasks.