Declarative Attention: Letting the Model Say Where the Kernel Should Read

TL;DR

Declarative Attention (DA) is a zero-shot protocol that makes a model state, inside its own chain-of-thought, which segments of a long context it needs to attend to at each reasoning step, by bracketing contiguous spans of reasoning with global, focus and local tags. A state machine beside the inference engine parses those tags and rewrites the KV-cache block table, so the long context is scanned only during declared global steps rather than at every step. Across 15 long-context sources, DA cuts average attended tokens per response by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B while giving up 1.27 and 2.75 accuracy points (Table 2), and a roofline projection estimates decode wall-clock time at 0.71x and 0.77x of vanilla on a single B200 (Table 3). Since nothing is trained, the authors deliberately frame these as lower-bound results.

Positioning

This is a protocol-and-serving paper, not an architecture paper: no new attention operator, no new weights, no kernel changes. Within the sparse-attention literature it occupies an orthogonal axis. Prior work predicts where attention will go — from fixed recency or heavy-hitter rules, or from a per-step scan of a cheaper surrogate of the KV cache — while DA has the model declare it, and reads that declaration as a tool call. The closest ancestor is Self-Selected Attention Span, which fine-tuned the behavior per task on hand-designed partitions inside 2K-token contexts on 7B models (Appendix A.5); DA's claim is elicitation rather than invention, tested with one fixed prompt across two model families, 15 tasks and contexts up to 244K tokens. Two consequences for how you should read the numbers: the empirical comparison is the model's own full attention plus a maskless ablation, not a head-to-head against Quest or DeepSeek's indexer, and the efficiency headline is an analytic roofline ceiling rather than measured latency.

Problem and Motivation

Transformers read the entire KV cache at every decoding step, and in the long-context regime that read, not the arithmetic, sets the clock. Section 1 puts a number on it: for Qwen-3.5-397B-A17B at a 1M-token context, roughly 15 GB of KV cache must be loaded per sequence at every step, a bandwidth demand comparable to loading the model's 17B active parameters. Yet attention mass concentrates on a small subset of tokens, and the subset moves from step to step (the paper cites Child et al. 2019, Zhang et al. 2023 and Tang et al. 2024 for this).

The difficulty is epistemic rather than merely computational: a token's true attention weight exists only after the full attention matrix has been formed, so a selector cannot look up the answer — it must predict or search. The two existing families fail in different, specific ways. Static rules such as recency and accumulated attention mass cannot anticipate which tokens a future query will need, and degrade on long-context tasks (the paper points to Li et al. 2025 and Moschella et al. 2026). Query-aware scans do track the query, but every one of them must touch a summary of every position each step, so complexity per step remains $O(N)$ with a smaller constant. That is the gap DA aims at: not a cheaper scan, but no scan.

Appendix A.1 supplies a second failure mode that is easy to miss and that DA is built to respect: selection accuracy is not the binding constraint, memory layout is. SparQ realizes only 1.3 to 3.0 times of its 6.4 times theoretical speedup because its chosen reads are scattered, and Double Sparsity finds that a contiguous layout alone accounts for most of the available speedup. Any mask that survives contact with HBM has to drop whole aligned blocks.

The motivating intuition is stated plainly in the introduction — wouldn't the model already know which parts of the context are relevant? — and it rests on two lines of evidence rather than on engineering taste: hidden states carry information about future tokens (Pal et al. 2023, Wu et al. 2024), and chain-of-thought surfaces latent computation as readable text (Wei et al. 2022, Korbak et al. 2025). DA extends that principle from what to think to where to attend.

Core Analysis

Where the baseline pays: two cost curves that bend differently

The paper's framing device is a roofline split of a decode step, and the asymmetry between its halves is precisely why DA is possible at all. For the matmul half,

and for the attention half,

In the first, $\mathrm{FLOPs}$ is the per-step matrix-multiply volume fixed by the active parameters, $\mathrm{Peak}\mathrm{FLOPS}$ is the accelerator's dense throughput and $\mathrm{MFU}$ is the achieved fraction of that peak, for which Appendix C.5 adopts 0.40 because per-step GEMMs at decode are skinny. In the second, $\mathrm{KV}\mathrm{bytes}$ is the cache read of one sequence at one step, $\mathrm{Peak}\mathrm{BW}$ is HBM bandwidth and $\mathrm{MBU}$ is the achieved bandwidth fraction, adopted at 0.70. The comparison is not cosmetic: FFN weight loads amortize across the batch, so that term is compute-bound and grows only with the number of decode steps, whereas each sequence owns its own cache, so the attention term stays memory-bound and grows with decode steps and with context length. Table 7 charges a single step at a 1M-token context and reports attention as 99.37% of it on Qwen-3.6-27B and 98.53% on Gemma-4-31B, and the conclusion is robust to the utilization choices — Table 8 shows that sweeping MFU over 40 to 60% and MBU over 60 to 85% moves every full-attention model's 1M share by under 3 points. The counterfactual that defines DA's scope is equally explicit: where architecture has already shrunk the read, leverage falls away. For indexer-based designs the $O(N)$ read is only 352 to 2,772 bytes per context token (Table 6), and GLM-5.3-Flash's attention share at 1M drops to 56.49%. DA is a method whose value tracks how much of the KV read the architecture has left un-shrunk.

What DA changes: the mask becomes generated text

There is no objective here, no gate and no retraining; the formal content of Section 2 is a decode-time state machine plus a constraint on the reasoning trace, and it is worth saying so plainly for readers hunting for a new attention operator. DA requires the model to organize its chain-of-thought into contiguous spans where the attention scope stays stable, and to bracket each span with a tag. Only one region of the prompt is variable — the segmented long input — while three regions stay attended in every mode: a 16-token attention sink at the head of the prompt, the question and DA instruction through the end of the prompt, and the entire response so far (Appendix B).

The essential mechanism is the resolution cost. Reading the closing angle bracket of a focus tag is an $O(1)$ event for the engine, where a DeepSeek-style indexer performs an $O(N)$ scan every step; $O(N)$ reads therefore survive only inside the model's declared global phases instead of as per-step overhead (Appendix A.1). Figure 1 shows the arithmetic on a real 25,466-token prompt: a global step still sees all 25,466 prompt tokens, a focus on one chunk sees 3,435 (an 86.5% reduction), and the concluding local step sees 1,124 (95.6%). Two further design choices are load-bearing. First, masking happens at block granularity: vLLM's kernels read whole blocks of 16 to 32 tokens, so kept spans are rounded outward, costing at most $b-1$ extra attended tokens per span edge against a 2,048-token segment, and allowing FlashAttention to run unmodified through a hook on the attention metadata builder with no kernel or scheduler change. Second, the context is delivered as "magic chunks": segmentation targets 2048 tokens with a 2,560 hard cap, cutting at the coarsest available boundary (paragraph break, then newline, sentence end, clause end, word), and each segment is presented inside a simulated get_magic_chunk tool-use transcript so that boundaries land on the special tokens that delimit user, assistant and tool messages during post-training. The paper's argument is that models track boundaries they have seen in training far more reliably than arbitrary novel delimiters — a reasonable hypothesis, though I note that no ablation in the paper compares addressing schemes or segment sizes, so this remains asserted rather than demonstrated.

Where the saving lands, and where it is capped

Appendix C.8 and C.10 decompose a decode step into three charges, which is the right lens for judging what DA can and cannot fix.

Here $b_{\mathrm{kv}}$ is the bytes of key and value entries per context token summed over the global-attention layers, defined in Appendix C.8 as $2imes{L}_{\mathrm{global}}imes{n}_{\mathrm{kv}}imes{d}_{\mathrm{head}}imes2$, $N$ is the number of attended positions, $\mathrm{B}{\mathrm{W}}_{\mathrm{eff}}$ is 5.6 TB/s on the modeled B200 (8 TB/s at MBU 0.70), $P$ is active parameters and $C$ the matching compute ceiling, 0.9 PFLOPS in bf16, while $s_{\mathrm{local}}$ is the fixed per-step read of the efficient layers — 838.9 MB for Gemma-4-31B's 50 sliding-window layers at a 1,024-token window, against 78.45 MB for Qwen-3.6-27B's 48 Gated DeltaNet states (Table 4). Only the first term contains the attended-token count, and only the first term is anything DA can touch. That single fact explains an asymmetry that runs through the whole results section: the mask lowers $A$, the number of positions read summed across steps, while the protocol's extra verbosity raises $D$, the number of steps, and $D$ multiplies into all three terms. The saving is levied on one line of the budget; the tax is levied on all three.

The boundary condition follows directly. If the efficient layers are cheap, a global reduction passes through almost undiluted; if they are expensive, it does not. Section 5.4 measures this in the roofline arithmetic: the local read is 5% of DA's attention time on Qwen but 42% on Gemma, which is why Gemma's larger global cut (1,100 GB to 528 GB) converts into a smaller total gain than Qwen's. DA also does not apply to those layers at all, because their per-step cost is bounded by a window or a recurrent state rather than by context length — a limitation that is simultaneously the reason the method is safe to deploy on hybrid architectures.

Experiments and Evidence

Table 2 reports the 15-source macro-average for the three arms on the two headline models:

The structure of this table is the argument. The maskless ablation isolates the prompt format: chunked tool-use rendering plus three-mode instructions costs essentially nothing in accuracy (identical 87.01 on Gemma, within 0.69 points on Qwen) but buys nothing in cost — it reads 66.2% more tokens than vanilla on Gemma and 28.8% more on Qwen, because the protocol makes the model write more. Figure 2 pins the overhead down: both DA arms run about 15 to 35% more decode steps than vanilla, and Table 5 records 332 versus 448 steps on Gemma and 573 versus 752 on Qwen. The mask then converts that overhead into a net win, cutting attended tokens relative to the ablation by 71.1% on Gemma and 46.5% on Qwen, and accounting for nearly all of the accuracy cost as well (minus 1.27 points on Gemma, minus 2.06 on Qwen against the ablation). So the paper's decomposition is unusually clean about its own mechanism: the savings and the harm both come from the mask, not from the formatting.

Aggregate numbers hide where selective attention actually pays. DA matches or beats vanilla on 7 of 15 tasks on Gemma and 5 of 15 on Qwen, with several matches sitting at the accuracy ceiling of the needle-in-a-haystack tasks, and the clearest single-task gains are longdep_qa on Gemma at plus 3.1 points and code_repo on Qwen at plus 5.6 points. Losses concentrate in multi-span reasoning rather than single-span retrieval, with category-average drops of 2.28 versus 0.78 points on Gemma and 3.59 versus 2.34 on Qwen — exactly what one would expect when a question requires synthesizing facts that segmentation put into different chunks. Absolute savings, by contrast, land where the raw cost is: code_repo saves 41.8M tokens per response on Gemma and 52.0M on Qwen, dialogue_history 22.1M and 39.1M. On Qwen, DA's attended tokens still exceed vanilla on five sources, most visibly qmsum and the two LongBench v2 QA sources, because longer generations outpace the per-step saving.

Scaling is the part of the story that supports the lower-bound framing. Figure 3a and Figure 6 report these endpoints:

Relative accuracy rises monotonically with size within both families, and the collapse at the smallest scale is a protocol-adherence failure rather than a reasoning failure: Gemma-4-E4B's 58% focus-parse success mirrors its accuracy drop, while focus attempts per response stay in a narrow 1.4 to 1.9 band across all sizes, so stronger models succeed by resolving their declarations, not by making fewer of them. Meanwhile the mechanism itself is scale-independent: the per-step attention ratio sits near 0.5 on five of the six models, at 0.65 on Gemma-4-E4B, which makes the ratio the cleaner measure of masking and the total the measure of elicitation. The one model that reads more than vanilla, Gemma-4-12B at 183% relative attended tokens, is a generation-length artifact rather than a mask failure.

Context-length scaling is where the practical case is strongest. Figure 4 shows DA attending about 1M fewer tokens than vanilla at short contexts and about 21M fewer in the longest bin, while relative accuracy holds within about one point of vanilla up to 32K tokens and then declines to roughly 96% in the longest bin — and crucially the maskless ablation does not show that decline, which localizes the long-context accuracy cost to the mask rather than to chunked prompting. Figure 5 explains both the growth and its cap on Gemma-4-31B: global accounts for about 27% of generated tokens, and focus and local together the remaining 73%, with focus reading about 12% and local about 6% of what a vanilla step reads, saving 76 to 99% per token. The global share rises to about 45% in the longest bucket, and global steps overall retain over 80% of DA's attended tokens (Section 8.2), so navigation is where the remaining money is.

Table 3 converts token counts into the roofline wall-clock estimate for one response, averaged over the 15 tasks, on a single B200 in bf16 at MFU 0.40 and MBU 0.70:

Read it as three columns telling one story: the global read dominates vanilla decode at 73% of the time on Gemma and 86% on Qwen, the two context-independent columns rise under DA because the protocol generates more steps, and the net lands at 0.71x and 0.77x. These are ceiling estimates, not measurements — Appendix C.8 says so explicitly, and they exclude prefill on the assumption of phase-disaggregated serving.

Table 10 reports the six sources where DA fails on Gemma-4-31B, and the two failure modes are structurally informative:

The first cluster is a decomposition failure: a global count over all segments, or a table split across chunks, does not survive focus seeing only the named segments. The second is a decode-length problem, not an accuracy one — the task emits one output unit per segment, and the model writes that long output under global at full attention. Notably, per-step attention still falls by 39 to 67% on all six, so the mechanism is working while the strategy is not. Judge quality, which matters a lot when the headline effect is 1.27 points, is checked in Table 9 and Figure 7: the Qwen-3.5-4B judge agrees with Gemini-3.1-Pro on 98.53% of 2,993 re-judged verdicts, with per-cell accuracies correlated at Pearson $r=0.992$ and symmetric disagreement (McNemar exact $p=0.65$).

On evidence strength overall: the strongest claims are cross-setting — the mask's contribution is shown by an ablation that holds the prompt fixed, the scaling trend holds in two independent model families, and the roofline conclusion survives a full sweep of plausible utilizations. The weaker claims are the ones that live at a single setting: contexts are capped at 244K in the traces while the 1M figures are analytic, thinking mode is disabled because preliminary experiments found models cannot follow the protocol inside thinking tags, and the benchmark uses manufactured segments on static corpora rather than the natural turn and tool boundaries of deployment. The paper also does not measure wall-clock time end to end, and it never compares DA against a learned indexer on the same tasks, which is the head-to-head a serving team would ask for.

Deep Insights and Summary

The transferable idea is a reattribution of work. Sparse attention has historically treated the mask as something a system must infer — from a heuristic, a surrogate score, or a learned index — and DA treats it as something the model can assert, converting an $O(N)$ per-step computation into an $O(1)$ parse plus a modest step-count tax. Section 8.4's framing as system-2 sparse attention is more than rhetoric: because the strategy is written in language, it can be changed by instruction, it improves with scale without changing the protocol, and it becomes auditable, since the tokens that make the plan readable are the tokens driving the KV reads. Two properties also distinguish DA from the eviction literature and give it engineering reach the paper only sketches: nothing is discarded, so a focus step can be followed by a global step that re-attends everything, and the attended set changes only at span boundaries that are announced in text before the tokens that read them — the exact handle that KV offloading and reversible context compaction currently lack.

The limitations are concrete and mostly attributable to zero-shot elicitation rather than to masking. The protocol's own overhead is unfixed: a third more decode steps, and global retaining over 80% of attended tokens, mean the remaining headroom is in surveying rather than in focusing. Small models may not get DA at all — Gemma-4-E4B at 29% relative accuracy and 58% parse success defines a capability floor. Segmentation destroys answers requiring global aggregation or intact tables, and several RULER sources were excluded outright because their prompts forbid intermediate reasoning, which quietly restricts the benchmark to tasks that reward chain-of-thought. Finally, the accuracy deltas are reported without confidence intervals, so it is the 15-source average and the maskless control that give the 1.27-point number weight, not its size at any single task.

The most valuable next result would replace the survey step: an in-context index of segment summaries, orders of magnitude smaller than the context, would attack the 80% of attended tokens that global still holds, and Appendix C.9's arithmetic shows why that matters most exactly as per-token KV bytes fall. Pairing the protocol with a scan-based indexer inside global steps, exposing DA's operations as native tool declarations so that interleaved thinking can carry it into long reasoning traces, and reinforcing mode selection with training that rewards accuracy and attended tokens jointly are the three directions this paper sets up rather than solves.