Splicing activations to test internal structure
The central tool is not a new benchmark but an intervention. The authors first run a prompt and record the inputs to each transformer decoder block across the span of the relationship statements, stacking these recordings into an activation tensor. They then build a "mosaic" by taking that tensor from one context and replacing selected slices with slices recorded from a different context, and finally read out the model's answer probabilities on a query. Because only the recorded internal states change while the query stays fixed, a shift in the answer distribution points to what the replaced slice was carrying.
This design matters for interpreting everything that follows. A model that merely answers relational questions correctly could be using surface heuristics, memorized name associations, or token position. Replacing a single entity's activation slice with a slice from a prompt containing different names lets the authors ask whether the model behaves as if that entity now occupies a different position in the same relational structure. Each condition used 50 randomly sampled entity assignments, and answers were scored as next-token probabilities over the full vocabulary, with each answer occupying a single token.
Scale gates the inference, not the format
The behavioral result sets the scope. Models beyond the ten-billion-parameter scale inferred unstated relationships between entities and generalized to synthetic entity names, which were random three-letter strings chosen to tokenize as a single token when preceded by a space. That name substitution is the important control: it removes prior knowledge about who Alice or Beth is, leaving only the relational structure supplied in the prompt.
The model set spans Gemma 3 at 4B, 12B and 27B, Llama 3.1 Instruct at 8B and 70B, and Qwen3 from 0.6B to 32B, so the scale boundary is drawn from within-family comparisons rather than a single pair. The paper reports that the mosaic patterns held across every relational-reasoning setting and every model tested, provided the model could perform the underlying task. That proviso is the real boundary: the representational story is offered for models that already solve the task, not as an explanation of why smaller models fail.
A single slice carries a structural role
The first mosaic experiment replaced the activation slice at one entity position in a control prompt with the corresponding slice from a donor prompt that used entirely different names but the same relational form. If the slice carried only a name, the model should treat the substituted entity as that name. Instead, models recovered relationships consistent with the entity having taken over the donor entity's place in the full structure: they recovered the direct relationship, a relationship involving the neighboring entity, and even the two-step relationship and its inverse.
The authors use this to rule out two simpler accounts. If the tensor encoded only token positions, the model would read the mosaic as a sequence and would not produce the two-step inference. If it encoded two separate mother–daughter pairs linked only by a shared name, the cross-pair inference would also fail. What fits all the observations is a representation of the whole relational structure, with entity names bound to structural roles, meaning an entity's position within the overall structure rather than its slot in the sentence.
Identity and role add rather than entangle
The next experiment tests how a name is attached to a role. Under the paper's parallelogram hypothesis, a row of the activation tensor for a name bound to a role decomposes approximately into a name-dependent component plus a role-dependent component. Geometrically, pairing any two names with any two roles should form a parallelogram, which predicts that the change in activity when an entity switches roles does not depend on which name it is.
The authors estimate that change, a role-swap vector, by subtracting the same name's rows from two donor prompts that place it in different roles, which cancels the shared name component. Adding the vector to one control entity and subtracting it from another should then exchange their roles. Models responded as predicted, and the pattern held across most relational-reasoning settings and models tested. The result is a linear decomposition: one component carries structure independent of names, the other carries the names that instantiate it.
Links are stored and composed at query time
A separate question is how the structural component organizes the observed relationships. The paper contrasts a retrospective–composition account, in which observed relationships are stored as pairwise links and combined when a query demands it, with a prospective account, in which each entity is placed in a global coordinate system in advance. The mosaic evidence favors the first: multi-step inference relied primarily on separately represented relationships that were composed when required by the query.
This distinction changes what "cognitive map" should mean here. The model is not maintaining a survey-like layout of all entities. It holds the edges it has seen and traverses them on demand, which is why an unstated two-step relation can be recovered without that relation being stored anywhere as a single entry.
Sparse key–value pairs bind names to roles
The binding between names and structural roles is mediated by key–value pairs, the memory mechanism transformers use to route information between token positions. The authors identify a sparse, distributed set of candidate pairs by iterative pruning, and in one reported case 237 keys and 15 values were retained, together comprising 3.94% of all key/value sites. Replacing keys first, values first, or both produced different patterns of degradation, which is how the paper separates name-to-role retrieval from role-to-name retrieval.
The strongest test is editing rather than erasure. Replacing the values of candidate role-to-name pairs associated with one entity with their counterparts from a donor memory left the model still identifying Beth as Alice's daughter, but now identifying Daisy rather than Alice as Beth's mother. The two answers cannot be reconciled into one consistent family tree, so the authors describe this as a unidirectional false memory: a predictable behavioral change produced by perturbing a small set of internal pairs. Related edits altered responses across relational-reasoning tasks and models.
What the analogy does and does not license
The paper's own framing is an analogy, not an identity claim. It argues that the internal organization is analogous to a cognitive map because a reusable structural component is instantiated with the particulars of the current context, which is the same division of labor proposed for schemata, mental models and cognitive maps in animals. The evidence for that division is the additive decomposition and the query-time composition, both established by intervention in open-weight models.
Two boundaries follow directly from the design. The tasks are controlled relational-reasoning prompts with single-token entity names and fixed-length slots, so the results describe how these models handle small, clean relational structures rather than open-ended text. And the causal claims rest on activation and key–value edits in the listed open-weight models; the paper does not establish that the same organization appears in closed models or in models below the scale threshold that failed the underlying task.
The most useful takeaway is a priority for interpreting model errors. If relationships are stored as pairwise links composed at query time, then a wrong multi-step answer is better explained by a failure in composition or in name-to-role binding than by a missing global map. The editing results sharpen this: because a small, sparse set of key–value pairs can be altered to produce a specific, directional false memory while leaving the inverse answer intact, the binding step is the place where a model's relational behavior is most precisely controllable.
Sources used in this answer
Cognitive Map in Large Language Models
Open-weight language models above roughly ten billion parameters represent relational structure separately from entity identity, store observed relationships as pairwise links composed at query time, and bind names to structural roles through sparse key–value pairs that can be edited to implant directional false memories.