What a memory must keep to answer later questions
The paper asks what an embodied assistant should preserve from past observations so it can answer spatial questions it could not anticipate. The setting is long egocentric video of everyday activity, where the assistant observes a person moving through scenes, handling objects, and revisiting places, without controlling where the person looks or what they do.
Four design choices organize the study: which objects enter memory, how changes in their locations are recorded, what is kept beyond 3D positions, and how earlier observations are retrieved as more scenes are added. The authors argue that restricting memory to objects involved in hand interactions can exclude visible objects that still matter later, and that storing only a position does not preserve an object's contents or its movement history.
Ledger, the memory the paper presents, is built offline from sampled egocentric recordings using supplied or estimated camera poses. An open-vocabulary detector finds objects, their observations are associated over time, and records are kept whether or not the wearer touched the object and after it leaves view. Each record holds timestamped positions, rest segments, and a short description of details such as what a container held or which surface it rested on.
Rest segments turn noisy positions into recorded moves
The mechanism that separates a stationary object from a moving one is the rest segment. Each object's timestamped 3D observations are grouped into segments that sit close to the same resting location, and a new segment, meaning a recorded move, is created only when multiple subsequent observations support the displacement. This persistence rule is the paper's answer to localization noise: without it, an agent can mistake a stationary object for a moving one simply because its estimated coordinates jitter.
The stored record is then read as text. A question first grounds the objects it refers to, their records are retrieved, and a language model answers from those records without seeing images or the original video. That design is what makes the memory reusable: it is built once per recording and read many times, rather than re-reading frames at every question.
Accuracy gains depend on which baseline is compared
On all 2,400 questions in HD-EPIC's 3D Perception and Object Motion categories, adding the memory raises GPT-5.4's prototype-averaged accuracy from 29.7% to 42.6%. On UCS-Bench, where the answerer sees 48 frames, adding Ledger yields 46.3% accuracy, compared with 42.9% when adding DirectMe's graph. On Ego4D VQ3D, queried by object name rather than image crop, the memory returns locations with a median error of 0.99 m among the queries for which it returns a prediction.
These numbers answer different questions and should not be merged. The HD-EPIC figure is a text-only memory against a blind question-and-options baseline; the UCS-Bench figure is a memory added on top of frames; the VQ3D figure is a localization error conditioned on the memory returning a prediction at all. The paper's own cost accounting shows the trade-off: on UCS-Bench, reading 48 frames costs about 9.6k input tokens per question against about 2.0k for the record, and frames alone score 44.4% against 38.5% for the record alone. The record substitutes for frames at roughly a fifth of the answer-time cost and complements them at a small fraction of it.
Persistence, descriptions and retrieval play different roles
Ablations separate the components. Removing event lines costs 2.2 points and removing object descriptions 1.8 points on the larger set of questions answered by both the reference and a given arm, while other removals stay within noise. No single removal moves the overall score by more than 1.3 points on the questions every arm answered, yet category-level changes reach 13 points, as when egocentric position is scored without 3D geometry. The authors conclude that persistence and contextual descriptions contribute more than coordinate refinement.
Retrieval shapes how large a representation's apparent advantage looks. Crossing two records with two readers on 2,777 paired questions, switching ReMEmbR's captions to Ledger's single-shot reader lowers accuracy by 5.0 points, while Ledger's records score 1.2 points below the ReMEmbR reference with its own reader and 0.8 points above it with ReMEmbR's agent. Routing spatial questions to Ledger's reader and the rest to ReMEmbR's agent reaches 42.0% on 3,357 questions. The paper's reading is that comparisons between memories depend on the reader, and that costlier iterative search does not improve every task.
Scene changes break construction as well as retrieval
The study on 100 stitched streams, each joining multiple scenes, exposes failures in both retrieval and construction, and building memory per scene partially recovers the performance lost across scene changes compared with single-scene streams. Memory retains its gains up to 75 minutes of recording length. Delayed questions, re-asked at the end of a stream about an earlier moment, cost the memories 3 to 4 points net of the blind control, and the single-shot reader is weaker there because its time bonus favors what was seen near the question, which for a delayed question is the end of the stream.
The scope conditions are specific. HD-EPIC uses calibrated poses, most UCS-Bench memories lack metric scale, and VQ3D is queried by name rather than crop. Construction is offline, so filtering records by question time does not establish causal online memory, and construction does not yet scale to the most object-dense hour-long recordings. Counting and continuous rest intervals remain unresolved, and recent frame readers outperform memory-only answering on HD-EPIC even though adding the memory to frames improves them.
Where the memory earns its place
The paper's most decision-relevant split is between substituting for frames and complementing them. When the budget is tight, the stored record is the cheaper read: about 2.0k input tokens in one call on UCS-Bench against about 9.6k for 48 frames, at 38.5% versus 44.4% accuracy. When frames are already being read, adding the record costs a further 0.9k tokens, about a tenth of the frame budget, and recovers 1.9 points on top of frames, reaching 46.3%.
The second split is between question types. Admitting untouched objects is what answers fixture questions, and persistence and descriptions carry more of the gain than coordinate refinement, while missing event evidence and fragmented identities limit interaction and counting. The authors' own recommendation follows from this: evaluations should measure answerable evidence alongside localization, and should state retrieval budgets and scene boundaries, because a memory's apparent advantage can come from how it is read rather than only from what it stores.
Sources used in this answer
Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos
Ledger, a persistent 3D object memory built offline from egocentric video, raises HD-EPIC accuracy from 29.7% to 42.6% and UCS-Bench accuracy from 33.8% to 38.5%, and localizes Ego4D objects with a 0.99 m median error on returned predictions.