[CVPR 2024] EgoGraph: Structuring Long-Term Memory for Ultra-Long Egocentric Video Understanding

TL;DR

Processing video that lasts for days is a nightmare for standard Transformers and RAG systems. EgoGraph solves this by ditching hierarchical text summaries in favor of a Temporal Knowledge Graph. It mimics human episodic memory by organizing "who, what, and where" into a structured, timestamped graph. The result? A system that doesn't just "watch" video but "remembers" it, outperforming Gemini-1.5-Pro and GPT-4o on long-term reasoning tasks.

Problem & Motivation: The Fragmentation Trap

Current SOTA for long-video (like EgoGPT) follows a hierarchical summarization path:

1. Break video into clips.
2. Generate captions.
3. Summarize captions into hours, then days.

The catch? This process is fragmented. If a person ("John") appears on Monday and again on Thursday, the hierarchical summary treats these as isolated textual events. The "connective tissue"—the entity-centric dependency—is lost. Furthermore, as the video grows to 7+ days, the sheer volume of unstructured text leads to retrieval noise and context window overflow.

The authors' key insight is that we need a representation where time is a first-class citizen and entities are the anchors.

Methodology: The Core of EgoGraph

1. The Egocentric Schema

Instead of a "wild west" of labels, EgoGraph enforces a strict schema inspired by human memory. It categorizes everything into four types: Person, Location, Object, and Event. Each type has specific attributes (e.g., a Person has "habits" and "preferences"). This ensures the graph remains semantically dense and organized.

2. Temporal-Aware Graph Construction

Unlike static graphs, every node ($v$) and edge ($e$) in EgoGraph is a tuple containing:

• $\mathcal{T}$ (Timestamps): A list of every time the entity/relation was seen.
• $\mathcal{D}$ (Descriptions): The evolving state of that entity.

Figure 1: The EgoGraph pipeline transforming raw video into a structured event-driven memory.

3. Dynamic Merging

To prevent the graph from exploding in size, EgoGraph uses embedding similarity to merge redundant nodes. If "John" is detected in 100 clips, he exists as one node with 100 timestamps, preserving his "evolutionary trajectory."

4. Temporal Filtering & Reasoning

When a user asks a question at a specific time ($t_q$), EgoGraph applies a Temporal Filter: ${\mathcal{G}}^{\le t_q}=\{(v,e)\in \mathcal{G}:\forall t\le t_q\}$ This prevents "temporal leakage" (the model accidentally seeing the future) and allows the LLM to resolve relative time queries like "What did I do yesterday?" by calculating $(t_q-1extday)$.

Experiments & Results

EgoGraph was tested on EgoLifeQA (7 days of wearable camera footage) and EgoR1-bench.

SOTA Performance

EgoGraph achieved 45.8% accuracy, towering over Gemini-1.5-Pro (36.9%) and GPT-4o (36.2%). In complex tasks like TaskMaster (reasoning about sequences), it cleared the 60% mark.

Table 1: Quantitative results showing EgoGraph leading across all sub-categories.

Graceful Scaling vs. Sudden Death

The most striking result is the Temporal Robustness study. As the video length increases from 1 to 7 days:

• Plain-text RAG: Accuracy drops from 43% to 8% (Context window failure).
• EgoGPT: Stagnates around 30%.
• EgoGraph: Stays stable, ending at 45.8%.

Critical Analysis & Conclusion

Takeaway: EgoGraph proves that for embodied AI to understand long-term human life, it must move away from "video as a sequence of frames" to "video as an evolving database of entities."

Limitations:

• Training-Free reliance: While being "training-free" is a strength for deployment, the system's accuracy is heavily bottlenecked by the underlying LLM's (GPT-4o) ability to extract clean triplets.
• Visual Captioner Quality: If the initial MLLM (InternVL) misses an object, it never enters the graph.

Future Outlook: This architecture is a blueprint for "Personal AI Assistants" that live on AR glasses. By maintaining a compact, searchable graph of your life, these systems can provide real-time memory augmentation without needing to re-process terabytes of raw video.