[ArXiv 2025] HyMEM: Bridging the Gap Between Open-Source GUI Agents and GPT-4o via Hybrid Self-Evolving Memory
Hybrid Self-evolving Structured Memory for GUI Agents
The paper introduces HyMEM (Hybrid Self-evolving Structured Memory), a graph-based external memory system for GUI agents that combines discrete symbolic strategy nodes with continuous multimodal trajectory embeddings. By implementing a self-evolving mechanism, it enables open-source models like Qwen2.5-VL-7B to achieve a +22.5% performance boost, surpassing closed-source giants like GPT-4o and Gemini-Pro-Vision on complex web navigation benchmarks.
TL;DR
Researchers from UC San Diego and Abel.ai have released HyMEM, a brain-inspired memory system that allows 7B-parameter GUI agents to outperform GPT-4o. By organizing experiences into a hybrid graph of discrete strategies and continuous visual embeddings, HyMEM enables agents to "evolve" their knowledge over time and refresh their focus during long-horizon tasks.
Background: The Problem with "Flat" Memory
Most current GUI agents suffer from a "goldfish memory" or a "shoebox memory." They either look at a single screenshot (reactive) or retrieve past experiences from a flat database (RAG-style).
- Discrete methods (summaries) lose the fine-grained visual nuances of UI elements.
- Continuous methods (embeddings) capture details but struggle with high-level strategic reasoning.
HyMEM addresses this by mimicking the human brain's dual-system: the hippocampus (rich multimodal episodes) and the neocortex (high-level symbolic regularities).
Methodology: The Hybrid Architecture
HyMEM structures memory as an evolving graph .
- Nodes (): Each node contains a High-level Strategy (textual heuristic), Middle-level Attributes (semantic tags), and Low-level Trajectory Embeddings (condensed visual tokens).
- Edges (): Connections are built based on shared attributes, allowing the agent to perform "multi-hop" retrieval—finding not just what looks similar, but what is conceptually relevant.
Figure 1: Overview of the HyMEM system showing the dual-pathway encoding and the four-phase inference cycle.
Self-Evolution: ADD, MERGE, REPLACE
Unlike static databases, HyMEM evolves. When a new successful trajectory is recorded, a VLM judge decides:
- ADD: If the strategy is brand new.
- MERGE: If it provides a new way to do an existing task (e.g., a different UI layout).
- REPLACE: If the new path is strictly more efficient (fewer steps).
Experimental Results: Small Models, Big Impact
The most striking result is the performance leap of open-source models. Qwen2.5-VL-7B, which typically struggles with complex workflows, saw its success rate jump from 12.5% to 35.0%.
Figure 2: Performance across WebVoyager, Mind2Web, and MMInA benchmarks. HyMEM enables 7B/8B models to beat Gemini-Pro and GPT-4o.
Key insights from the experiments:
- On-the-fly Refresh: By detecting "phase shifts" (e.g., moving from 'searching' for a product to 'checkout'), HyMEM refreshes the working memory, preventing the agent from being distracted by outdated context.
- Graph Compression: As the memory grows, the number of nodes grows sub-linearly. This means the agent gets smarter without requiring massive hardware overhead.
Critical Analysis & Future Outlook
HyMEM proves that agent intelligence isn't just about parameter count; it's about how the agent manages its history.
Limitations: Currently, the "update" decisions rely on VLM heuristics. Future iterations could benefit from Reinforcement Learning (RL) to let the agent learn which memories are truly valuable. Furthermore, testing this on 70B+ models could potentially unlock even higher levels of automation.
Conclusion: HyMEM provides a cost-effective, scalable blueprint for the next generation of GUI agents. It moves us closer to a world where AI doesn't just "see" the screen, but "remembers" how to navigate the complex digital world with human-like intuition.