[NVIDIA] SideQuest: Teaching Agents to "Garbage Collect" Their Own KV Cache

TL;DR

Scaling autonomous agents to handle "deep research" (multi-hour, multi-step tasks) is currently blocked by the massive memory footprint of the KV cache. While standard compression methods use simple math (heuristics) to delete tokens, SideQuest uses the model’s own intelligence. By running a parallel "memory management" thread, the model identifies which parts of its history are no longer needed, slashing peak memory usage by up to 65% while maintaining SOTA reasoning accuracy.

The Problem: The "Non-Monotonic" Importance Trap

In a typical RAG (Retrieval-Augmented Generation) or agentic workflow, the context is a messy stream of search results, webpage snippets, and internal thoughts.

Traditional compression (like H2O or SnapKV) assumes that if a token hasn't been attended to recently, it's safe to delete. This is wrong for agents. A webpage URL retrieved in step 1 might be ignored for 50 steps of research, only to be essential in step 51 for the final citation. Heuristics would have deleted it long ago, leading to "hallucinations" or model collapse.

Methodology: The Parallel "SideQuest"

SideQuest shifts memory management from a system-level heuristic to a model-level reasoning task.

1. Parallel Auxiliary Threads

Instead of interrupting the agent's "thought" to clean up memory (which adds latency), SideQuest forks a parallel thread. This thread shares the same KV cache but focuses exclusively on a maintenance sub-routine.

2. Semantic Garbage Collection

The model is trained with a specific trigger phrase: ** Memory management mode **. When triggered, the model doesn't try to solve the user's query; it looks at its own context "cursors" (id-tagged tool outputs) and determines which ones are logically obsolete based on the current research state.

Figure 1: SideQuest architecture showing the main reasoning thread and the parallel auxiliary thread performing context cleanup.

3. Hindsight Training

NVIDIA researchers trained this capability using only 215 high-quality samples. They used "hindsight annotation": they took successful research traces, looked at which documents were never used again after a certain point, and used that as ground-truth for training the "garbage collector" logic.

Experiments: Efficiency without the Accuracy Tax

The team tested SideQuest on FRAMES and BrowseComp, benchmarks that require processing over 100k tokens of retrieved web data.

• SOTA Pareto Frontier: SideQuest achieved a massive reduction in peak memory (56-65%) while staying within 2-5% of the "Uncompressed" baseline accuracy.
• Reliability: Unlike heuristic methods which often caused the model to produce "unparsable" gibberish (due to accidental deletion of syntax tokens), SideQuest maintained structural integrity.

Figure 2: Performance comparison—SideQuest (red star) sits significantly higher and further left than heuristic baselines.

Serving Performance

Implemented in SGLang on an NVIDIA H100, the results are even more practical:

• Throughput: +83.9% increase in tokens/sec.
• Batch Size: Systems can handle higher concurrency because each request occupies 53.9% less memory.

Critical Analysis & Insight

The brilliance of SideQuest lies in its orthogonality. It doesn't replace architectural optimizations like Mamba or GQA; it works on top of them.

Limitations: Currently, SideQuest only prunes external tool outputs (webpages/docs). It doesn't yet "compress" the agent's own internal chain-of-thought. As agents move toward "System 2" thinking (spending 10 minutes thinking for 1 second of output), pruning the "thought" cache will be the next frontier.

Conclusion

SideQuest proves that as agents become more intelligent, we should stop treating them as "black boxes" that need external management. By giving the model a "SideQuest" to maintain its own brain, we can build agents that perform deep research for hours without hitting the GPU memory wall.

Metadata Summary:

• Model: gpt-oss-20b (NVIDIA)
• Key Metric: 65% KV Cache reduction
• Framework: SideQuest (Parallel Auxiliary Reasoning)