T-DRQN: Balancing Temporal Memory and Precision in Reinforcement Learning

TL;DR

The paper introduces T-DRQN (Deep Recurrent Q-Network with Truncated History) and T-DARQN, which bridge the gap between fixed-frame DQNs and infinite-memory DRQNs. By explicitly controlling the "length of the past" ($n$) and incorporating visual attention, these models achieve robust performance across diverse Atari games while significantly reducing parameter counts.

Background: The Memory Dilemma

In Deep Reinforcement Learning, the agent's ability to "remember" determines its success in Partially Observable Markov Decision Processes (POMDPs).

• DQN uses a static stack of 4 frames. It is efficient but "forgetful."
• DRQN/DARQN uses an LSTM to maintain state from the start of the episode. While powerful, this can lead to "historical noise"—where irrelevant events from far in the past interfere with current decision-making.

The authors argue for a Trunated History approach: providing the agent with a specific, adjustable window of the past to focus its attention.

Methodology: Truncated History & Visual Attention

Unlike standard DRQN, which passes only the current frame and the previous hidden state, T-DRQN(n) feeds a sequence of $n$ frames directly into the LSTM for every decision.

1. Architecture Evolution

As shown in the architecture comparison, the model moves from simple frame-stacking to a sequence-aware recurrent structure.

2. The Power of Attention

T-DARQN adds a Soft-Attention layer between the CNN and the LSTM. It calculates a context vector $z_t$ based on the spatial features $v_t$ and the previous hidden state $h_{t-1}$. This allows the agent to not only look back in time but also "squint" at the most important part of the screen—like the ball in Breakout or a hidden enemy in Seaquest.

Experiments and Results

The authors tested their models on six Atari 2600 games. The results highlight a crucial insight: Optimal history length is game-dependent.

Performance Highlights

• Seaquest: T-DARQN significantly outperformed the standard DQN, proving that mid-range memory is vital for tasks like rescuing divers and managing oxygen.
• Space Invaders: A clear trend emerged—increasing the truncated history from $n=4$ to $n=8$ led to higher scores, suggesting that some games require "deeper" look-backs than others.

Efficiency Gains

One of the most striking findings was the parameter efficiency of the Attention mechanism. In Breakout, T-DARQN(4) used only ~840K parameters compared to T-DRQN(4)’s ~3.7M, while maintaining higher performance. This suggests that "attending" to features acts as a powerful dimensionality reduction tool.

Visualizing the "Mind's Eye"

By visualizing the attention weights, the authors show the agent focusing on the critical dynamics: the moving ball, upcoming enemies, or player projectiles.

Critical Analysis & Conclusion

Takeaway

The "Truncated History" (T-DRQN) approach provides a necessary middle ground. It grants the temporal benefits of recurrence (Inductive Bias toward time-series) without the stability issues of backpropagating through excessively long sequences.

Limitations & Future Work

The primary drawback is that $n$ (history length) is still a manual hyperparameter. A future "Time Attention" mechanism—where the model learns which specific frames in the past to focus on—could further optimize this.

In conclusion, T-DARQN is a more robust, parameter-efficient alternative to standard recurrent RL models, proving that sometimes, learning to forget the distant past is just as important as remembering it.