Pessimistic Auxiliary Policy for Offline Reinforcement Learning

[TNNLS 2025] Pessimistic Auxiliary Policy: Shifting the Sampling Paradigm to Cure Offline RL Overestimation

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

Summary

Problem

Method

Results

Takeaways

Abstract

The paper introduces the Pessimistic Auxiliary Policy (PAP), a novel framework designed to enhance offline Reinforcement Learning by sampling more reliable, low-uncertainty actions for Temporal Difference (TD) updates. By integrating PAP into existing methods like TD3BC and Diffusion-QL, it achieves significant performance boosts across D4RL and NeoRL-2 benchmarks.

TL;DR

Offline Reinforcement Learning (RL) is notoriously fragile due to overestimation bias—where the agent "hallucinates" high rewards for actions it has never seen. This paper proposes the Pessimistic Auxiliary Policy (PAP): instead of just constraining the policy, it creates a "shadow" auxiliary policy that specifically samples actions with high confidence and low uncertainty. By using these reliable samples during TD updates, it cuts estimation errors by up to 95% and shatters SOTA benchmarks on complex tasks like AntMaze.

The "OOD" Nightmare: Why Offline RL Fails

In offline RL, we only have a static dataset. When the agent tries to optimize its policy, it naturally explores actions. If an action $a^{'}$ is Out-of-Distribution (OOD), the Q-network (which hasn't seen this state-action pair) might return a random, highly inflated value. In the standard Bellman equation: $Q (s, a) \leftarrow r + γ Q (s^{'}, π (s^{'}))$ If $π (s^{'})$ picks an OOD action with an overestimated value, that error flows backward, infecting the entire value function. This "error accumulation" eventually leads to policy degeneration.

Methodology: Pessimism via Taylor Expansion

The authors' core "Aha!" moment is the construction of a Pessimistic Auxiliary Policy ( $π_{p}$ ).

1. Epistemic Uncertainty

They quantify the "strangeness" of an action using the variance between ensemble Q-functions: $δ_{Q} (s, μ) = \frac{1}{2} ∣ Q^{1} (s, μ) - Q^{2} (s, μ) ∣$ High variance means the model is guessing (OOD).

2. The Lower Confidence Bound (LCB)

Instead of trusting the raw Q-value, they use a conservative estimate: $Q_{L B} = e x t M e an (Q) - β \cdot δ_{Q}$

3. Finding the Direction of Reliability

To make this computationally efficient, they apply a first-order Taylor expansion to $Q_{L B}$ . This allows them to find a new action direction that maximizes "safe value" within a small neighborhood ( $σ$ ) of the current policy.

Model Architecture Fig 1: Schematic of the Pessimistic Auxiliary Policy shifting sampled actions toward low-uncertainty regions.

The resulting closed-form update for the auxiliary action $μ_{p}$ is:

abla_a Q_{LB}(s, a)|_{a=\mu}$$ This direction effectively "nags" the agent to stay where the data is densest and the model is most certain. ## Experiments: Breaking the AntMaze The researchers integrated PAP into **TD3BC** and **Diffusion-QL**. The results were staggering, particularly in "sparse reward" and "narrow distribution" environments like AntMaze. | Task | TD3BC (Base) | TD3PA (Ours) | Improvement | | :--- | :--- | :--- | :--- | | AntMaze Total | 163.8 | **425.6** | **+159.8%** | | Adroit Total | 9.9 | **124.6** | **Huge Leap** | | NeoRL-2 (Real World) | 448.6 | **465.6** | **+3.8%** | ### Why does it work? By analyzing the value estimation error, the authors found that in the `halfcheetah-medium-expert` task, the Q-estimation error dropped from **341.42** (TD3BC) to just **16.43** (TD3PA). ![Result Comparison](https://cdn.atominnolab.com/wisdoc/jobs/20260303-3a36e663-0522-4755-adce-a254ed42de7b/page_006_block_007.png) *Table: Comparison of overestimation reduction across different tasks.* ## Critical Insight & Conclusion Most offline RL methods are "defensive"—they punish the agent for moving away from the dataset. PAP is "proactive"—it provides the agent with a **better compass**. Instead of just saying "don't go there," it calculates a specific direction towards "reliable high-value" actions. **Limitations**: The method relies on the quality of the ensemble uncertainty estimation. If the underlying ensemble fails to capture the uncertainty (e.g., both Q-networks are equally wrong), PAP might still sample OOD actions. **Future Outlook**: This approach is model-free and mathematically elegant. It opens the door for using auxiliary policies in **Transformers (Decision Transformers)** or **World Models** to guide latent state transitions toward high-certainty manifolds.

Find Similar Papers

Try Our Examples

Find recent papers from 2024-2025 that use Epistemic Uncertainty to address the distribution shift problem in Offline Reinforcement Learning.
Which paper first proposed the use of a Lower Confidence Bound (LCB) in Q-learning, and how does this paper's Taylor-expansion approach differ in implementation?
Explore if the Pessimistic Auxiliary Policy concept has been applied to Multi-Agent Offline RL or Offline RL with Human-in-the-loop datasets.

Contents

[TNNLS 2025] Pessimistic Auxiliary Policy: Shifting the Sampling Paradigm to Cure Offline RL Overestimation

1. TL;DR

2. The "OOD" Nightmare: Why Offline RL Fails

3. Methodology: Pessimism via Taylor Expansion

3.1. 1. Epistemic Uncertainty

3.2. 2. The Lower Confidence Bound (LCB)

3.3. 3. Finding the Direction of Reliability

4. Experiments: Breaking the AntMaze

4.1. Why does it work?

5. Critical Insight & Conclusion