[TNNLS 2025] Pessimistic Auxiliary Policy: Shifting the Sampling Paradigm to Cure Offline RL Overestimation

TL;DR

Offline Reinforcement Learning (RL) is notoriously fragile due to overestimation bias—where the agent "hallucinates" high rewards for actions it has never seen. This paper proposes the Pessimistic Auxiliary Policy (PAP): instead of just constraining the policy, it creates a "shadow" auxiliary policy that specifically samples actions with high confidence and low uncertainty. By using these reliable samples during TD updates, it cuts estimation errors by up to 95% and shatters SOTA benchmarks on complex tasks like AntMaze.

The "OOD" Nightmare: Why Offline RL Fails

In offline RL, we only have a static dataset. When the agent tries to optimize its policy, it naturally explores actions. If an action $a^{\prime }$ is Out-of-Distribution (OOD), the Q-network (which hasn't seen this state-action pair) might return a random, highly inflated value. In the standard Bellman equation: $$Q(s,a)\leftarrow r+\gamma Q(s^{\prime },\pi (s^{\prime }))$$ If $\pi (s^{\prime })$ picks an OOD action with an overestimated value, that error flows backward, infecting the entire value function. This "error accumulation" eventually leads to policy degeneration.

Methodology: Pessimism via Taylor Expansion

The authors' core "Aha!" moment is the construction of a Pessimistic Auxiliary Policy (${\pi }_p$).

1. Epistemic Uncertainty

They quantify the "strangeness" of an action using the variance between ensemble Q-functions: $${\delta }_Q(s,\mu )=\frac{1}{2}|Q^1(s,\mu )-Q^2(s,\mu )|$$ High variance means the model is guessing (OOD).

2. The Lower Confidence Bound (LCB)

Instead of trusting the raw Q-value, they use a conservative estimate: $$Q_{LB}=extMean(Q)-\beta \cdot {\delta }_Q$$

3. Finding the Direction of Reliability

To make this computationally efficient, they apply a first-order Taylor expansion to $Q_{LB}$. This allows them to find a new action direction that maximizes "safe value" within a small neighborhood ($\sigma$) of the current policy.

Fig 1: Schematic of the Pessimistic Auxiliary Policy shifting sampled actions toward low-uncertainty regions.

The resulting closed-form update for the auxiliary action ${\mu }_p$ is: $${\mu }_p=\mu +\alpha abl{a}_a{Q}_{LB}(s,a){|}_{a=\mu }$$ This direction effectively "nags" the agent to stay where the data is densest and the model is most certain.

Experiments: Breaking the AntMaze

The researchers integrated PAP into TD3BC and Diffusion-QL. The results were staggering, particularly in "sparse reward" and "narrow distribution" environments like AntMaze.

Why does it work?

By analyzing the value estimation error, the authors found that in the halfcheetah-medium-expert task, the Q-estimation error dropped from 341.42 (TD3BC) to just 16.43 (TD3PA).

Table: Comparison of overestimation reduction across different tasks.

Critical Insight & Conclusion

Most offline RL methods are "defensive"—they punish the agent for moving away from the dataset. PAP is "proactive"—it provides the agent with a better compass. Instead of just saying "don't go there," it calculates a specific direction towards "reliable high-value" actions.

Limitations: The method relies on the quality of the ensemble uncertainty estimation. If the underlying ensemble fails to capture the uncertainty (e.g., both Q-networks are equally wrong), PAP might still sample OOD actions.

Future Outlook: This approach is model-free and mathematically elegant. It opens the door for using auxiliary policies in Transformers (Decision Transformers) or World Models to guide latent state transitions toward high-certainty manifolds.