Is learning the advantage function directly more sample-efficient than Q-learning?

Replacing the Q-function with two shared estimates

Standard Q-learning stores one number per state-action pair and updates it toward a bootstrapped target. VA-learning instead keeps two estimates: a value function V(x), which is a single number per state, and an advantage function A(x, a), which records how much better action a is than the state's average. A Q-value is reconstructed on demand as Q(x, a) = V(x) + A(x, a), so no separate Q-table or Q-head is maintained.

The advantage function is the quantity policy improvement actually needs, since it measures each action against the state's baseline. In Q-learning it is only recovered afterwards by subtracting an average of Q-values. The paper's question is what happens if that subtraction is built into the learning target rather than applied after the fact.

Why a shared value estimate speeds up bootstrapping

The paper's explanation rests on how information propagates backward. In Q-learning, an update to the pair (y, a) only improves the estimate for that specific action at state y. If a preceding state x needs to bootstrap from a different action b at y, that target stays uninformative until (y, b) happens to be sampled.

VA-learning splits the same backup target into a value target and an advantage target, so sampling action a at state y updates both V(y) and A(y, a). When state x later bootstraps from Q(y, b) = V(y) + A(y, b), it can already use the improved V(y) even though A(y, b) was never updated. The value function is shared across actions, so backup targets from all actions at a state pool into it, and the reconstructed Q-function inherits that pooled information.

Tabular experiments show faster advantage convergence

The tabular tests use randomly generated MDPs with 20 states, 5 actions and a discount factor of 0.99, with trajectories collected under a fixed behavior policy. In the policy evaluation setting the behavior policy is uniform, and the target policy is a convex combination of a deterministic policy and the uniform policy.

Measured by the squared error of the advantage estimate against the true advantage function, VA-learning converges faster and reaches better asymptotic accuracy than Q-learning at any given data budget. On the Q-function error itself, the improvement is smaller: VA-learning shows only a slightly faster decay rate. This distinction matters, because the large reported gain is on the advantage estimate, the quantity tied to policy improvement, not on the Q-function error.

Control tasks and Atari-57 results

In the control setting, the behavior policy is set to a mixture of a fixed deterministic policy and the uniform policy, with a mixing weight that lets the authors vary how stochastic the data collection is. Performance is scored as the value of the greedy policy derived from the learned Q-function, averaged uniformly over states.

For deep RL, the authors build on a DQN agent with double Q-learning and 5-step bootstrapping, trained for 200M frames on the Atari-57 suite, and report the interquartile mean score across random seeds. VA-learning, behavior dueling and uniform dueling do not beat n-step Q-learning on every individual game, but the aggregate comparison favors the decomposed parameterization. Learning rate was the only hyperparameter tuned across variants, and the decomposed methods used a smaller rate than the n-step baseline.

The link to dueling architectures

The paper identifies a close connection between VA-learning and the dueling architecture, which also splits a Q-network into a value head and an advantage head. This partially explains why a simple architectural change to DQN agents tends to improve performance.

The connection is sharpened by what the authors call behavior dueling: parameterizing the advantage as f(x, a) − f(x, µ), where µ is the behavior policy distribution rather than the uniform distribution used in the original dueling design. The paper proves that this choice uniquely minimizes the weighted squared norm of the advantage function, which can be read as minimizing unshared information across actions and maximizing the shared component. Tabular runs across 20 randomly generated MDPs confirm that behavior dueling attains a lower squared advantage norm than uniform dueling.

The equivalence is not exact in practice. With the Huber loss used in the deep RL implementation, the stated equivalence between VA-learning and behavior dueling no longer holds, which the authors flag as a potentially larger discrepancy at scale.

When sharing a value function can hurt

The paper states plainly that VA-learning is not always more sample efficient than Q-learning. The decomposition assumes it is useful to share V(x) across actions from the same state. If the gap between Q-values of different actions at one state is much larger than the gap between value functions of different states, learning those Q-values separately can be preferable to forcing a common value term.

The authors still expect the sharing to pay off in many practical scenarios, and their tabular and deep RL experiments follow that expectation. The condition is worth keeping in view when applying the method: the benefit comes from structure in the problem, not from the parameterization alone.

Where the advantage estimate is the number to watch

The paper's most decision-relevant comparison is not the Q-function error but the advantage error. In the tabular policy evaluation runs, VA-learning's advantage error improves substantially over Q-learning at the same data budget, while its Q-function error improves only slightly. Readers comparing methods on Q-value accuracy alone would therefore understate the difference the paper is claiming.

The practical priority that follows is to check whether the problem has the structure the decomposition assumes: actions at a state sharing a common baseline value. When they do, the paper's evidence supports expecting faster learning from a value-plus-advantage parameterization, with the behavior-policy-weighted advantage normalization as the version with the cleanest theoretical justification.

Sources used in this answer