In many policy-gradient RL algorithms (e.g., PPO, A2C), it is necessary to estimate the advantage of an action — how much better or worse it turned out to be compared to expectations.
However, estimating the advantage is non-trivial: you can obtain low variance but strong bias, or get an unbiased result with enormous noise.
GAE was proposed by Schulman et al. in 2016 as a way to find the optimal balance between these two extremes, providing more stable and efficient training.
Recap: what is the advantage
The advantage shows how much better than expected a specific action in state s turned out to be.
A_t = Q(s_t, a_t) - V(s_t)
Practical n-step approximation
In practice we know neither Q nor V exactly. Usually an approximate formula via n-step return is used:
A_t = r_t + γ r_{t+1} + … + γ^{n-1} r_{t+n-1} + γ^n V(s_{t+n}) - V(s_t)Trade-off in n-step
GAE introduces a weighted summation of these n-step estimates with controllable smoothing.
- If we take n = 1, we get a very noisy estimate.
- If we take n = T, there will be bias due to errors in V.
The essence of GAE: exponential smoothing of TD deltas
Instead of a fixed step, GAE computes the advantage as an exponentially decaying sum of TD errors (δ):
δ_t = r_t + γ V(s_{t+1}) - V(s_t)GAE formula
And then:
Â_t^{GAE(γ, λ)} = δ_t + γ λ δ_{t+1} + (γ λ)^2 δ_{t+2} + …Role of λ
The parameter λ ∈ [0, 1] controls the weight of future steps:
- λ = 0: only the current TD error (minimum variance, maximum bias),
- λ = 1: full T-step return (minimum bias, maximum variance),
- Values from 0.9 to 0.97 — the optimal compromise in practice.
Why GAE matters
GAE provides:
- more stable gradients,
- less noise in policy training,
- better convergence on complex tasks.
GAE in PPO
In PPO, for example, using GAE is the de facto standard. Without it, one observes:
- poor stability in tasks with delayed rewards,
- high sensitivity to training hyperparameters.
Intuition: how to think about GAE
Imagine the agent received rewards r_0, r_1, r_2, …, and wants to figure out: was the action at time t a good one?
The TD-error estimates are "small signals" of how strongly expectation diverged from reality. GAE says:
"Don't only look at the nearest error, but also consider future errors, with a decaying weight."
That is exactly what GAE does — it integrates information about the future, but does not let it blow up the gradients.
Practical implementation (in words)
Suppose we have:
- an array of rewards r,
- an array of values V,
- parameters γ, λ.
Algorithm
Then the algorithm is:
- Iterate backwards through the episode.
- For each t, compute: δ_t = r_t + γ V_{t+1} - V_t
- GAE updates recursively: A_t = δ_t + γ λ A_{t+1}
- The result is a smoothed advantage vector A, which feeds into the PPO loss.
Parameters and their effect
Low λ → fast but noisy gradient.
High λ → stable but slow.
- γ — typically 0.99 — how important future rewards are
- λ — 0.95–0.97 — balance between noise and bias
- V(s) — neural network — must be sufficiently accurate for TD
Conclusion
GAE is a fundamental component of modern RL algorithms, allowing one to:
- estimate action quality more accurately and stably,
- control the balance between long-horizon credit assignment and stability,
- accelerate training and reduce overfitting on noisy episodes.
Although mathematically simple, GAE has a deep impact on agent behavior, especially in tasks with delayed rewards, stochasticity, or long episodes.