The λ parameter in Generalized Advantage Estimation (GAE) plays a key role in reinforcement learning. It controls how much the agent trusts future steps when estimating the advantage of an action. However, choosing λ is not just a tuning hyperparameter — it is a decision about the character of training: fast and noisy, or stable and slow.
In this article we will look at:
- what exactly λ does in GAE,
- what effects it produces in practice,
- and whether it is worth adapting it over time as part of a training strategy.
What does λ do in GAE?
Recall: GAE computes the smoothed advantage of an action as an exponentially decaying sum of TD errors:
Â_t = Σ_{l=0}^∞ (γ λ)^l δ_{t+l}TD error
Where δ_t = r_t + γ V_{t+1} - V_t
The parameter λ ∈ [0, 1]:
- λ = 0.0 — only the nearest TD error is used; advantage is computed quickly but noisily.
- λ = 1.0 — the entire path until the end of the episode is taken into account; the signal becomes smooth but more affected by errors in the value function.
The trade-off
Thus, λ controls the trade-off between:
- variance (noise) and
- bias (errors due to inaccurate V).
How λ affects training
At low λ — the agent reacts quickly but is often destabilized. This is useful at the start, when the value function is still rough.
At high λ — training is more stable but slower and more dependent on the accuracy of the value estimate.
- λ = 0.0 — Fast, sharp updates, very sensitive to noise
- λ = 0.9–0.97 — Balance between stability and sensitivity
- λ = 1.0 — Soft but inertial behavior, depends on the V function
How to choose λ: practical recommendations
Static value (most common):
- In PPO, A2C, A3C the typical choice is λ = 0.95 or 0.97
- This value works well in most tasks and does not require tuning.
Dependence on the environment
However…
- If episodes are short → use a slightly lower λ.
- If there is a delayed reward (long-term rewards) → λ closer to 1.0.
Dependence on the critic
- If the value function is poorly trained — better to lower λ to reduce bias.
- If the critic is stable and accurate — λ can be increased to use smoother signals.
Should λ be adapted over time?
This is a little-explored but promising idea.
Potential strategy:
- Early training stages: the agent does not understand how the environment works, the value function is inaccurate, use a low λ (e.g., 0.90) to react quickly.
- Mature stages: the value function is reliable, behavior stabilizes, switch to λ = 0.97–0.99 to smooth the signals and fine-tune the policy.
Possible adaptation formula
Where:
- t — step / epoch / training percentage,
- k — growth rate,
- λ_min, λ_max ∈ [0.8, 0.99]
λ_adaptive(t) = λ_min + (λ_max - λ_min) · (1 - e^{-kt})Possibility of learning λ
Theoretically, λ can be made a learnable parameter, as part of the model. However, this requires:
- stable gradients through the entire GAE computation,
- architectural support and regularization,
- a large number of episodes.
In practice
- In end-to-end systems — rarely used.
- More often — fixed or manually adapted λ.
Visualization and diagnostics
If you want to understand whether λ is appropriate:
- plot the advantages for different λ (we did this earlier),
- check smoothness vs variance,
- analyze convergence: if the policy oscillates — λ is too low.
Conclusion
λ in GAE is a regulator of the memory horizon and the degree of trust in the future. The right choice of this parameter can seriously affect:
- convergence speed,
- policy stability,
- sensitivity to noise and critic errors.
Although in most cases a fixed value works (e.g., 0.95), in complex tasks or advanced systems it is worth considering adaptive strategies, especially in staged training.