Gradient Clipping — PROGMATIC

Two key parameters that determine the stability and efficiency of training: max_norm for controlling gradients and clip_param for limiting policy updates in PPO.

What is it?

MAX_NORM

This is a threshold for clipping the gradient norm during neural network training. Gradient clipping is used to prevent "exploding gradients". When calling torch.nn.utils.clip_grad_norm_(policy.parameters(), max_norm=max_norm), the network gradients are normalized so that their total norm does not exceed max_norm.

How max_norm works

If the total L2 norm (the square root of the sum of squared gradients of all parameters) exceeds the threshold, the gradients are rescaled:

g ← g × (max_norm / ||g||₂)

This guarantees that the optimization step magnitude is bounded, stabilizing the training process.

Effect of changing max_norm

DECREASING MAX_NORM (E.G., 0.5 → 0.3)

Constrains gradients more strongly. Can improve training stability. But may slow convergence.

INCREASING MAX_NORM (E.G., 0.5 → 1.0)

Allows gradients to be larger in magnitude. Can speed up convergence. But raises the risk of exploding gradients.

How to manage max_norm

PRACTICAL RANGE

Typical practical values: 0.1 ≤ max_norm ≤ 2.0

Start with a reasonable default value (0.5 is a good baseline)
If you see frequent NaNs or unstable, jumpy losses, decrease the value
If convergence is very slow but training is stable, try increasing the value

clip_param (PPO policy clipping)

WHAT IS IT?

A parameter of the PPO algorithm (Proximal Policy Optimization) that limits how much the policy can change in a single training iteration. PPO uses a clip function so that policy updates happen gradually rather than abruptly.

The PPO loss is:

L^PPO(θ) = E[min(r_t(θ)A_t, clip(r_t(θ), 1-ε, 1+ε)A_t)]

Here r_t(θ) is the ratio of the new action probability to the old, A_t is the advantage, and ε is clip_param.

How clip_param works

clip_param limits the ratio of the new to the old action probability (r_t). If the new policy tries to change the probability by more than ± clip_param, the change is ignored. Thus, optimization steps stay within a trust region, never moving too far from the previous policy.

Effect of changing clip_param

DECREASING CLIP_PARAM (E.G., 0.1 → 0.05)

Constrains policy changes more strongly. Improves training stability. May slow learning.

INCREASING CLIP_PARAM (E.G., 0.1 → 0.2)

Allows the policy to change more. Speeds up learning. Increases the risk of instability.

How to manage clip_param

PRACTICAL RANGE

Typical practical values: 0.05 ≤ clip_param ≤ 0.3

0.1 is the gold standard for PPO
If the agent adapts too slowly and losses are stable, you can increase it (0.15–0.2)
If agent behavior is unstable and performance jumps wildly, decrease the value (0.05–0.08)

Comparison table

Parameter — Controls — Risk if too large — Risk if too small — Typical range
max_norm — Gradient norm (training speed) — Exploding gradients, NaN, instability — Slow convergence, insufficient speed — 0.1–2.0
clip_param — PPO policy change per iteration — Policy instability, performance jumps — Too slow improvement — 0.05–0.3

Summary recommendations

If your model suffers from instability and sudden jumps, focus first on lowering these parameters
If the model is stable but progress is slow, gradually raise the values and watch the result
Never change parameters too aggressively (adjust by ±10–20% from the current value)

In summary

By managing these two parameters together, you gain control over training quality and stability and balance adaptation speed against instability risk.

max_norm controls the stability of the optimization process (gradients)
clip_param controls the stability of the PPO policy (changes in action probabilities)