Vanishing Gradients

How to solve the vanishing gradients problem in deep neural networks and ensure stable training across all layers.

What are vanishing gradients?

DEFINITION

When training neural networks we use backpropagation: the output error is propagated back through the layers, adjusting weights. But in deep networks gradients can become too small as they move toward the early layers. This is vanishing gradients — the situation where the "error signal" fades and early layers stop learning.

Why it happens

1. Activation math. Many activation functions (e.g., sigmoid, tanh) squash values into a narrow range. Their derivatives in those zones are small. Multiplying these small derivatives at every layer causes the gradient to shrink exponentially.

Example: if the derivative ~ 0.1, then after 10 layers the gradient becomes on the order of 10^(-10) — essentially zero.

2. Chain rule effect. Backpropagation uses the product rule of derivatives. If every link is small, the product collapses to zero.

3. Network depth. The more layers, the higher the risk that a useful gradient simply does not reach them.

Why it is a problem

CONSEQUENCES

Early layers do not train — the network fails to form good features. Training slows down — even with many epochs the model "stays in place". Sensitivity to initialization — with poor starting weights the gradient can vanish from the first steps.

How to fight it

1. Choose activation functions with "live" gradients.

Recommended activations

ReLU — max(0, x)
LeakyReLU — max(0.01x, x)
ELU — x if x >= 0, alpha(e^x - 1) if x <= 0
GELU — x * Phi(x)

Avoid in deep networks

Advantage: ReLU and its variants do not squash positive values and keep the gradient = 1 in the active region. This reduces the risk of vanishing gradients compared to sigmoid/tanh.

Sigmoid — 1/(1 + e^(-x))
Tanh — (e^x - e^(-x))/(e^x + e^(-x))

Weight initialization aware of depth

INITIALIZATION SCHEMES

Xavier (Glorot) initialization: sigma² = 2/(n_in + n_out). He initialization: sigma² = 2/n_in.

Principle: scale initial weights so that the variance of signal and gradient stays roughly constant across layers. This avoids values that are too large or too small at the start.

Normalization

BATCH NORMALIZATION

Normalizes activations across the batch, aligning their distribution and keeping them in a "working" range.

Layer Normalization normalizes activations within a layer, stabilizing gradients and accelerating training.

Effect: aligns the activation distribution, keeping it in a working range where derivatives do not vanish. In addition, BN acts as a regularizer.

Residual (skip) connections

Principle: in ResNet and Transformer, layers can pass the signal through directly, bypassing several transformations. This helps the gradient propagate without losses, even if there are layers with small derivatives inside.

y = F(x) + x  // Residual connection

Gradient clipping and scaling

Typically used for exploding gradients, but combined with other methods it helps stabilize signal propagation.

Comparison of methods

Method — When to use — Advantages — Disadvantages
ReLU activations — Deep networks, most tasks — Simplicity, efficiency, no vanishing gradients — Dying ReLU problem
Proper initialization — Always when creating new architectures — Simplicity, universality — Does not solve everything
Batch Normalization — Deep networks, CNN, MLP — Stability, faster training — Dependence on batch size
Skip connections — Very deep networks (ResNet, Transformer) — Efficiency, simple gradient flow — Architectural complexity
Gradient clipping — Unstable training — Simplicity, stability — May slow training

Practical recommendations

Step-by-step plan to fight vanishing gradients:

Start with ReLU activations — replace sigmoid/tanh with ReLU or its variants
Use proper initialization — Xavier for tanh/sigmoid, He for ReLU
Add Batch Normalization — after each layer in deep networks
Consider skip connections — for very deep architectures
Monitor gradients — apply gradient clipping when needed

Key takeaways

SUMMARY

Vanishing gradients are a fundamental problem of deep neural networks. A combination of methods works better than any single fix. Proper choice of activations is critical for stable training. Normalization and skip connections are modern standards for deep networks. Gradient monitoring helps detect problems early.