Residual Connections

How residual connections solve the vanishing gradient problem and improve the training of deep neural networks by creating direct paths for information flow.

What are residual connections?

The problem with deep networks

The solution: residual connections

How it works

Instead of simply passing data through a layer, we do:

output = F(x) + x

where:

• x — input to the layer
• F(x) — the transformation applied by the layer
• + x — direct pass of the original input to the output

Intuitive understanding

Visualizing a residual block

Input (x)
   ↓
[Transformation F(x)]   [Direct path: + x (skip connection)]
   ↓
Output: F(x) + x

Advantages of residual connections

• Training stability — even very deep networks (hundreds of layers) can be trained stably; residual connections prevent vanishing gradients
• Fast convergence — fewer issues with vanishing gradients mean faster and more stable training
• Modularity — can be combined with any architectures: CNNs, RNNs, Transformers and others

Mathematical intuition

Why this works:

Gradient: ∂L/∂x = ∂L/∂(F(x) + x) = ∂L/∂F(x) + ∂L/∂x
The gradient can flow directly through the skip connection without decaying.

Optimization: the network learns H(x) = F(x) + x, where F(x) = H(x) - x
Instead of the complex transformation H(x), the network learns the residual F(x).

Practical examples

• ResNet (Residual Networks) — Problem: traditional CNNs with 20+ layers showed worse results than shallower ones. Solution: adding residual connections enabled training networks with 152+ layers and achieving the best results on ImageNet.
• Transformer Architecture — Problem: deep Transformers suffer from vanishing gradients. Solution: residual connections around each sublayer (attention + feed-forward) ensure stable training.
• RNN/LSTM — Problem: long sequences lead to vanishing gradients. Solution: residual connections between time steps help preserve information over long distances.

Implementation in code

Simple Residual Block:
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, padding=1)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, padding=1)
        self.relu = nn.ReLU()

    def forward(self, x):
        residual = x
        out = self.relu(self.conv1(x))
        out = self.conv2(out)
        out += residual  # Residual connection
        return self.relu(out)

Variants of residual connections

• Identity — F(x) + x — Same dimensions — Simplicity, efficiency
• Linear — F(x) + Wx — Different dimensions — Dimension flexibility
• Dense — F(x) + x1 + x2 + ... — DenseNet — Multiple paths
• Highway — g(x)F(x) + (1-g(x))x — Highway Networks — Adaptive weights

When to use residual connections

• Deep networks — when the number of layers exceeds 10–15, residual connections become critically important
• Gradient problems — when signs of vanishing or exploding gradients appear
• Complex architectures — in Transformers, deep CNNs, RNNs with long sequences
• Experimental models — when developing new architectures to ensure training stability

Practical recommendations

Implementation tips:

• Use for deep networks — starting at 10+ layers, residual connections become useful
• Handle different dimensions — use linear projections for skip connections
• Place activations correctly — typically ReLU is applied after the addition
• Monitor gradients — residual connections should stabilize their flow
• Experiment with the architecture — different types of residual connections suit different tasks

Key takeaways

Bridge analogy