Huber Loss

How to use a loss function that combines the strengths of MSE and MAE for robust handling of outliers.

Why MSE struggles with outliers

Mean Squared Error squares the difference between prediction and reality. This means a single large outlier can have tens or hundreds of times more impact on the total error than many small inaccuracies.

Intuition behind Huber Loss

Huber Loss behaves differently depending on the magnitude of the error:

• If the error is small — it works like MSE, penalizing proportionally to the squared error, which is useful for precise fitting
• If the error is large — it switches to MAE mode, penalizing linearly and reducing the impact of outliers

The δ (delta) parameter

This is the boundary between a "small" and a "large" error.

• Error < δ → treated as ordinary, behaves like MSE
• Error ≥ δ → treated as an outlier, behaves like MAE

Worked example

Price forecasting:

• Error 0.5 — small: Huber Loss penalizes softly (like MSE), preserving accuracy on normal data.
• Error 3.0 — large: Huber Loss penalizes moderately (like MAE), preventing a single point from spoiling the entire model.

Mathematical formula

Huber Loss:
L_δ(y, ŷ) = {
  0.5(y - ŷ)²,        if |y - ŷ| ≤ δ
  δ|y - ŷ| - 0.5δ²,   if |y - ŷ| > δ
}

where δ is the parameter defining the boundary between the quadratic and linear regions

Advantages of Huber Loss

• Outlier robustness — the model does not "go crazy" because of isolated outliers, keeping training stable.
• High accuracy — high accuracy is preserved on normal data thanks to the quadratic region.
• Stable training — training is faster and more stable, especially on data with outliers.

Comparison with other loss functions

• MSE — outlier behavior: very sensitive; accuracy: high on normal data; implementation complexity: simple
• MAE — outlier behavior: robust; accuracy: medium; implementation complexity: simple
• Huber Loss — outlier behavior: robust; accuracy: high; implementation complexity: medium
• Smooth L1 — outlier behavior: robust; accuracy: high; implementation complexity: complex

When to use Huber Loss

• Data with outliers — when the data contains anomalous values that may distort model training.
• Financial time series — in trading, extreme price movements are common and may be outliers.
• Sensor data — IoT devices and sensors often produce noisy data with outliers.
• Medical data — biomedical measurements may contain artifacts and anomalous values.

Practical implementation

Step-by-step rollout plan:

• Analyze the data — identify the presence and nature of outliers
• Choose the δ parameter — typically δ = 1.0 or δ = 1.345 (95% efficiency for the normal distribution)
• Implement the loss function — use existing implementations or build your own
• Tune hyperparameters — the learning rate may need adjustment
• Evaluate results — compare against MSE and MAE on a validation set

Choosing the δ parameter

• δ = 1.0 — standard value, suitable for most tasks. A good balance between MSE and MAE.
• δ = 1.345 — optimal value for the normal distribution (95% efficiency). Recommended for statistical tasks.

Key takeaways

• Huber Loss — a compromise between MSE and MAE for handling outliers
• Robust to outliers — does not let anomalous points distort training
• Preserves accuracy — delivers high accuracy on normal data
• Versatile — fits a wide range of tasks with outliers
• Easy to tune — requires only a single parameter δ

Practical recommendations

• Start with δ = 1.0 — a good starting point for most tasks
• Use validation — test different δ values on a validation set
• Consider the data — δ can be reduced when outliers are strong
• Monitor training — watch for convergence and stability
• Combine approaches — pair with other outlier-handling techniques