Magnitude-Aware Loss

How to use a loss function that accounts for relative accuracy instead of the absolute magnitude of the error.

The problem with classical MSE

In classical MSE all errors are measured by absolute magnitude. This produces a skew: large values with the same percentage error contribute disproportionately more to the total loss.

Example of the problem

Imagine:

Errors:

Problem: Although the relative error is the same in all cases — 50% — the last pair contributes 10,000 times more to the loss than the first. The model starts to overly "care" about large values and ignore small ones, even though they are equally bad in percentage terms.

• Prediction: [0.1, 1.0, 10.0]
• Truth: [0.2, 2.0, 20.0]
• 0.1 vs 0.2 — error 0.1 → MSE = 0.01
• 1.0 vs 2.0 — error 1.0 → MSE = 1.0
• 10.0 vs 20.0 — error 10.0 → MSE = 100.0

The idea behind Magnitude-aware Loss

Instead of penalizing the absolute difference, we bring all values to a common scale by dividing by the magnitude of the true value:

Normalization:
norm_pred = pred / (|true| + ε)
norm_true = true / (|true| + ε)

Result of normalization

After normalization all errors become comparable:

Result: All errors become comparable in scale. Now the model focuses on percentage accuracy rather than absolute magnitude.

• 0.1 / 0.2 = 0.5 — normalized value
• 1.0 / 2.0 = 0.5 — normalized value
• 10.0 / 20.0 = 0.5 — normalized value

Mathematical formula

Magnitude-aware Loss:
L_ma(y, ŷ) = MSE(y_norm, ŷ_norm)

where:
y_norm = y / (|y| + ε)
ŷ_norm = ŷ / (|y| + ε)

and ε is a small constant to avoid division by zero

Alternatively, you can use the relative error:
L_rel(y, ŷ) = MSE((y - ŷ) / (|y| + ε))

Advantages of Magnitude-aware Loss

• Uniform importance — both large and small values are equally important; the model does not ignore small but meaningful changes.
• Relative accuracy — the model is optimized for relative error, which matters in tasks with a wide range of values.
• Reduced overfitting — the risk of overfitting to large numbers is reduced, since all values are brought to a common scale.

When to apply Magnitude-aware Loss

• Forecasting prices or returns — when percentage accuracy matters more than absolute precision. For example, a 1% error matters equally whether we are predicting a price of $1 or $1000.
• Predicting physical quantities — in tasks with values across many orders of magnitude — temperatures, pressures, velocities and other metrics where relative accuracy matters.
• Time series with a variable scale — when the data has a wide range of values or is subject to seasonal swings that change the scale over time.
• Financial data — trading volumes, asset prices, returns — any metric where percentage accuracy relative to the current level matters.

Comparison with other loss functions

• MSE — error type: absolute; scale: not considered; applicability: universal
• MAE — error type: absolute; scale: not considered; applicability: universal
• Huber Loss — error type: adaptive; scale: not considered; applicability: with outliers
• Magnitude-aware Loss — error type: relative; scale: considered; applicability: wide range
• Sign-Aware Loss — error type: direction; scale: not considered; applicability: when direction matters

Practical implementation

Step-by-step rollout plan:

• Analyze the data — determine the value range and presence of variable scale
• Choose ε — typically ε = 1e-8 or ε = 1e-6 to avoid division by zero
• Implement the loss function — use normalization or the relative error formulation
• Tune hyperparameters — the learning rate may need adjustment
• Evaluate results — compare against MSE on a validation set

Use cases

• Financial time series — forecasting stock prices where the relative change matters more than the absolute price. A 2% error matters equally for a $10 stock and a $1000 stock.
• Sensor data — IoT devices produce data over a wide range — from microvolts to kilovolts. Magnitude-aware Loss provides uniform accuracy across all ranges.

Key takeaways

• Magnitude-aware Loss — a loss function that focuses on relative accuracy
• Solves the MSE problem — eliminates the skew toward large values
• Provides uniformity — all values are equally important regardless of scale
• Versatile applicability — fits tasks with a wide range of values
• Simple to implement — only requires normalization of the values

Practical recommendations

• Use ε = 1e-8 — the standard value to avoid division by zero
• Test on validation — compare against MSE to assess effectiveness
• Consider a combined approach — can be paired with other loss functions
• Monitor training — watch for convergence and stability
• Adapt to the task — additional tuning may be required for the data