Model Diagnostics — PROGMATIC

How to use Magnitude Correlation and Magnitude Ratio to assess forecast quality and understand model behavior in time series prediction tasks.

What is model diagnostics?

BEYOND RAW ERROR

When training a model to predict time series or any quantitative values, we need to measure not only its accuracy in terms of error, but also the quality of the shape and scale of predictions.

Two key diagnostic indicators

1. MAGNITUDE CORRELATION

Magnitude correlation — shows how synchronously predicted values move with real ones. +1: ideal positive correlation; 0: no relationship; -1: ideal negative correlation.

2. MAGNITUDE RATIO

Ratio of magnitudes — shows how well the average scale of predictions matches the average scale of real data. 1.0: ideal scale; <1: underestimation; >1: overestimation.

Magnitude Correlation: correlation of magnitudes

WHAT IT MEASURES

Magnitude Correlation shows how synchronously predicted values move with real ones. It is the key indicator for understanding whether the model correctly captures the direction and shape of changes.

Key value: high correlation means the model correctly captures relative changes — even if the scale is not yet ideal.

+1.0 — Ideal positive correlation: when the real value rises, the model predicts a rise, and vice versa.
0.0 — No relationship: the model does not capture the link between rises/falls of the real value and its forecast.
-1.0 — Ideal negative correlation: the model moves opposite to reality.

Magnitude Ratio: ratio of magnitudes

WHAT IT MEASURES

Magnitude Ratio shows how well the average scale of predictions matches the average scale of real data. This is critical for understanding whether the model correctly estimates the amplitude of changes.

Key value: correct scale is critical for practical use of the model in real tasks.

1.0 — Ideal scale: model and reality have matching amplitudes.
< 1.0 — Underestimation: the model predicts less than reality (e.g., half).
> 1.0 — Overestimation: the model predicts more than reality (e.g., 10x).

Complementary roles of the indicators

HOW THEY WORK TOGETHER

Magnitude Correlation answers for the correctness of the shape of changes, while Magnitude Ratio answers for the correctness of the scale. Together they give a fuller picture of model quality.

Combined analysis example: a model can excellently "guess" the direction and shape of fluctuations (high correlation) while being too cautious or too aggressive in amplitude (imperfect ratio).

Combinations of correlation and scale

HIGH CORRELATION + CORRECT SCALE

Excellent model: correctly captures both direction and amplitude of changes.

HIGH CORRELATION + WRONG SCALE

Good foundation: the model understands patterns but needs scale tuning.

LOW CORRELATION + CORRECT SCALE

Random coincidence: the scale may be accidental, the logic needs work.

LOW CORRELATION + WRONG SCALE

Problem model: needs serious rework or retraining.

Mathematical formulas

Magnitude Correlation:
Corr = Cov(ŷ, y) / (σ_ŷ × σ_y)

where:
ŷ — predicted values
y — real values
σ_ŷ, σ_y — standard deviations

Magnitude Ratio:
Ratio = σ_ŷ / σ_y

where:
σ_ŷ — std of predictions
σ_y — std of real data

Practical examples

Example 1: Conservative model. Results: Magnitude Correlation = 0.85, Magnitude Ratio = 0.6. Interpretation: the model captures direction well (high correlation) but predicts changes 1.67x smaller than real (Ratio < 1). The model is too conservative.

Example 2: Aggressive model. Results: Magnitude Correlation = 0.78, Magnitude Ratio = 1.8. Interpretation: the model captures direction (good correlation) but predicts changes 1.8x larger than real. The model is too aggressive.

Example 3: High-quality model. Results: Magnitude Correlation = 0.92, Magnitude Ratio = 1.05. Interpretation: an excellent model — highly correlated with reality and with nearly ideal scale (only 5% deviation).

When to use these indicators

TIME SERIES FORECASTING

Assessing the quality of models predicting prices, volumes, returns, and other financial metrics.

REGRESSION TASKS

Any regression task where it matters to understand not just accuracy but also the quality of scale.

MODEL COMPARISON

Objective comparison of different architectures and approaches to the task.

MODEL DEBUGGING

Understanding exactly where the model errs — in direction or in scale.

Practical recommendations

How to interpret results:

Start with correlation — if it is low (< 0.5), the model does not capture patterns
Assess scale — if Ratio is far from 1.0, the model misjudges amplitude
Prioritize problems — first fix low correlation, then scale
Monitor during training — track changes in both indicators
Use for validation — include these metrics in cross-validation

Key takeaways

SUMMARY

Magnitude Correlation reflects correctness of shape and direction. Magnitude Ratio reflects correctness of scale and amplitude. Together they form complementary metrics that give the full picture. They are critical for real-world model deployment and serve as a diagnostic tool for pinpointing where the model errs.

Philosophy

BOTH MATTER

A high-quality model needs both high correlation and correct scale. One without the other does not yield a complete result. These indicators move you from simply measuring error to a deep understanding of how the model interprets and reproduces patterns in the data.