How to use attention mechanisms to dynamically select important moments in time series and improve forecast quality.
What are time series?
WHAT ARE TIME SERIES?
When working with time series, the model receives a sequence of observations: x_1, x_2, x_3, …, x_T. Each element contains information about the state of the system at time t — for example, the price of an asset, trading volume, indicators, etc. Key question: how do we extract the most important information from this entire sequence for forecasting?
Classical approach: fixed window and convolutions
TRADITIONAL METHODS
Time-series features are traditionally aggregated through: a sliding window — take the last N points and average/maximize; RNN/LSTM/GRU — run through the sequence while keeping a hidden state; 1D-CNN — extract local patterns via convolutions. Downside: all these methods either consider only neighboring points or "mix" information uniformly without highlighting truly important moments.
The idea of attention
Attention lets the model assign each moment in time its own "importance weight".
Formally: Attention(Q, K, V) = softmax(QK^T / sqrt(d)) V
- Q — query: what we are looking for (the current forecasting context)
- K — keys: what we compare against (past states)
- V — values: what we extract (features of the states)
Softmax normalizes the weights, turning them into a probability distribution of importance.
Working intuition
INTUITION
Imagine a trader looking at a price chart over the last 100 days. For forecasting, what matters are: local extrema — peaks and troughs; spikes in volume — anomalous activity; analogous patterns in the past — recurring structures. Attention is a "gaze mechanism": it tells the model where exactly in the past to look rather than processing everything uniformly.
Types of attention in time series
- Self-attention — every moment in time looks at all the others and weights their contributions. Examples: Transformer, Informer.
- Cross-attention — a sequence of features of one type (e.g., prices) "looks at" another sequence (e.g., volumes).
- Multi-head attention — several "views" simultaneously: some focus on short-term patterns, others on long-term ones.
Advantages of attention-based aggregation
- Selecting relevant moments — the model can ignore noise and focus on key events
- Long-range dependencies — even data 200 steps back can receive a high weight if it is important
- Interpretability — attention weights can be visualized to understand what the model focused on
- Adaptivity — the model automatically adapts to different pattern types
Example: financial time series
Scenario: forecasting tomorrow's price.
Data: 60 days of price and volume history.
- RNN approach — looks at the whole window, but memory decays — events 60 days ago hardly matter.
- Attention approach — can "fire" a weight on an event 45 days ago if it resembles the current situation (e.g., a similar correction after a rally).
How aggregation works
PIPELINE
1) Extract features for each time step (LSTM, CNN, or simply a linear layer). 2) Compute attention weights between the current state and the entire history. 3) Multiply features by the weights — important moments are amplified, irrelevant ones are dampened. 4) Sum the weighted features — obtain an aggregated representation of the time series.
Comparing approaches
- Sliding window — Pros: simplicity, speed — Cons: fixed importance, information loss — Use case: simple problems
- RNN/LSTM — Pros: sequential processing, memory — Cons: vanishing gradients, training difficulty — Use case: medium complexity
- 1D-CNN — Pros: local patterns, parallelism — Cons: fixed receptive field — Use case: patterns, trends
- Attention — Pros: dynamic selection, interpretability — Cons: computational complexity — Use case: complex dependencies
Key takeaways
KEY TAKEAWAYS
Attention-based aggregation is a way to dynamically select important points in history. It improves forecast quality by focusing on relevant events. It improves model robustness against noise and irrelevant data. It provides interpretability — you can understand what the model looked at. It allows you to work with long-range dependencies (>100 steps).
Practical recommendations
RECOMMENDATIONS
Start with self-attention for baseline aggregation of temporal features. Use multi-head attention to capture different types of patterns. Consider cross-attention for multimodal time series. Visualize attention weights to interpret results. Combine with other methods for maximum effectiveness.