Attention Mechanisms

Modern decision-making problems in high-dimensional and unstable environments — such as financial markets — require models capable of adaptively selecting relevant information from a stream of data and using it to form optimal behavior. One of the most effective mechanisms for this task is the attention mechanism, which lies at the heart of Transformer-style architectures.

This article examines how attention enables context-dependent information selection needed to optimize an agent's behavior under uncertainty. In particular, attention is interpreted as a generalized filtering and focusing mechanism that addresses the problem of information relevance under resource constraints.

Attention: formal description

Attention is a module inside Transformer architectures that allows a model to aggregate information from an input sequence in a weighted manner. The base scaled dot-product attention formula has the following form:

Attention(Q, K, V) = softmax((QK^T) / √d_k) V

Where:

• Q ∈ R^(n×d) — matrix of queries
• K ∈ R^(n×d) — matrix of keys
• V ∈ R^(n×d) — matrix of values
• d_k — dimensionality of the keys

Encoder architecture: overall structure

Each encoder consists of N identical blocks (typically N = 6), each containing two main subsystems:

• Multi-head Self-Attention
• Fully connected feedforward block

Schematic of one encoder block:

Input →
    +------------------------+
    | Multi-Head Attention   |
    | + Residual + LayerNorm |
    +------------------------+
               ↓
    +------------------------+
    | Feedforward (MLP)      |
    | + Residual + LayerNorm |
    +------------------------+
               ↓
         Output Representation

Each layer includes a residual connection and layer normalization, which ensures gradient stability and accelerates training.

Residual connection: Output = x + Sublayer(x)
Layer Normalization: Norm(x) = (x - μ) / √(σ² + ε)

Input data: positional encoding

Since the Transformer has no built-in representation of order (unlike an RNN), positional encoding is used and added to the input embeddings.

This allows the model to take into account both the relative and absolute position of elements.

X = Embedding(T) + PositionalEncoding(P)

Where:

• T ∈ N^L — sequence of token indices
• X ∈ R^(L×d) — input to the Transformer
• P ∈ R^(L×d) — positional codes

Thus, every input token receives:

• a semantic representation via the embedding
• a position in the sequence via the positional component

Important: all subsequent processing in the model takes place in a continuous space — after the embedding layer, the model no longer operates on discrete symbols.

Multi-Head Self-Attention

This is the central mechanism of the encoder. It allows each token to extract information from all other tokens with respect to context.

Each head learns to highlight its own type of relationship between tokens (for example, local dependencies, long-range connections, etc.).

Feedforward layer

After the attention block, each token passes through an independent fully connected neural network.

This increases the expressive power of the model through a nonlinear transformation of the token's local context.

Residual + LayerNorm

Each main block (attention and feedforward) is wrapped with the following mechanisms:

• Residual connection — allows gradients to pass through directly
• Layer normalization — stabilizes the distribution of activations

Embeddings as a generalized way of representing any objects

Although embeddings have traditionally been applied to words, their scope is significantly broader:

• Text — Tokens / words
• Time series — Events / features
• Tables — Categories / flags
• Graphs — Nodes / edges
• Images — Patches / regions
• Agent behavior — Actions / states
• Products/users — Identifiers

Each embedding is a way of translating structure and semantics into the space of linear algebra, where optimization becomes possible.

Conclusion

Embeddings are the foundation of all modern neural network models. They allow you to:

• unify discrete information with continuous mathematics
• structure data in a learnable space
• ensure scalability and flexibility of the model
• build a unified architecture for diverse inputs

In Transformer-style architectures, embeddings are not merely an input transformer but a part of the overall optimization system, trained jointly with attention, feedforward and other components.