Embeddings

Embeddings are a fundamental mechanism for transforming discrete objects (words, symbols, categories) into continuous vector representations. They let neural network models work with heterogeneous data within a single mathematical space.

This article covers how embeddings work, their properties and their use in modern deep learning architectures.

Core principles

An embedding is a mapping function that transforms discrete objects into vectors of fixed dimensionality:

E: X → R^d

Where:

• X — set of discrete objects
• R^d — d-dimensional real space
• E — embedding function

Properties of embeddings

Good embeddings have the following properties:

• Semantic proximity — similar objects have similar vectors
• Arithmetic operations — vectors support meaningful operations
• Scalability — efficient handling of large vocabularies
• Trainability — parameters are optimized during training

Mathematical foundation

Embeddings can be represented as a weight matrix where each row corresponds to the vector of a specific object:

E ∈ R^(|V| × d)

Where:

• |V| — vocabulary size
• d — embedding dimensionality
• E[i] — vector for the i-th object

Lookup operation

An object's embedding is retrieved via an indexing operation:

embedding(x) = E[x]

Training embeddings

Embeddings are trained jointly with the main model via backpropagation:

∂L/∂E[x] = ∂L/∂h · ∂h/∂E[x]

Where:

• L — loss function
• h — model output
• E[x] — embedding of object x

Gradient descent

Embeddings are updated by the formula:

E[x] ← E[x] - α · ∂L/∂E[x]

Types of embeddings

Static embeddings

Pretrained vectors that are not modified during model training:

• Word2Vec
• GloVe
• FastText

Dynamic embeddings

Vectors that adapt to context and are trained jointly with the model:

• BERT embeddings
• GPT embeddings
• Task-specific embeddings

Multi-level embeddings

Complex models use hierarchical embeddings:

e_t = f(e_{t-1}, context_t)

Where:

• e_t — embedding at step t
• context_t — contextual information
• f — update function

Application to financial models

In financial markets, embeddings are used for:

• Symbols — Trading pairs — Asset classification
• Timestamps — Hours / days of the week — Seasonality
• Events — News / announcements — Sentiment analysis
• Actions — Trade operations — Agent policy

Optimizing embeddings

Various techniques are used to improve embedding quality:

• Regularization — preventing overfitting
• Dropout — randomly zeroing out elements
• Layer normalization — normalizing activations
• Weight tying — sharing weights between layers

Visualizing embeddings

Dimensionality reduction methods are used to analyze embeddings:

• t-SNE
• PCA
• UMAP
• Multidimensional scaling

Conclusion

Embeddings are a key component of modern neural network architectures, providing:

• A unified representation of heterogeneous data
• Efficient training on discrete objects
• Semantic understanding of relationships between objects
• Scalability to large vocabularies

In financial modeling, embeddings make it possible to build rich representations of market data, which improves forecasting quality and decision-making.