Optimisation Methods

Advanced reinforcement learning algorithms for optimizing trading strategies.

Reinforcement learning in trading

Reinforcement Learning (RL) is a powerful approach to optimizing trading strategies, where the agent learns to make decisions under the uncertainty of financial markets.

The key challenge of RL in trading is to find the optimal balance between return and risk, using various metrics for evaluating strategy quality.

Train Index

The train index is a metric that determines whether the current model has become better or worse compared to previous states. The traditional approach uses a simple formula:

train_idx = (1 + dpnl) / (1 + dd_pct) - 1

Where:

• dpnl — return (PNL / initial_balance)
• dd_pct — maximum drawdown in the range [0, 1]

However, this approach has limitations: with an equal index, the model is not saved even if it earned more under proportionally equal risk.

Linear Bonus to profit

The first method adds a proportional coefficient that allows selecting the state with higher return at proportionally equal risk.

def train_index_linear(dpnl: float,           # PNL / initial_balance
                       dd_pct: float,         # max_dd  in  [0..1]
                       thr: float = .10,      # drawdown_threshold 10 %
                       k:   float = .30):     # bonus "weight"
    """
    • base = (1+dpnl)/(1+dd_pct) - 1          (as before)
    • bonus = k·dpnl·max(0, 1 - dd_pct/thr)   (0 if dd above threshold)
    """
    base  = (1.0 + dpnl) / (1.0 + dd_pct) - 1.0
    bonus = k * dpnl * max(0.0, 1 - dd_pct / thr)
    return base + bonus

Behavior: when dd → 0 the bonus = k·dpnl; at dd = thr the bonus = 0; beyond that the index matches the base value.

Scaled Profit premium on the numerator

The second method amplifies the formula's numerator proportionally to the strategy's safety.

def train_index_scaled(dpnl: float, dd_pct: float,
                       thr=.10, k=.50):
    """
    • profit↑               -> numerator is amplified
    • risk above thr        -> no amplification
    """
    scale = 1 + k * max(0.0, 1 - dd_pct/thr)     # in range [1, 1+k]
    return (1.0 + dpnl * scale) / (1.0 + dd_pct) - 1.0

Behavior: a safe (low dd) variant gets a multiplier less than or equal to 1+k applied directly to the formula's numerator.

Smooth S-shaped Sigmoid Bonus

The third method uses a sigmoid function for a smooth transition between states.

from math import exp

def train_index_sigmoid(dpnl, dd_pct, thr=.10, k=.40, sharp=15):
    """
    • bonus weighs k·dpnl
    • activated by a smooth sigmoid:  wild->0  safe->~k
    """
    # sigma(dd) ~ 1 while dd is below thr; quickly ->0 when dd->thr
    sigma = 1 / (1 + exp( sharp * (dd_pct - thr)))
    base  = (1+dpnl)/(1+dd_pct) - 1
    bonus = k * dpnl * sigma
    return base + bonus

Behavior: delivers practically the full bonus when dd is well below the threshold, and removes it along an "S-curve".

Improved Sortino-like index

The fourth method penalizes only the "harmful" portion of drawdown above the threshold.

def train_index_sortino(dpnl, dd_pct, thr=.10):
    """
    • penalize not the entire drawdown, but only the "harmful" part above thr
    """
    downside = max(0.0, dd_pct - thr)
    return (1.0 + dpnl) / (1.0 + downside) - 1.0

Behavior: if risk is less than or equal to thr, denominator = 1, so the index equals pure return; once dd exceeds thr, we get the familiar risk-adjusted form.

Comparison of methods

Each method solves the task of "at equal risk, choose the more profitable variant" in its own way, providing different approaches to optimization:

• Linear Bonus — a simple and clear approach with linear decay
• Scaled premium — amplifies the numerator proportionally to safety
• S-shaped bonus — smooth transition using a sigmoid
• Sortino-like — penalizes only the harmful portion of risk