Risk / Reward Balance

Improving the reward function with consideration of drawdowns and trading costs.

Introduction: the need to account for drawdowns and costs

In algorithmic trading based on reinforcement learning (RL), the agent's goal is to maximize profit. However, chasing profit without considering risk can lead to strategies with dangerously high drawdowns and excessive trading costs.

Real markets involve commissions and slippage. Ignoring these factors can lead to strategies with excessive trading frequency or suboptimal position sizes. Including these factors in the reward function produces a more stable and realistic strategy.

Bonus for drawdown reduction and penalty for drawdown

When including drawdown in the reward function, the question of concrete implementation arises. The core idea is to reward the agent for reducing drawdown and to penalize exceeding the allowed drawdown threshold.

Application logic

• Reward for recovery — the agent receives a bonus for reducing drawdown, approaching new capital peaks
• Penalty for deep drawdown — when the drawdown threshold (e.g. 10%) is exceeded, a penalty is applied

The correct order of applying the bonus and penalty is critical. The recommended approach is:

# Base reward for the step
reward = profit_change

# Step 1: penalty for drawdown above threshold
if current_drawdown > DD_THRESHOLD:
    excess = current_drawdown - DD_THRESHOLD
    reward -= PENALTY_RATE * excess * volume

# Step 2: bonus for drawdown reduction
if drawdown_reduced:
    reward *= (1 + K_BONUS * dd_gain_fraction)

Implementing the "stick then carrot" logic

The concrete implementation of penalties and bonuses in the agent's reward function involves sequential processing of the base reward with consideration of drawdown and trade volume.

Reward calculation scheme

# 1. Base reward
total_base_reward = raw_pnl - commission - slippage

# 2. Update balance and compute drawdown
old_peak = self.peak_balance
prelim_balance = prev_balance + total_base_reward
new_peak = max(old_peak, prelim_balance)
drawdown = (new_peak - prelim_balance) / (new_peak + 1e-8) * 100

# 3. Penalty for exceeding drawdown threshold
excess_dd_pct = max(0.0, drawdown - self.drawdown_threshold) / 100
drawdown_penalty = excess_dd_pct * operating_volume
reward_after_penalty = total_base_reward - drawdown_penalty

After applying the drawdown penalty, the bonus for drawdown reduction is computed:

# 4. Bonus/penalty for drawdown change
dd_before = self.last_drawdown
dd_after = drawdown
dd_gain = max(0.0, dd_before - dd_after)  # drawdown decreased
dd_loss = max(0.0, dd_after - dd_before)  # drawdown increased

thr = self.drawdown_threshold + 1e-8
k_plus = self.k_bonus      # e.g. 0.5
k_minus = getattr(self, "k_penalty", 0.2)  # can be added to config

scale_up = 1 + k_plus * (dd_gain / thr)    # 1 ... 1+k_plus
scale_down = 1 - k_minus * (dd_loss / thr) # 1 ... (1-k_minus)

# Ensure non-negative multiplier
net_scale = max(scale_up * scale_down, 0.0)
reward_after_bonus = reward_after_penalty

if reward_after_bonus > 0:  # amplify only profit
    reward_after_bonus *= net_scale
else:  # but a loss can be made harsher
    reward_after_bonus *= (1 + k_minus * (dd_loss / thr))

Additionally, a bonus for reducing trade volume can be applied:

# 5. Volume decrease bonus (optional)
growth = max(prelim_balance - old_peak, 0.0)
volume_bonus = (growth if self.use_volume_decrease_bonus else 0.0) \
    * (1 / (1 + total_trade_volume / operating_volume))

total_reward = reward_after_bonus + volume_bonus
norm_total_reward = total_reward / operating_volume

After computing the reward, the agent's internal state is updated:

# Update states
self.current_balance = prelim_balance + volume_bonus  # already accounts for penalty/bonus
self.peak_balance = new_peak
self.last_drawdown = drawdown
self.max_drawdown = max(self.max_drawdown, drawdown)

Details of the symmetric penalty for drawdown growth

The symmetric penalty for drawdown growth is implemented through the k_penalty parameter, which can be added to the configuration with a default value of 0.2-0.3.

This approach provides a balance between rewarding improvement and punishing deterioration, contributing to more stable trading strategies.

Symmetric penalty for drawdown growth

The traditional approach provides a bonus for reducing drawdown, but the absence of sanctions for its growth can lead to undesirable behavior. The agent may aggressively scale up positions or hold losing trades, deepening the drawdown.

The symmetric approach assumes a penalty for drawdown growth analogous to the bonus for its reduction:

# Penalty for drawdown increase
if drawdown_increased:
    reward *= (1 - K_PENALTY * dd_loss_fraction)
    # where dd_loss_fraction = how much the drawdown grew

Accounting for turnover and slippage: nonlinear trading costs

Beyond risk management via drawdowns, accounting for trading costs is critical. The traditional approach includes a linear commission on trade volume, but real markets are characterized by nonlinear cost growth as volume increases.

Square-root impact law

Research shows that the impact of a trade on price scales according to a square-root law:

impact ~ C · sigma · sqrt(Q/V)

where:
Q — trade volume
V — typical volume (daily turnover)
sigma — volatility
C — constant

Implementing nonlinear costs in the reward function:

linear_cost = COMM_RATE * volume
impact_cost = IMPACT_COEFF * sqrt(volume_ratio)
total_cost = linear_cost + impact_cost
reward = profit_change - total_cost

Extended criteria for selecting the best model

During agent training, the "best" model is periodically saved based on certain criteria. Traditionally this criterion was total profit (PNL), but a comprehensive evaluation requires considering risk and efficiency.

Key evaluation metrics

• Sharpe ratio — measures return per unit of accepted risk (volatility of returns)
• Capital turnover — number of trades and associated costs
• Maximum drawdown — the largest decline in capital from peak to trough
• Calmar ratio — the ratio of return to maximum drawdown

Composite model selection criterion:

def is_better_model(new, best):
    # Check primary return
    if new.profit > best.profit * (1 + 0.01):
        return True

    # At equal profit compare risks
    if abs(new.profit - best.profit) < best.profit * 0.01:
        if new.max_drawdown < best.max_drawdown * 0.9:
            return True  # drawdown materially lower
        if new.turnover < best.turnover * 0.7:
            return True  # turnover significantly lower
        if new.sharpe > best.sharpe:
            return True  # better Sharpe ratio

    return False

Additional evaluation metrics

Beyond the Sharpe ratio, there are other metrics for evaluating strategy quality:

• Sortino ratio — (R - Rf) / sigma_d — Considers only downside semi-volatility, penalizes drawdowns more strongly
• Calmar ratio — Annual return / Max DD — Direct ratio of return to maximum drawdown
• Sterling ratio — (R - Rf) / (Max DD - 10%) — Treats a 10% minimum drawdown as base risk

Conclusion: the value of a comprehensive approach

Implementing the methods discussed delivers the following advantages:

• Drawdown risk control — drawdowns become "expensive" in the reward function, bringing the agent's behavior closer to that of an experienced trader who values capital preservation
• Real cost accounting — a nonlinear penalty on turnover mimics market impact, making strategies more realistic
• Improved robustness — strategies become more resilient to market shocks and changes in trading conditions
• Comprehensive quality evaluation — extended selection criteria ensure choosing strategies with optimal risk-return balance