Risk-sensitive reinforcement learning. v2.7's training loop is built around a single headline principle: the agent does not optimise expected return alone — it optimises the SHAPE of the loss tail. Actions whose return distribution leans heavily to the left are pushed away from even when their mean Q is competitive. Four mechanisms make this work together.
1. Predict the full return distribution, not a single number. Every critic head emits 32 quantiles of the return distribution per n-step horizon. The actor sees the entire shape — mean, spread, left tail — for every candidate action, instead of a flattened average.
2. Optimise the left tail explicitly (CVaR). The actor's training target is mean(Q) + 0.4 × CVaR₄₀%(Q): a 60% mean / 40% tail-aversion blend (cvar_alpha = 0.4, actor_cvar_aux_coeff = 0.4). The quantile-Huber κ is held small (1.0) so the loss behaves like L1 in the tails — a sharp gradient exactly where CVaR is read. Actions with a heavy left tail are penalised even if their expected return looks fine.
3. Ensemble of distinct critics + min-aggregation. Two independent Q-networks of different capacity (1024 × {2, 3} layers, dropout {0.10, 0.20}) predict the distribution separately. The actor uses min(Q1, Q2) per quantile (TD3-pessimistic, OOD-safe). Where the critics disagree — typically out-of-distribution or thin-data regions — the min pulls the value DOWN, so the actor is naturally pushed away from over-confident corners of the policy space.
4. Distribution-consistency penalty. A small q_crossing_coeff = 0.06 enforces monotonicity of the quantile distribution (it must be non-decreasing). Without it the critic collapses its quantiles together — on v2.3 we saw avg_q_spread drop 0.20 → 0.05 in a single cycle, the ∂Q/∂a gradient flatten, and the actor stop learning. Raised from 0.03 in v2.4.1 specifically to keep distributions well-spread and self-consistent.
Actor gradient horizon. actor_q_horizon_idx = 0 — only the shortest horizon (n = 4) drives the actor gradient (cleanest, lowest-variance signal). The longer-horizon heads (n = 8 / 12 / 16) still train the critic and enrich the value estimate, but stay out of the actor's update path to avoid noise contamination.
— Implementation —
Critic. Quantile Huber loss per horizon, per ensemble member. Conservative backup: min(Q1, Q2) over the independent twin networks.
Generalizer. Trained via deterministic policy gradient:
actor_loss = −mean(min(Q1, Q2)(s, a_gen))
The q_dist fed to the generalizer during training is evaluated at buffer actions — no circular dependency with a_gen.
Off-policy TD3 loop. Polyak target networks for both critics and the generalizer; target-policy noise smoothing on the buffer's actions; sign-balanced PER over UP / DOWN / FLAT pnl_signed clusters.
Data. 21 windows from January 2024 to date, covering trending regimes (+106%, +71%) and corrective ones (−57%, −29%). Tests run on separate time slices that never intersect with training.