TD3 training loop inherited from the v2.4 family — separate actor / critic encoders, TransformerEncoder symbol mixer, sign-balanced PER, Polyak targets.
Per-head quantile loss. The quantile Huber loss is computed head-by-head; the quantile axis is the same for every head: for h in range(H): loss_h = quantile_huber(pred[:, h], backup[:, h]) where pred[:, h] is (B, N, 32) for head h and backup[:, h] is that head's n_h-step target.
Shared τ grid. The τ-fractions (i + 0.5) / 32 are identical across all four heads — the same 32-quantile grid. twin-min, the q_crossing penalty and CVaR all operate along the 32-quantile axis, separately within each horizon.
Equal-nq invariant. All heads must carry the same num_quantiles (enforced by assert) — this is required for the per-quantile twin-min between paired ensemble critics to line up.
Why multiple horizons. The four heads give the actor a bias / variance spectrum of the same Q*: a 4-step head reacts fast but is bootstrap-biased; a 16-step head is closer to the Monte-Carlo return but noisier. Aggregating across horizons stabilises the policy gradient.