TD3 with deterministic policy and target-policy noise smoothing.
Branch loss. Per-symbol critic provides Q_per_sym(s, a); critic_loss is standard TD3 twin-Q distributional Huber on this per-symbol signal alone. Actor loss is −E[min_i(Q_per_sym_i(s, π(s)))] over symbols — no portfolio-level term in the gradient.
V_global loss. TD on the reward_shape stream (drawdown / portfolio shaping signal), with twin V heads for a conservative target. Hard gradient barrier on input — V_global gradients do not flow into branch features or the critic_encoder.
Polyak target update. Runs independently on every module: actor_encoder, critic_encoder, mini_mixer, per-sym critics, and the V_global observer.
CEM / MPC inference. V_global is removed from the action-ranking objective — under detach() it is action-independent, i.e. constant per state, and would only inject noise into the ranking.
State-dict migration. Skip-head → linear collapse, drop legacy actor_adapter weights, drop legacy QGMLP keys. Migration logic preserves the trained per-sym critic and the v2.3 separate-encoder weights.
Sampler / replay. Random-k KMeans clustering with per-cycle seed jitter. Sign-balanced PER over UP / DOWN / FLAT pnl_signed clusters. Two reward normalizers — z-score EMA on the vec (PnL) and shape (reward_shape) streams.