Trajectory Collection

In reinforcement learning, the agent does not receive a fixed dataset, as in supervised learning. Instead, it generates data itself by interacting with the environment. This data — trajectories — serves as the main source of information for training the policy and the critic.

Properly organized trajectory collection is critically important for:

• the quality of gradients,
• training stability,
• the model's generalization capability.

What is a trajectory in RL

A trajectory (also rollout, episode, path) is a sequence of steps in which the agent interacts with the environment:

Trajectory: τ = (s_0, a_0, r_0), (s_1, a_1, r_1), ..., (s_T, a_T, r_T)

Components of a step

Each step consists of:

• s_t — the environment state,
• a_t — the agent's action,
• r_t — the reward,
• s_{t+1} — the new state.

Additional fields sometimes stored

• the log-probability of the action (for policy gradient),
• discounted returns,
• advantage, if computed on the fly.

Why we need trajectories

Trajectories are the data on which the agent learns. From them are extracted:

• estimates of state and action values (for the critic),
• action advantage estimates (for the policy gradient),
• gradients to update the model parameters,
• statistics on behavior and training stability.

Without high-quality trajectory collection, the agent cannot learn effectively: there will be noisy gradients, overfitting, and training breakdowns.

Key questions when collecting trajectories

On-policy vs Off-policy:

• On-policy (PPO, A2C): trajectories are collected using the current policy. → data cannot be reused after an update.
• Off-policy (DQN, SAC, DDPG): you can train on data collected by another policy, using a replay buffer. → reusing data improves sample efficiency.

Full episodes vs fixed windows

• Full episodes: collected until the episode ends (done=True). → suitable for tasks with a clear termination.
• Fixed windows (n-step, truncated trajectory): the agent acts for N steps, then is cut off. → convenient for long or infinite episodes.

Collection with multiple agents (parallelization)

In algorithms like A3C or PPO (in batched mode), trajectories are collected in parallel across several environments:

• accelerates data generation,
• reduces correlation between episodes,
• provides more statistics per unit time.

Trajectory buffer structure

Typically the buffer is structured as a dictionary of numpy arrays or tensors:

buffer = {
    'states': [],
    'actions': [],
    'rewards': [],
    'values': [],
    'log_probs': [],
    'dones': []
}

After collection, the buffer is turned into a batch and fed into training. Sometimes the data is normalized, processed via GAE, or averaged.

Subtleties and problems

Sampling strongly correlated trajectories: if the agent learns from data it has just generated itself, an autocorrelation problem arises: the behavior is too uniform, with no diversity.

Solutions:

• parallel environments,
• entropy regularization,
• random start,
• off-policy buffers.

Reward hacking

If the reward is poorly formulated, the agent can find a "cheating" way to score points. This may render all collected trajectories useless, since they do not reflect the desired behavior.

Solutions:

• proper reward shaping,
• penalty terms,
• manual filtering of bad episodes.

Data imbalance

In tasks with rare rewards or discrete outcomes:

• most trajectories carry no value for training.

Solutions for imbalance

• prioritized replay sampling,
• bootstrap learning,
• early termination of bad episodes (early cutoff).

Trajectory analytics

For monitoring and debugging it is worth analyzing:

• the reward distribution,
• episode length,
• advantage density,
• action entropy (diversity),
• correlation between adjacent trajectories.

What it helps you understand

• whether the agent is overfitting,
• the degree of exploratory behavior,
• whether there are stuck regimes.

Augmentations and trajectory transformations

In some tasks one applies:

• reward normalization (zero-mean centering),
• reverse trajectories (for symmetric environments),
• re-evaluation of advantage with new value functions.

In models like Decision Transformer or Trajectory Transformer, trajectories are fed as sequences for transformer training.

Conclusion

Trajectory collection is the foundation of training in RL. This is where the agent's training experience is formed, and the efficiency of the entire process depends on the quality of this experience.

Well-organized trajectory collection:

• ensures gradient stability,
• accelerates convergence,
• allows the agent to explore the environment and adapt to it.

Proper engineering at this stage is no less important than the neural network architecture itself.