[Help Wanted] Incompatible log_probs shape when combining trajectories about torchgfn

[saleml]

I have never implemented a prioritized replay buffer, but your idea looks like it can be extended to storing trajectories rather than terminal states.

For example, you can use the provided ReplayBuffer class, and keep a sorted list of indices (the indices of the trajectories). Every time you add a trajectory, you sort again the list of indices (using the corresponding trajectory reward for sorting). Then you could use the sorted list of indices to subsample your replay buffer, rather than using the sample() method.

[saleml]

before trajectories.extend(offline_trajectories) seems to work but I don't know if there will be unexpected behavior downstream. It seems that log_probs needs to be padded after Trajectories.revert_backward_trajectories(). I would appreciate your insight.

If you actually don't care about the log_probs attribute of the obtained trajectories, then yeah your suggestion should work fine.

would it be better to sample forward trajectories stored in ReplayBuffer instead?

I'm not sure what you mean by that. Did you mean "to store" instead of "to sample" ? If so, note that a replay buffer works essentially with extends. The difference with what you wrote is that a ReplayBuffer object has a limited capacity. What do you think such a replay buffer would be helpful for ?

[JosephRRB]

If you actually don't care about the log_probs attribute of the obtained trajectories, then yeah your suggestion should work fine.

Alright thank you! I can set on_policy=False on the loss function so that the log_probs attribute won't be used during training, right? I was just wondering if I was converting backward trajectories into forward ones correctly. Since I'm also using them in training

I'm not sure what you mean by that. Did you mean "to store" instead of "to sample" ? If so, note that a replay buffer works essentially with extends. The difference with what you wrote is that a ReplayBuffer object has a limited capacity. What do you think such a replay buffer would be helpful for ?

I'm implementing a replay buffer where we can sample from the top highest (and lowest) reward terminal states seen so far. Its PRT from https://arxiv.org/abs/2305.07170. We need it because our rewards are very skewed.

Currently, every time I add terminal states to the buffer, I remove any duplicated terminal states and I sort them according to their corresponding rewards. So when I sample from the buffer, I only sample from the first n states and last n states. If I reach the maximum capacity, I remove states from the middle of the buffer.

My question was, instead of storing just terminal states and sampling the highest (and lowest) reward terminal states from them, can I store the trajectories themselves and find a way to sample those trajectories with the desired rewards? This is so that I wouldn't need to generate the backward trajectories from the terminal states and convert them to forward ones.

What do you think is the best way to implement such a buffer?

[JosephRRB]

Alright, let me try that out! Thank you :)