High frequency in RPMs when include action buffer in observation space can couse problems in real hardware about gym-pybullet-drones

[JacopoPan]

Hi @piratax007, apologies for the late answer.

• The buffer is something I introduced only in a second moment to make the training examples faster (and because it's a feature you see in other similar drone RL works), it's not something intended to facilitate sim2real transfer
• I would start attempting sim2real with the smoother policy first, I'm a bit skeptic about deploying a very noisy controller + a filter in real hardware.
• What I refer to as action buffer has nothing to do with SB3 per se: it is simply the concatenation of the actions commanded to the environment in the last N seconds to the observation the environment returns. Its size depends on CTRL_FREQUENCY because the faster the CTRL_FREQUENCY, the more actions will have been sent in the last N seconds.
• CTRL_FREQUENCY is the frequenty at which the policy interacts with the drone, PYB_FREQUENCY is the frequency at which Bullet is called to update the state of the simulation (it must be a multiple of CTRL_FREQUENCY).

[piratax007]

Thanks for your time and answer.

I extracted the action buffer from the observation space and training using curriculum learning the training process takes a couple of hours for a complicated task.

Now I'm trying sim2real using crazyflie 2.x and Vicon system, I have issues because the observations from Vicon come with noise and the policy was trained in a perfect environment. Any suggestions for the sim2real transfer part?

Best regards.