The later the model training, the slower it is about lightzero

[puyuan1996]

• Hello, regarding the time issue you encountered with the running of the cartpole_muzero_config.py configuration, where the runtime is reasonable at 20,000 steps but significantly increases at 100,000 steps, we have analyzed the possible reasons and offer the following suggestions:

  • Firstly, we recommend that you check the metrics recorded in TensorBoard, such as collector/collect_time, to confirm whether the time required for each data collection increases as the number of environment steps increases. This will help to rule out any errors in your time measurement.

  • Secondly, in the CartPole task, the maximum number of steps for each game episode is limited to 200 steps. By default, the capacity of the replay buffer is set to 1,000,000. As the number of game steps accumulates, the amount of data that the MuZero algorithm needs to store will gradually increase, which will inevitably take up more memory resources and disk space and may impact the performance of the program. Additionally, after reaching 20,000 steps, you should check to see if any other resource-intensive programs have been initiated in the system. This could lead to resource contention and thus affect the efficiency of the MuZero algorithm. Therefore, we suggest monitoring the usage of the CPU, GPU, and storage devices to determine if there are any resource bottlenecks.

• In our local environment and cluster tests, we usually do not encounter this time issue mentioned above. Therefore, a detailed investigation of resource usage and potential program conflicts is likely key to solving the problem you are facing. Best Wishes!