[Question] Does hyperparameter tuning support custom vectorized environments? about rl-baselines3-zoo

[araffin]

Looking at the source code, it seems like it could be done by adding an if/else in

In your case, the best is probably to fork the RL Zoo to adapt it to your needs (you can still install it as an editable package so you don't have to integrate it in your codebase).
gym.make() is supposed to return a gym.Env, not a VecEnv.

Curious if this might break things down the line, and/or if there is an already built solution I'm missing? (I'd rather not have to integrate the entire rl_zoo3 repo in my project for cleanliness' sake)

We have something similar for a tentative PR with envpool integration: #355

[antoinedang]

Looking at the source code, it seems like it could be done by adding an if/else in

In your case, the best is probably to fork the RL Zoo to adapt it to your needs (you can still install it as an editable package so you don't have to integrate it in your codebase). gym.make() is supposed to return a gym.Env, not a VecEnv.

Curious if this might break things down the line, and/or if there is an already built solution I'm missing? (I'd rather not have to integrate the entire rl_zoo3 repo in my project for cleanliness' sake)

We have something similar for a tentative PR with envpool integration: #355

Sounds good, thanks for letting me know. I'll fork and make the changes.

In case anyone else has the same question, I'll be updating the code here:
https://github.com/mcgill-robotics/Humanoid-rl-baselines3-zoo

[antoinedang]

Looking at the source code, it seems like it could be done by adding an if/else in

rl-baselines3-zoo/rl_zoo3/exp_manager.py

Line 622 in aa38145

env = make_vec_env(

such that make_vec_env is not called if the environment is already a vec_env, and instead we just set env = make_env(**env_kwargs)? More specifically, replace lines 622-632 with:

if self._hyperparams.get("env_is_vectorized", False):
        env = make_env(num_envs=n_envs, **env_kwargs)
else:
        env = make_vec_env(
            make_env,
            n_envs=n_envs,
            seed=self.seed,
            env_kwargs=env_kwargs,
            monitor_dir=log_dir,
            wrapper_class=self.env_wrapper,
            vec_env_cls=self.vec_env_class,  # type: ignore[arg-type]
            vec_env_kwargs=self.vec_env_kwargs,
            monitor_kwargs=self.monitor_kwargs,
        )

Curious if this might break things down the line, and/or if there is an already built solution I'm missing? (I'd rather not have to integrate the entire rl_zoo3 repo in my project for cleanliness' sake)

[araffin]

Maybe, could you do a pr to update the docs?

[antoinedang]

Maybe, could you do a pr to update the docs?

Sure, however I'm new to the repo so I'm not sure the standards / where to do this. What exactly should I update and with what information? Should I do something along the lines of "If your custom environment implements the Stable Baselines 3 VecEnv class, you will have to update the source code (see issue [....])." in https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/docs/guide/custom_env.rst?

[araffin]

What exactly should I update and with what information?
https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/docs/guide/custom_env.rst

yes, this file.

with an explanation/link (link to this issue) on what to do when you have VectorEnv that are not gym.Env.

Something like https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#sb3-and-procgenenv