jaggbow / saf Goto Github PK

Integrate the MARLGrid environment into the codebase and include coordination and heterogeneity levels

Put all relevant files under the folder src/envs/marlgrid/. Once the environment code is ready, make sure it's callable from src/envs/__init__.py using the get_env function.

We still need to implement additional PPO training details to benefit from the full performance of IPPO and MAPPO [1,2]. Here are the things that should be implemented:

• Feature Pruning: Form a state by concatenating environment provided global state and agent's local observation and then prune out redundant information. This is highly environment specific so we might need to change the obs_to_state_wrapper to account for that. No change needed elsewhere.
• Value Normalization: Regress value network output to the normalized value target. This was found to help the training significantly for MAPPO
• Recurrent-MAPPO: MAPPO that operates with RNNs (GRU for example) instead of simple MLPs
• Frame stacking: Provide a stack of observations instead of only one

References:

[1] The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

[2] (https://github.com/marlbenchmark/on-policy/tree/main/onpolicy)[https://github.com/marlbenchmark/on-policy/tree/main/onpolicy]

Add the possibility to use a CNN architecture for all implemented algorithms in the code base. Appropriate reshaping might be needed, in that case, make sure runner.py is compatible with it too

The model code should be under src/policies and it should follow the same logic as in ippo.py and mappo.py.

Once it's created, create a new yaml config file for the SAF model in configs/policy/