cavalier023 / pico

An algorithm for exploiting Reinforcement Learning (RL) on Multi-agent Path Finding tasks.

PICO is an algorithm for exploiting Reinforcement Learning (RL) on Multi-agent Path Finding tasks. It is developed by the Multi-Agent Artificial Intelligence Lab (MAIL) in East China Normal University and the AI Research Institute in Geekplus Technology Co., Ltd. PICO is constructed based on the framework of PRIMAL:Pathfinding via Reinforcement and Imitation Multi-Agent Learning and focuses more on the collision avoidance rather than manual post-processing when collision occurs. Exploiting the design of decentralized communication and implicit priority in these secenarios benifits better path finding. To emphasis, more details about PICO can be found in our paper Multi-Agent Path Finding with Prioritized Communication Learning, which is accepted by ICRA 2022.

Distributed Assembly

Reinforcement learning code to train multiple agents to collaboratively plan their paths in a 2D grid world.

Key Components of PICO

• pico_training.py: Multi-agent training code. Training runs on GPU by default, change line "with tf.device("/gpu:0"):" to "with tf.device("/cpu:0"):" to train on CPU (much slower).Researchers can also flexibly customized their configuration in this file.
• mapf_gym.py: Multi-agent path planning gym environment, in which agents learn collective path planning.
• pico_testing.py: Code to run systematic validation tests of PICO, pulled from the saved_environments folder as .npy files and output results in a given folder (by default: test_result).

Installation

git clone https://github.com/mail-ecnu/PICO.git
cd PICO
conda env create -f conda_env.yml
conda activate PICO-dev

Before compilation: compile cpp_mstar code

• cd into the od_mstar3 folder.
• python3 setup.py build_ext (may need --inplace as extra argument).
• copy so object from build/lib.*/ at the root of the od_mstar3 folder.
• Check by going back to the root of the git folder, running python3 and "import cpp_mstar"

Quick Examples

pico_training.py:

episode_count          = 0
MAX_EPISODE            = 20
EPISODE_START          = episode_count
gamma                  = .95 # discount rate for advantage estimation and reward discounting
#moved network parameters to ACNet.py
EXPERIENCE_BUFFER_SIZE = 128
GRID_SIZE              = 11 #the size of the FOV grid to apply to each agent
ENVIRONMENT_SIZE       = (10,20)#(10,70) the total size of the environment (length of one side)
OBSTACLE_DENSITY       = (0,0.3) #(0,0.5) range of densities
DIAG_MVMT              = False # Diagonal movements allowed?
a_size                 = 5 + int(DIAG_MVMT)*4
SUMMARY_WINDOW         = 10
NUM_META_AGENTS        = 3
NUM_THREADS            = 8 #int(multiprocessing.cpu_count() / (2 * NUM_META_AGENTS))
# max_episode_length     = 256 * (NUM_THREADS//8)
max_episode_length     = 256
NUM_BUFFERS            = 1 # NO EXPERIENCE REPLAY int(NUM_THREADS / 2)
EPISODE_SAMPLES        = EXPERIENCE_BUFFER_SIZE # 64
LR_Q                   = 2.e-5
ADAPT_LR               = True
ADAPT_COEFF            = 5.e-5 #the coefficient A in LR_Q/sqrt(A*steps+1) for calculating LR
load_model             = False
RESET_TRAINER          = False
gifs_path              = 'gifs'
from datetime import datetime
TIMESTAMP = "{0:%Y-%m-%dT%H-%M/}".format(datetime.now())

GLOBAL_NET_SCOPE       = 'global'

#Imitation options
PRIMING_LENGTH         = 2500    #0 number of episodes at the beginning to train only on demonstrations
DEMONSTRATION_PROB     = 0.5

Then

python pico_training.py

Custom testing

Edit pico_testing.py according to the training setting. By default, the model is loaded from the model folder.

Then

python pico_testing.py

Requirements

• Python 3.4
• Cython 0.28.4
• OpenAI Gym 0.9.4
• Tensorflow 1.3.1
• Numpy 1.13.3
• matplotlib
• imageio (for GIFs creation)
• tk
• networkx (if using od_mstar.py and not the C++ version)

Citing our work

If you use this repo in your work, please consider citing the corresponding paper (first two authors contributed equally):

@InProceedings{lichen2022mapf,
  title =    {Multi-Agent Path Finding with Prioritized Communication Learning},
  author =   {Wenhao, Li* and Hongjun, Chen* and Bo, Jin and Wenzhe, Tan and Hongyuan, Zha and Xiangfeng, Wang},
  booktitle =    {ICRA},
  year =     {2022},
  pdf =      {https://arxiv.org/pdf/2202.03634},
  url =      {https://arxiv.org/abs/2202.03634},
}

License

Licensed under the MIT License.