coac / commnet-bicnet

CommNet and BiCnet implementation in tensorflow

CommNet-BiCnet

CommNet and BiCnet implementation in tensorflow

Training

Train CommNet using DDPG algorithm

python train_comm_net.py

Hypersearch

To find the optimal hyperparameters such as actor_lr or critic_lr, a simple grid search has been implemented. It launches multiple instances of the trainer in parallel based on the number of CPU cores.

python hypersearch.py

Guessing sum environment

It is a simple game described in the BiCnet paper for testing if the communication works. The environment implements the crucial methods of the core gym interface from OpenAI

Each agent receives a scalar sampled between [−10, 10] under a truncated Gaussian. Each agent needs to output the sum of all inputs received among the agents. An agent gets a normalized reward between [0, 1] based on the absolute difference between the sum and its output.

Results

Training CommNet in the Guessing sum env with 2 agents

commnet-bicnet's Issues

variable sharability among critic and actor

Thanks for reply, I have been busy at another project last few days, recently I get spare time.
I have noticed that at comm_net, the variables of communication part(maybe along with encoder part) are not shared between critic and actor,
I don't know whether it should be like these way in regular algorithms trained by DDPG like comm_net?

Actor loss function

Hi Coac, i really like your BicNet implementation! My goal is to run your BicNet implementation on an environment where every agent gets -1 reward for each time step it needs to finish the env. But there is a problem with your actor loss implementation, because the loss of the actor is defined as the prediction of the critic, the rewards needs to converges to zero if the agents performs perfect, isn't it?

loss_actor = -self.critic(state_batches, clear_action_batches).mean()

Can you explain to me why you implemented it this way? Also, is there a possibility that the reward doesn't converges to 0 when the Agents performs good (linke in the environment i mentioned above)?

In the file named 'comm_net.py'

At row 104, the code is ''h = tf.slice(H, [j, 0], [1, HIDDEN_VECTOR_LEN]) ''
shouldn't it be h = tf.slice(H, [0,j,0],[-1, 1, HIDDEN_VECTOR_LEN]) ?

Commnet on waterworld

I'd like to use commnet at waterworld because it seems like a good decision for agents to communicate and get a reward, but why didn't it converge in the end and not as good as maddpg