vy007vikas / pytorch-actorcriticrl

PyTorch implementation of DDPG algorithm for continuous action reinforcement learning problem.

PyTorch-ActorCriticRL

PyTorch implementation of continuous action actor-critic algorithm. The algorithm uses DeepMind's Deep Deterministic Policy Gradient DDPG method for updating the actor and critic networks along with Ornstein–Uhlenbeck process for exploring in continuous action space while using a Deterministic policy.

DDPG

DDPG is a policy gradient alogrithm, that uses stochastic behaviour policy for exploration (Ornstein-Uhlenbeck in this case) and outputs a deterministic target policy, which is easier to learn.

Policy Estimation (Actor)

Actor Network consists of a 3-layer neural network taking into input the state (s) and outputs the action (a) which should be taken denoted by Pi(s).

Policy Evaluation (Critic)

Critic Network consists of a 3-layer neural network taking into input the state (s) and correspoding action (a) and outputs the state-action value function denoted by Q(s,a).

Actor Optimization

The policy is optimized by minimizing the loss :- sum ( -Q(s,a) ).

Critic Optimization

The critic is optimized by minimzing the loss :- L2( r + gamma*Q(s1,Pi(s)) - Q(s,a) ).

Soft Updates

The above updates however don't tend to converge according to DeepMind's paper and they hence use soft policy updates by maintaing a target actor and critic whose weights are updated after above optimizations as follows :-

target_actor = beta*actor + (1-beta)*target_actor
target_critic = beta*critic + (1-beta)*target_critic

where beta = 0.001

Performance of DDPG on OpenAI Envs

Pendulum-v0

Below is the performance of the model after 70 episodes. Full Video

BiPedalWalker-v2

Below is the performance of the model after 900 episodes. Full Video

References

• DDPG paper by DeepMind - DeepMind's DDPG paper
• DDPG blog by penami4911 - A very nicely explained blog with code in Tensorflow
• A3C paper by DeepMind - DeepMind's A3C paper
• A3C blog by yanpanlau - A3C explained nicely along with Ornstein-Uhlenbeck expoloration

pytorch-actorcriticrl's Issues

mul() received an invalid combination of arguments

Traceback (most recent call last)
36 state = np.float32(observation)
37
---> 38 action = trainer.get_exploration_action(state)
39 # if _ep%5 == 0:
40 # # validate every 5th episode

.../train.py in get_exploration_action(self, state)
62 """
63 state = Variable(torch.from_numpy(state))
---> 64 action = self.actor.forward(state).detach()
65 new_action = action.data.numpy() + (self.noise.sample() * self.action_lim)
66 return new_action

.../model.py in forward(self, state)
97 action = torch.tanh(self.fc4(x))
98
---> 99 action = action * self.action_lim
100
101 return action

TypeError: mul() received an invalid combination of arguments - got (numpy.int64), but expected one of:

• (Tensor other)
  didn't match because some of the arguments have invalid types: (numpy.int64)
• (Number other)
  didn't match because some of the arguments have invalid types: (numpy.int64)

Acknowledgment

I am writing to thank you for what you write in the README, which helps me totally understand the idea of DDPG. Thanks a lot

FileNotFoundError: [Errno 2] No such file or directory: './Models/0_actor.pt'

Any suggestions with this error: ?

Traceback (most recent call last):
File "main.py", line 72, in
trainer.save_models(_ep)
File "/home/sayomakinwa/.mujoco2/mujoco-py/PyTorch-ActorCriticRL/train.py", line 115, in save_models
torch.save(self.target_actor.state_dict(), './Models/' + str(episode_count) + '_actor.pt')
File "/home/sayomakinwa/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 219, in save
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/home/sayomakinwa/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 142, in _with_file_like
f = open(f, mode)
FileNotFoundError: [Errno 2] No such file or directory: './Models/0_actor.pt'

RuntimeError: matrices expected, got 1D, 2D tensors

Traceback (most recent call last):
  File "/Users/tom/PycharmProjects/PyTorch-ActorCriticRL/main.py", line 38, in <module>
    action = trainer.get_exploration_action(state)
  File "/Users/tom/PycharmProjects/PyTorch-ActorCriticRL/train.py", line 64, in get_exploration_action
    action = self.actor.forward(state).detach()
  File "/Users/tom/PycharmProjects/PyTorch-ActorCriticRL/model.py", line 94, in forward
    x = F.relu(self.fc1(state))
  File "/Library/anaconda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/anaconda/lib/python2.7/site-packages/torch/nn/modules/linear.py", line 54, in forward
    return self._backend.Linear()(input, self.weight, self.bias)
  File "/Library/anaconda/lib/python2.7/site-packages/torch/nn/_functions/linear.py", line 10, in forward
    output.addmm_(0, 1, input, weight.t())
RuntimeError: matrices expected, got 1D, 2D tensors at /Users/soumith/miniconda2/conda-bld/pytorch_1493756739997/work/torch/lib/TH/generic/THTensorMath.c:1232

Wrong training ?

in train.py

loss_critic = F.smooth_l1_loss(y_predicted, y_expected)
self.critic_optimizer.zero_grad()
loss_critic.backward()
self.critic_optimizer.step()

Is it correct to set the gradients in the optimizer to zero after calculating the loss or should it be the other way around i.e. setting all gradients to zero, then taking the loss and performing an update?