jhxu1 / dqnflaapybird

Using DQN to Play Flappy Bird

Overview

This project encapsulates Flaapy Bird into gym form and use DDQN to train an agent how to play with DDQN.

Installation Dependencies:

• Python 2.7 or 3
• TensorFlow 1.4
• pygame
• PIL

How to train an agent ?

git clone [email protected]:AugusXJ/DQNFlaapyBird.git
cd DQNFlaapyBird
python DQN.py

The model will then start training and be saved to the dqn folder. If you need to observe the trained results, you can run the test function in the code.

Deep Q-Network Algorithm

Initialize replay memory D to size N
Initialize action-value function Q with random weights
for episode = 1, M do
    Initialize state s_1
    for t = 1, T do
        With probability ϵ select random action a_t
        otherwise select a_t=max_a  Q(s_t,a; θ_i)
        Execute action a_t in emulator and observe r_t and s_(t+1)
        Store transition (s_t,a_t,r_t,s_(t+1)) in D
        Sample a minibatch of transitions (s_j,a_j,r_j,s_(j+1)) from D
        Set y_j:=
            r_j for terminal s_(j+1)
            r_j+γ*max_(a^' )  Q(s_(j+1),a'; θ_i) for non-terminal s_(j+1)
        Perform a gradient step on (y_j-Q(s_j,a_j; θ_i))^2 with respect to θ
    end for
end for

References

[1] Mnih Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level Control through Deep Reinforcement Learning. Nature, 529-33, 2015.

[2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. NIPS, Deep Learning workshop

[3] Kevin Chen. Deep Reinforcement Learning for Flappy Bird Report | Youtube result

This work is highly based on the following repos:

1. https://github.com/yenchenlin/DeepLearningFlappyBird#references
2. https://github.com/sourabhv/FlapPyBird