xiaowei-hu / pysc2-agents

This is a simple implementation of DeepMind's PySC2 RL agents.

PySC2 agents

This is a simple implementation of DeepMind's PySC2 RL agents. In this project, the agents are defined according to the original paper, which use all feature maps and structured information to predict both actions and arguments via an A3C algorithm.

Requirements

• PySC2 is a learning environment of StarCraft II provided by DeepMind. It provides an interface for RL agents to interact with StarCraft II, getting observations and sending actions. You can follow the tutorial in PySC2 repo to install it.

pip install s2clientprotocol==1.1
pip install pysc2==1.1

• Python packages might miss: tensorflow and absl-py. If pip is set up on your system, it can be easily installed by running

pip install absl-py
pip install tensorflow-gpu

Getting Started

Clone this repo:

git clone https://github.com/xhujoy/pysc2-agents
cd pysc2-agents

Testing

• Download the pretrained model from here and extract them to ./snapshot/.

• Test the pretrained model:

python -m main --map=MoveToBeacon --training=False

• You will get the following results for different maps.

	MoveToBeacon	CollectMineralShards	DefeatRoaches
Mean Score	~25	~62	~87
Max Score	31	97	371

Training

Train a model by yourself:

python -m main --map=MoveToBeacon

Notations

• Different from the original A3C algorithm, we replace the policy penalty term with epsilon greedy exploration.
• When train a model by yourself, you'd better to run several times and choose the best one. If you get better results than ours, it's grateful to share with us.

Licensed under The MIT License.

pysc2-agents's Issues

What does the mask mean?

In the a3c_agent.py, what does the line 59 - 72 mean?

I can't understand what's the purpose of

self.valid_spatial_action
self.spatial_action_selected
self.valid_non_spatial_action
self.non_spatial_action_selected
self.value_target

why text the the trained model ，the agent don't move all the time!

i test author's model and my model (trained),but the the agent don.t move and i can operate the game incredibly.

Question about your logic to pick target

I just wonder why the logic for picking a target on the screen is like picking a point yourself(by the network) and then a rectangular which has random height and random width and locates close to the point will be the 'target', is this the best way to model this? Did you adapted it from other's code?

Value loss function selection

Regarding your a3c_agent.py

# Compute losses, more details in https://arxiv.org/abs/1602.01783
# Policy loss and value loss
action_log_prob = self.valid_spatial_action * spatial_action_log_prob + non_spatial_action_log_prob
advantage = tf.stop_gradient(self.value_target - self.value)
policy_loss = - tf.reduce_mean(action_log_prob * advantage)
value_loss = - tf.reduce_mean(self.value * advantage)

Shouldnt the value loss just be MSE? ie
value_loss = tf.reduce_sum( tf.square( self.value_target - self.value ) )
Also why do you use the mean instead of sum?

Tensor shape not matching

Hi,

I'm getting this error when I try to run the basic pre-trained agent available for download.

The call stack has my python version, my GPU, etc.

I'm using the newest GIT version of pysc2, which I think means my pysc2 and s2clientprotocol version are 1.2 instead of 1.1.

C:\Users\Rick\Desktop\pysc2-agents>python -m main --map=MoveToBeacon --training=false
W0203 14:01:37.877619 1860 tf_logging.py:118] From C:\Users\Rick\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\clip_ops.py:110: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
2018-02-03 14:01:38.338341: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-02-03 14:01:38.600618: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.64GiB
2018-02-03 14:01:38.600763: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
I0203 14:01:40.828442 1860 tf_logging.py:110] Restoring parameters from ./snapshot/MoveToBeacon/fcn\model.pkl-100001
Traceback (most recent call last):
File "C:\Users\Rick\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call
return fn(*args)
File "C:\Users\Rick\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn
status, run_metadata)
File "C:\Users\Rick\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [5,5,42,16] rhs shape= [5,5,23,16]
[[Node: save/Assign_34 = Assign[T=DT_FLOAT, _class=["loc:@sconv1/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](sconv1/weights/RMSProp, save/RestoreV2_34)]]

Code out of sync with newer Psysc2

It looks like there were some breaking changes in the latest release of psysc2. When I try to run the code, I get an error:

File "D:\Dev\StarcraftAI\Agents\xhujoy\pysc2-agents\main.py", line 42, in
flags.DEFINE_enum("agent_race", None, sc2_env.races.keys(), "Agent's race.")
AttributeError: module 'pysc2.env.sc2_env' has no attribute 'races'

"races" is now called Race. Similar issue with "difficulties" vs. Difficulty.

Select action from policy network

at a3c_agent.step,
action is chosen by act_id = valid_actions[np.argmax(non_spatial_action[valid_actions])]

However I think they should be chosen randomly by their probability because non_spatial_action and spatial_action value refers policy value.
(check this post https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2)

By the way, it's still not clear when to mask invalid actions. (before soft-max? after soft-max?)

System resources issues

Hi @xhujoy,

thanks for creating and sharing this code, I immensely appreciate it!

I have some questions though. On what system have you been running the simulations?
I run it on a desktop with configuration: i5-6500 (4 cores), 8 GB RAM and GTX-1060 6GB,
because so far I don't have access to decent servers to run it on.
however, for running the original 16 threads my setup seems largely insufficient.

1. The first clear problem is that I don't have a 16 core CPU, so when running more than 3-4 parallel
   threads, the simulation is really slow. Is there some way to make running more threads on a quadcore more feasible?

2. The second issue is RAM, it seems that the program is using a lot of RAM and when the simulation
   runs it just takes up more and more RAM. Is this expected behavior? Or are there some resources
   that could be freed during the runtime but are not?

3. The third issue is with GPU VRAM when the simulation is started, TF allocates 4GB of memory on the GPU, however as the simulation progresses, after some time TF starts allocating more and more VRAM until it eventually crashes the whole simulation because it consumes all GPU memory. Once again, is this an inevitable behavior or is it possible to turn something off so that it just uses the memory that is allocated on start-up?

All of these issues have to do with the system resources of my PC and the fact that the A3C algorithm
requires running a lot of instances in parallel. Is it hopeless or is there some way, to make the A3C algorithm work on a lower-tier machine? It seems to me that in my case it is not optimal by design
because it is parallel and actually requires many threads in order to approximate drawing i.i.d. samples during training (if I understood the algorithm idea correctly). So that means that running the simulation with 3 or 4 parallel threads should bear much worse results than running a 16 thread one because the samples will be a lot more correlated.
Once again, am I doomed to either, getting access to a better machine or switching to DeepQ learning or is there some way in which it is possible to efficiently use the A3C algorithm even on a lesser machine?

Thanks for your answer in advance!

Cannot start training successfully

Hi, @XHUJOY

Thanks for sharing your cool work!

I ran into it and quickly deployed the running environment as README.

Unfortuately, I cannot start training successfully right now :(

So, could you help me to figure it out? Thanks in advance. :-)

First, I downloaded linux version of SC2 from here, the ladder maps, and the mini maps, and unzip them accordingly.

Then, I did successfully downloaded and installed pysc2. When typing python -m pysc2.bin.agent --map Simple64, following messages flies by:

Version: B56787 (SC2.3.17)
Build: Aug 25 2017 15:00:49
Command Line: '"/opt/share0/data/SC2/StarCraftII/Versions/Base56787/SC2_x64" -listen 127.0.0.1 -port 16333 -dataDir /opt/share0/data/SC2/StarCraftII/ -tempDir /tmp/sc-RJxxSV/ -displayMode 0'
Starting up...
Startup Phase 1 complete
Startup Phase 2 complete
Creating stub renderer...
Listening on: 127.0.0.1:16333 (16333)
Startup Phase 3 complete. Ready for commands.
Requesting to join a single player game
Configuring interface options
Configure: raw interface enabled
Configure: feature layer interface enabled
Configure: score interface enabled
Configure: render interface disabled
Entering load game phase.
Launching next game.
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Game has started.
Sending ResponseJoinGame
Performing a full restart of the game.
Launching next game.
Next launch phase started: 2
Next launch phase started: 3
Next launch phase started: 4
Next launch phase started: 5
Next launch phase started: 6
Next launch phase started: 7
Next launch phase started: 8
Game has started.
RequestQuit command received.
Closing Application...
WARNING:root:Terminating attempt 0...
WARNING:root:Terminated.
   0/no_op                                              ()
   1/move_camera                                        (1/minimap [64, 64])
   2/select_point                                       (6/select_point_act [4]; 0/screen [84, 84])
   3/select_rect                                        (7/select_add [2]; 0/screen [84, 84]; 2/screen2 [84, 84])
   4/select_control_group                               (4/control_group_act [5]; 5/control_group_id [10])
   5/select_unit                                        (8/select_unit_act [4]; 9/select_unit_id [500])
 230/Effect_Spray_screen                                (3/queued [2]; 0/screen [84, 84])
 264/Harvest_Gather_screen                              (3/queued [2]; 0/screen [84, 84])
  12/Attack_screen                                      (3/queued [2]; 0/screen [84, 84])
 331/Move_screen                                        (3/queued [2]; 0/screen [84, 84])
 332/Move_minimap                                       (3/queued [2]; 1/minimap [64, 64])
 333/Patrol_screen                                      (3/queued [2]; 0/screen [84, 84])
 334/Patrol_minimap                                     (3/queued [2]; 1/minimap [64, 64])
  13/Attack_minimap                                     (3/queued [2]; 1/minimap [64, 64])
 274/HoldPosition_quick                                 (3/queued [2])
 451/Smart_screen                                       (3/queued [2]; 0/screen [84, 84])
  52/Build_Extractor_screen                             (3/queued [2]; 0/screen [84, 84])
 452/Smart_minimap                                      (3/queued [2]; 1/minimap [64, 64])
 453/Stop_quick                                         (3/queued [2])
   6/select_idle_worker                                 (10/select_worker [4])
   9/select_larva                                       ()
 335/Rally_Units_screen                                 (3/queued [2]; 0/screen [84, 84])
 336/Rally_Units_minimap                                (3/queued [2]; 1/minimap [64, 64])
 343/Rally_Workers_screen                               (3/queued [2]; 0/screen [84, 84])
 344/Rally_Workers_minimap                              (3/queued [2]; 1/minimap [64, 64])
 467/Train_Drone_quick                                  (3/queued [2])
 168/Cancel_Last_quick                                  (3/queued [2])
  51/Build_EvolutionChamber_screen                      (3/queued [2]; 0/screen [84, 84])
  84/Build_SpawningPool_screen                          (3/queued [2]; 0/screen [84, 84])
 483/Train_Overlord_quick                               (3/queued [2])
  11/build_queue                                        (11/build_queue_id [10])
  59/Build_Hatchery_screen                              (3/queued [2]; 0/screen [84, 84])
 269/Harvest_Return_quick                               (3/queued [2])
Took 46.603 seconds for 2500 steps: 53.645 fps
Wrote replay to: /opt/share0/data/SC2/StarCraftII/Replays/RandomAgent/Simple64_2017-09-14-07-14-21.SC2Replay

Seems ok and I moved to pysc2-agents and typed python -m main --map=MoveToBeacon to start training and the log is as follow:

2017-09-14 15:20:50.525097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: 
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.7335
pciBusID 0000:03:00.0
Total memory: 7.92GiB
Free memory: 7.81GiB
2017-09-14 15:20:50.525430: W tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x888b850
2017-09-14 15:20:50.797637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 1 with properties: 
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.7335
pciBusID 0000:04:00.0
Total memory: 7.92GiB
Free memory: 7.81GiB
2017-09-14 15:20:50.798905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 1 
2017-09-14 15:20:50.798925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y Y 
2017-09-14 15:20:50.798932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1:   Y Y 
2017-09-14 15:20:50.798944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:03:00.0)
2017-09-14 15:20:50.798953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:04:00.0)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-9:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-10:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-11:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-15:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Exception in thread Thread-16:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 158, in <module>
    app.really_start(_main)
  File "/usr/local/lib/python2.7/dist-packages/google/apputils/app.py", line 220, in really_start
    sys.exit(main(argv))
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 148, in _main
    run_thread(agents[-1], FLAGS.map, FLAGS.render)
  File "/opt/share1/Project/Generic/RL/pysc2-agents/main.py", line 79, in run_thread
    visualize=visualize) as env:
  File "/usr/local/lib/python2.7/dist-packages/pysc2/env/sc2_env.py", line 128, in __init__
    raise ValueError("All arguments must be passed as keyword arguments.")
ValueError: All arguments must be passed as keyword arguments.

Change app in main.py to absl.app

Following pysc2 commit f2f96c08c32ea92fa60485cc6d7307bc17c1c6db. The pysc2 repo does not contain app class anymore and it was replaced by Google's abseil (absl.app).
Therefore the current version of main.py no longer works. For the conveniences of new users could you please update main.py by changing
from pysc2.lib import app
import gflags as flags
to
from absl import app
from absl import flags

The device and trainning time?

Thanks for sharing your code. Now, I'm trying to train this model on my computer to see if my device can support its trainning. But it seems like I can not find any output value. Maybe it has not trigger the situation when is_done==True yet. Could you tell me how many GPU or CPU you have use to train your model? And how long you trainning process has lasted?