GithubHelp home page GithubHelp logo

pygta5's Introduction

Python Plays Grand Theft Auto 5 - Reboot

This is a reboot of the project from 2017, but with a whole new approach. This time, instead of collecting data by hand and training a classification model, we opted to create a whole new system for the data collecting, training and playing, and to do all of that live. Our models are now using regression (with other output types being still possible), training data is collected automatically and we stream models during their training phase so anyone can join and watch the progress.

Our AI, besides having multiple alter-egos (models), is still called Charles, like in 2017.

For a full explanation, refer to the System page, but for a quick summary - we’re using a central server that all other parts connect to. The Data Collectors run in separate GTA5 instances and our NPCs are collecting and Balancing the data, which is sent through the Server to the Trainer. The Trainer buffers these data, trains the model and updates the Player. The Player is using the Dual-Camera system to play so we can watch the 3rd person camera (also called cinematic camera) while the model is fed the Hood Camera.

For more information, refer to these other pages:



Stream and current stream layout:

The progress is being streamed on Twitch: https://www.twitch.tv/sentdex. Below we keep a list of dates and streamed models.

Stream's layout consists of: layout.jpg

  • top-left tile is the 3rd person camera of the car driving, the main camera to observe Charles
  • this main tile, in its lower-right corner, shows the current model and when it's been created
  • top-right tile called Hood Camera is exactly what the models "see" - the input to the convolutional backbone of the model
  • middle-right tile called Convcam shows the reshaped output of the CNN backbone and lets us observe how the CNN part trains
  • bottom-right tile called Player Console shows current driving predictions along with additional information
  • bottom-middle tile called Server/Trainer Console shows training progress along with some basic training information
  • bottom-left tile called Tensorboard shows the loss of the training process


Driving examples:

 



Model list:



Streaming timeline

May 1st:

May 2nd:

May 7th:

May 8th:

May 14th:

May 16th:

  • Stopped streaming model_0006_inceptionresnetv2_v1 at batch 60500
  • The stream has been offline for several days since we had to send back the Comino machine and setting up another one took longer than we expected

May 28th:

  • After new machine was set and new models developed, we restarted streaming
  • Started streaming model_0009_irv2_cr_tl_v1 from scratch

May 29th:

July 5th:

  • Stopped streaming model_0004_inceptionresnetv2_v3 at batch 752000
  • The stream has been stopped as there have been no further improvements in the model and to catch up with documentation and do some off-stream development


Project timeline

This is not a full project log and contains only the key changes that are visible on the stream (and a few from before the stream has started). There are many more things going on "behind the scenes".

2017:

November 2021:

  • The idea of bringing back this project has born

Early 2022:

  • The project has started the development phase of the main system, the initial models, and the game mod

April 8th:

April 9th:

April 10th:

April 11th:

April 15th:

April 19th:

April 22nd:

April 24th:

April 30th:

May 1st:

  • The Convcam has been created
  • Started streaming on Twitch.tv
  • Started streaming model_0003_xception_v2 from scratch

May 2nd:

May 3rd:

May 7th:

May 8th:

May 13th:

May 14th:

May 14th:

May 16th:

  • Stopped streaming model_0006_inceptionresnetv2_v1 at batch 60500
  • Stream has been offline for several days since we had to send back the Comino machine and setting up another one took longer than we expected

May 22nd:

May 23th:

May 24th:

May 25th:

May 26th:

May 28th:

May 29th:

May 30th:

May 31th:

June 1th:

June 3rd:

June 4th:

June 9th:

June 10th:

June 13th:

July 5th:

  • Stopped streaming model_0004_inceptionresnetv2_v3 at batch 752000
  • The stream has been stopped as there have been no further improvements in the model and to catch up with documentation and do some off-stream development

pygta5's People

Contributors

0xc0392b avatar aniket965 avatar cclauss avatar daniel-kukiela avatar nils-werner avatar schlumpyj avatar sentdex avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pygta5's Issues

Tensorboard issue

WARNING:tensorflow:Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
WARNING:tensorflow:Found more than one metagraph event per run. Overwriting the metagraph with the newest event.

Got these warnings while running tensorboard, and i cant figure out whats wrong.
after the warnings nothing happends at all

suggestion

try to test either one of the following models on that:

  • Wavenet;
  • deep convolutional recursive swarm of hybrid bdi and ANN

cuDNN status not initialized

Dear sentdex,

can You help me please? I despair...
When I want to run the test_model.py, always python would not respond when this in the console happend:

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED

cuDNN is installed and the %PATH%-variable holds the path to the cuDNN folder...
Can You send me Your %PATH%-variable as example? Maybe there is my problem....

I hope You can help me!

(Sorry for my bad english knowledgement)

Two questions about training

Hello!
After i have trained, should i delete the training data and replace with new or let it be and keep on filling up with new?
Also should i rename something between each training to make sure that tensorboard can follow it?
Had problems with tensorboard and i am not sure if i have trained wrong and that is what causes it...

Headless execution of the code

Is it possible to get the code to work when we have no GUI (for example running the code on a server)? The current code wants the window in the corner or visible in order to capture screenshots - any workaround the same?

Thanks!

PressKey isn't working?

Hey,

Not sure where the best place to put this is. Pythonprogramming doesn't have a comments section?

I'm trying to run through these tutorials, but I'm getting stuck on the input? Everything is exactly as it should be, but the "PressKey(W)" just doesn't do anything in GTA. As soon as I tab out i get a lot of w's, but it's just not working in the game. Anyone have any ideas?

Balance data for latest version

I tried to adapt balance data code for the latest version, the one that supports this:

w = [1, 0, 0, 0, 0, 0, 0, 0, 0]
s = [0, 1, 0, 0, 0, 0, 0, 0, 0]
a = [0, 0, 1, 0, 0, 0, 0, 0, 0]
d = [0, 0, 0, 1, 0, 0, 0, 0, 0]
wa = [0, 0, 0, 0, 1, 0, 0, 0, 0]
wd = [0, 0, 0, 0, 0, 1, 0, 0, 0]
sa = [0, 0, 0, 0, 0, 0, 1, 0, 0]
sd = [0, 0, 0, 0, 0, 0, 0, 1, 0]
nk = [0, 0, 0, 0, 0, 0, 0, 0, 1]

But when I try to train such model, I get value out of range, errors. I suppose I do something wrong in balance data.

Is there an updated code anywhere? Or am I not supposed to balance data myself? Without it neural net always uses forward.

Training data incorrect

Hello Sentdex,
First I 'd like to thank you for your wonderful tutorials
I have been following along with your tutorials and I am on part 9-10 of “Python plays GTA”
So I ran into an issue with the “create training data script”, after recording training data when I go to balance I notice that I only have

                                               0          1

0 [[3, 3, 6, 7, 7, 6, 6, 11, 6, 5, 6, 6, 4, 3, 3... [0, 1, 0]
1 [[3, 4, 6, 8, 8, 7, 7, 14, 9, 7, 6, 7, 5, 4, 4... [0, 1, 0]
2 [[4, 4, 6, 8, 8, 8, 8, 17, 11, 7, 7, 7, 5, 4, ... [0, 1, 0]
3 [[4, 5, 7, 9, 9, 8, 9, 20, 14, 8, 7, 8, 6, 5, ... [0, 1, 0]
4 [[5, 5, 8, 10, 10, 10, 9, 27, 19, 9, 8, 9, 7, ... [0, 1, 0]
Counter({'[0, 1, 0]': 3000})

looks like only one output from keys

and that's going forward only no left or right
I have pywin32 installed, any idea what could be wrong?

  1. Just started learning python so I fairly new to all of this, but I have a grasp on what's going on with the scripts so far .. is there another way of getting key input into the code?

Any help would be appreciated, I really looking forward to the rest of the series but I'm stuck here.

It seems to have some problem with shuffle function in balance.py

There is the simple 2000 data (no shuffle) with 1682 forwards, 161 lefts, 157 rights

But after using shuffle in build-in random library, It comes out with unpredictable result. (and I already test for more than 5 times, every time with different result.)

_2017_04_28_17_47_20_127

so I try to use shuffle method in numpy library. And the result is always correct.

_2017_04_28_17_48_32_618

@Sentdex Would you mind for testing it, Thx.

Tensorslicereader error

RESTART: C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\test_model.py
curses is not supported on this machine (please install/reinstall curses for an optimal experience)
Scipy not supported!
Traceback (most recent call last):
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
return fn(*args)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
status, run_metadata)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\contextlib.py", line 66, in exit
next(self.gen)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\pygta5-car-fast-0.001-alexnetv2-10-epochs-300K-data.model
[[Node: save_1/RestoreV2_34 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_34/tensor_names, save_1/RestoreV2_34/shape_and_slices)]]
[[Node: save_1/RestoreV2_34/_45 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_9_save_1/RestoreV2_34", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\test_model.py", line 48, in
model.load(MODEL_NAME)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\models\dnn.py", line 282, in load
self.trainer.restore(model_file, weights_only, **optargs)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\helpers\trainer.py", line 452, in restore
self.restorer.restore(self.session, model_file)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1457, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
run_metadata_ptr)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\pygta5-car-fast-0.001-alexnetv2-10-epochs-300K-data.model
[[Node: save_1/RestoreV2_34 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_34/tensor_names, save_1/RestoreV2_34/shape_and_slices)]]
[[Node: save_1/RestoreV2_34/_45 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_9_save_1/RestoreV2_34", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op 'save_1/RestoreV2_34', defined at:
File "", line 1, in
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\idlelib\run.py", line 130, in main
ret = method(*args, **kwargs)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\idlelib\run.py", line 357, in runcode
exec(code, self.locals)
File "C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\test_model.py", line 47, in
model = alexnet(WIDTH, HEIGHT, LR)
File "C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\alexnet.py", line 40, in alexnet
max_checkpoints=1, tensorboard_verbose=0, tensorboard_dir='log')
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\models\dnn.py", line 64, in init
best_val_accuracy=best_val_accuracy)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\helpers\trainer.py", line 147, in init
allow_empty=True)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1056, in init
self.build()
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1086, in build
restore_sequentially=self._restore_sequentially)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 691, in build
restore_sequentially, reshape)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 669, in restore_v2
dtypes=dtypes, name=name)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in init
self._traceback = _extract_stack()

NotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for C:\Users\gustav\Downloads\pygta5-master (1)\pygta5-master\pygta5-car-fast-0.001-alexnetv2-10-epochs-300K-data.model
[[Node: save_1/RestoreV2_34 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_34/tensor_names, save_1/RestoreV2_34/shape_and_slices)]]
[[Node: save_1/RestoreV2_34/_45 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_9_save_1/RestoreV2_34", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Dont know if i have done anything wrong or forgot something

train_model.py weird output

I had to recreate my issue because I didn't do a good job of explaining it before. I'm having an issue with train_model.py on Windows. I have linked the output below that I receive on epoch 8 (it's the same for the previous ones). It does not name the model files correctly (it saves them as model_alexnet-22.data-00000-of-00001.tempstate18218624911869073751). When I run the same script on the same set of code from my Macbook it works and correctly names my output file. My desktop is much faster than my Macbook though and I'd like to figure out why the data isn't saving correctly on Windows, so if you or anyone can provide any help, it'd be greatly appreciated.

Oh and at the very beginning I get Scipy not supported! even though it's installed.

Training Step: 78 | total loss: �[1m�[32m0.58550�[0m�[0m | time: 2.035s
| Momentum | epoch: 008 | loss: 0.58550 - acc: 0.8028 -- iter: 064/664
�[A�[ATraining Step: 79 | total loss: �[1m�[32m0.55399�[0m�[0m | time: 4.048s
| Momentum | epoch: 008 | loss: 0.55399 - acc: 0.8151 -- iter: 128/664
�[A�[ATraining Step: 80 | total loss: �[1m�[32m0.52058�[0m�[0m | time: 6.050s
| Momentum | epoch: 008 | loss: 0.52058 - acc: 0.8276 -- iter: 192/664
�[A�[ATraining Step: 81 | total loss: �[1m�[32m0.49628�[0m�[0m | time: 8.038s
| Momentum | epoch: 008 | loss: 0.49628 - acc: 0.8356 -- iter: 256/664
�[A�[ATraining Step: 82 | total loss: �[1m�[32m0.48760�[0m�[0m | time: 10.098s
| Momentum | epoch: 008 | loss: 0.48760 - acc: 0.8380 -- iter: 320/664
�[A�[ATraining Step: 83 | total loss: �[1m�[32m0.45747�[0m�[0m | time: 11.487s
| Momentum | epoch: 008 | loss: 0.45747 - acc: 0.8495 -- iter: 384/664
�[A�[ATraining Step: 84 | total loss: �[1m�[32m0.45454�[0m�[0m | time: 12.883s
| Momentum | epoch: 008 | loss: 0.45454 - acc: 0.8520 -- iter: 448/664
�[A�[ATraining Step: 85 | total loss: �[1m�[32m0.45365�[0m�[0m | time: 14.871s
| Momentum | epoch: 008 | loss: 0.45365 - acc: 0.8502 -- iter: 512/664
�[A�[ATraining Step: 86 | total loss: �[1m�[32m0.44107�[0m�[0m | time: 16.870s
| Momentum | epoch: 008 | loss: 0.44107 - acc: 0.8495 -- iter: 576/664
�[A�[ATraining Step: 87 | total loss: �[1m�[32m0.41673�[0m�[0m | time: 18.863s
| Momentum | epoch: 008 | loss: 0.41673 - acc: 0.8583 -- iter: 640/664
�[A�[ATraining Step: 88 | total loss: �[1m�[32m0.60144�[0m�[0m | time: 22.472s
| Momentum | epoch: 008 | loss: 0.60144 - acc: 0.8069 | val_loss: 0.13101 - val_acc: 0.9660 -- iter: 664/664

grabscreen takes all screens

When I use the grabscreen code it grabs my entire desktop of 3 monitors. But it only has visual info on a youtube video.

I changed the imports a bit as I had No module named ... errors. Imports are now as follows:
from win32 import win32gui import win32ui, win32con, win32api

So, the grabscreen doesn't focus on the GTA window, which is 800x600, and takes images of the size of 3 monitors and only has visual info on a youtube video on another monitor.

new collector error on startup

got this error on startup of the collector

File exists, loading previous data!
4
3
2
1
Traceback (most recent call last):
  File "D:\Development\python\pygta5\collecting-data-for-larger-fpv-model.py", line 109, in <module>
    main()
  File "D:\Development\python\pygta5\collecting-data-for-larger-fpv-model.py", line 85, in main
    screen = grab_screen(region=(0,40,800,640))
TypeError: grab_screen() got an unexpected keyword argument 'region'

Slow training due to TensorBoard verbosity

Hey @Sentdex, watching latest youtube video got me wondering why your two titans were training at the same rate. I thought it might be a cpu bottleneck somewhere in the code so I had a look through and the only obvious thing I could see was the TensorBoard output.

After a quick test, with short training sessions changing the verbosity from 2 to 0, I got around 20x speedup.
The line in alexnet.py:

model = tflearn.DNN(network, checkpoint_path='model_alexnet',
                    max_checkpoints=1, tensorboard_verbose=2, tensorboard_dir='log')

To:

model = tflearn.DNN(network, checkpoint_path='model_alexnet',
                    max_checkpoints=1, tensorboard_verbose=0, tensorboard_dir='log')

I can now train a reasonable model in under an hour. My guess is that all the data processing/saving for TensorBoard is happening on the cpu with extra slowdown writing it to the drive. Therefore it cant keep up. So if you don't need the extra verbosity its a good idea to turn it down.

If anyone can confirm this I would appreciate it, I was a bit shocked and cynical when I saw the epochs flying by!

Edit: It also results in a log file that is a few Megabyte vs a few hundred and TensorBoard no longer struggles to load it. Hopefully it will stop those crashes too.

A session error occurs.

I tried to run 0.03v in the provided tutorial, but the problem is the same as the picture.
Why is the problem happening?
I have already collected a small amount of train data and have encountered problems trying to do the learning.
image

Help with final stages

Hey Harrison,

I managed to get everything working up until the last step. I'm doing this on Windows like you. I had been using Python2 which doesn't support Tensorflow on Windows. So I switched to Python3 but now it says that Python3 doesn't support cv2. What did you use to get this to work?

Implement xInput for fine control

Hey @Sentdex,
Creating an issue just for visibility as I no longer think a pull request is necessary. I don't know enough about machine learning (yet!) to properly work it in to your code.

Head over to https://github.com/bayangan1991/PYXInput for my python package for creating a virtual xbox controller. Fully documented and a legit PyPI package.

I have tested this with Rocket League, the only car game I own, and it works perfectly.

I am super excited to see what you can do with my code!

Movement / Visual memory

At the moment, I think memory will prove to be the most useful enhancement to what we have so far.

Thus far, I've considered an encoder/decoder-like setup, with first the convnet, then the recurrent net which will take the final prediction, the previous 100 frames of moves, and then put out the output.

I am not certain this is totally necessary, I am also tinkering with just simply taking a mean of predicted movements in the last 5-10 frames, with some caveats.

Improving FPS capture rate

Check out my gist where I've ported some C code from another project to capture an window handle:
A. No dependencies needed besides regular python (no need for PIL or PyWin32).
B. Can capture specific window handles instead of all the desktop.
C. Very fast, about 16ms capture (even for full screen) compared to 47ms capture using ImageGrab with 800x600 bounding box.
D. Can reuse an previous allocated numpy array for the result.

https://gist.github.com/tzickel/5c2c51ddde7a8f5d87be730046612cd0

Error when running test_model.py

Thank you for your code. I tried to run it. It shows me the error below.

runfile('C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code/test_model.py', wdir='C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code')
curses is not supported on this machine (please install/reinstall curses for an optimal experience)
Traceback (most recent call last):

File "", line 1, in
runfile('C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code/test_model.py', wdir='C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code')

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
execfile(filename, namespace)

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code/test_model.py", line 47, in
model = alexnet(WIDTH, HEIGHT, LR)

File "C:\Users\Xu Zhang\Desktop\pygta5-master\pygta5-master\Tutorial Codes\Part 8-13 code\alexnet.py", line 40, in alexnet
max_checkpoints=1, tensorboard_verbose=2, tensorboard_dir='log')

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tflearn\models\dnn.py", line 64, in init
best_val_accuracy=best_val_accuracy)

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tflearn\helpers\trainer.py", line 167, in init
self.session.run(init)

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 767, in run
run_metadata_ptr)

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 965, in _run
feed_dict_string, options, run_metadata)

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1015, in _do_run
target_list, options, run_metadata)

File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)

ResourceExhaustedError: OOM when allocating tensor with shape[5120,4096]
[[Node: FullyConnected/W/Initializer/truncated_normal = Add[T=DT_FLOAT, _class=["loc:@FullyConnected/W"], _device="/job:localhost/replica:0/task:0/gpu:0"](FullyConnected/W/Initializer/truncated_normal/mul, FullyConnected/W/Initializer/truncated_normal/mean)]]

Caused by op 'FullyConnected/W/Initializer/truncated_normal', defined at:
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 227, in
main()
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\spyder\utils\ipython\start_kernel.py", line 223, in main
kernel.start()
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 474, in start
ioloop.IOLoop.instance().start()
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 887, in start
handler_func(fd_obj, events)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 390, in execute_request
user_expressions, allow_stdin)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 501, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2827, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
runfile('C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code/test_model.py', wdir='C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code')
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
execfile(filename, namespace)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Xu Zhang/Desktop/pygta5-master/pygta5-master/Tutorial Codes/Part 8-13 code/test_model.py", line 47, in
model = alexnet(WIDTH, HEIGHT, LR)
File "C:\Users\Xu Zhang\Desktop\pygta5-master\pygta5-master\Tutorial Codes\Part 8-13 code\alexnet.py", line 30, in alexnet
network = fully_connected(network, 4096, activation='tanh')
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tflearn\layers\core.py", line 157, in fully_connected
restore=restore)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 177, in func_with_args
return func(*args, **current_args)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tflearn\variables.py", line 65, in variable
validate_shape=validate_shape)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 988, in get_variable
custom_getter=custom_getter)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 890, in get_variable
custom_getter=custom_getter)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 348, in get_variable
validate_shape=validate_shape)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 333, in _true_getter
caching_device=caching_device, validate_shape=validate_shape)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 684, in _get_single_variable
validate_shape=validate_shape)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variables.py", line 197, in init
expected_shape=expected_shape)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variables.py", line 274, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 673, in
shape.as_list(), dtype=dtype, partition_info=partition_info)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\init_ops.py", line 239, in call
dtype, seed=self.seed)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\random_ops.py", line 176, in truncated_normal
value = math_ops.add(mul, mean_tensor, name=name)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 73, in add
result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 763, in apply_op
op_def=op_def)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\Xu Zhang\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1226, in init
self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[5120,4096]
[[Node: FullyConnected/W/Initializer/truncated_normal = Add[T=DT_FLOAT, _class=["loc:@FullyConnected/W"], _device="/job:localhost/replica:0/task:0/gpu:0"](FullyConnected/W/Initializer/truncated_normal/mul, FullyConnected/W/Initializer/truncated_normal/mean)]]

No balance_Data.py?

Currently, to use the latest version of this AI, you will need to run first "create_training_data.py," then balance this data with "balance_Data.py."

From what I understand, the finished product is what's in the root of the git folder.

I don't see a balance_Data.py file.

Am I supposed to use the one in the Part14-15 folder or is it already included in the train_model.py folder?

Step by Step installation and running code

Step by Step installation and running code on Mac OS x. Please guide. I have used pip install to install NumPy. How to install Open CV ? Do i need to install opnCV in the root directory / Tensorflow visual env ?

Why are you using pooling layers?

I'm really new to all this AI and NN stuff, so please correct me if I'm wrong.
After reading some papers about the DeepMind Atari agent, I learned, that they removed all pooling layers from their convolutional neural nets. Pooling layers are there to make the net nonsensitive to positions of objects, but since that's also very important, shouldn't we remove all pooling layers from the model?

memoryerror with numpy

I've an error on wille using this on windows 10 with any x64 processor ( but an x32 python version ).
Here the error.

C:\Users\marius\Desktop\pygta5-master>python "1. collect_data.py"
File exists, moving along 1
File does not exist, starting fresh! 2
4
3
2
1
STARTING!!!
100
200
300
400
500
Traceback (most recent call last):
File "1. collect_data.py", line 116, in
main(file_name, starting_value)
File "1. collect_data.py", line 97, in main
np.save(file_name,training_data)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\numpy\lib\npyio.py", line 509, in save
pickle_kwargs=pickle_kwargs)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\numpy\lib\format.py", line 576, in write_array
pickle.dump(array, fp, protocol=2, **pickle_kwargs)
MemoryError

C:\Users\marius\Desktop\pygta5-master>

Making a general learning Model for any game

Hey,
I would like to make this code suitable to learn any (simple) game.
I already changed it so it could read every keyboard input and the mouse position and work with it.
(https://github.com/JohannesKnecht/pygta5)

Is there an easy way to train this model with more than 3 variables or do you need to build a completely new model for it to work ? Or am I just not training long enough for it to work ?

Would appreciate a lot if someone could help me with that ^^

Using RNN

Hello

Are you going to start using reinforcement learning for you driver?

Input beyond video stream

Hey friends,

As far as I understood in the YouTube videos and judging from what I read in the comments, it is also desirable for you to give the neural network input beyond the video stream. E.g. you would like to know the position and speed and maybe the planned trajectory of other vehicles around you.
I found this project on GitHub and I was wondering if we could integrate this: https://github.com/lgrahl/scripthookvpy3k
To get the information beyond the video stream. It shall give us access to a rich API that is "documented" (rather enumerated) here: http://www.dev-c.com/nativedb/

What do you think?

Regards
Patrik

xbox 360 emulation example...and help me :)

Starting file: https://github.com/Sentdex/pygta5/blob/master/vjoy.py

So with test(), you can successfully rotate, and the world is great.

Now, I am just trying to wrangle this.

My #1 interest with this is for use when walking around as a ped, looking left/right/up/down, but it would also be great to be able to use in a car to control throttle, braking, and left/right turning.

If anyone has some insight on how to properly control these, I am all ears. Some helper functions or something would be epic, or just some examples of actually controlling and resetting joys would be great.

Instructions for installing all the required stuff at the top of the vjoy.py script.

Error when trying to train_model from new collector

hdf5 is not supported on this machine (please install/reinstall h5py for optimal experience)
curses is not supported on this machine (please install/reinstall curses for an optimal experience)
Scipy not supported!
---------------------------------
Run id: pygta5-car-fast-0.001-alexnetv2-10-epochs-300K-data.model
Log directory: log/
---------------------------------Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python35\lib\site-packages\tflearn\data_flow.py", line 187, in fill_feed_dict_queue
    data = self.retrieve_data(batch_ids)
  File "C:\Program Files\Python35\lib\site-packages\tflearn\data_flow.py", line 222, in retrieve_data
    utils.slice_array(self.feed_dict[key], batch_ids)
  File "C:\Program Files\Python35\lib\site-packages\tflearn\utils.py", line 187, in slice_array
    return X[start]
IndexError: index 7704 is out of bounds for axis 0 with size 400


Training samples: 8100
Validation samples: 2025
--

is this cause the train/alexnet is not ready for the new data?

Concept inquiry

Hi,

This is not a pure code issue, i wonder something conceptual.

You state in the channel that "Charles only sees exactly what you see" and that is only partially true. We humans can infer 3D information from 2D images by imagining it based on our experience, Charles can't do that and therefore it will only achieve limited ability while driving.

I think it would be more interesting to give it a set of sensors that a Tesla would have for example and then let it operate in that kind of input, instead of just the images alone.

p.s. Awesome project anyway!

curses is not supported on this machine error

Error on running train model.py
GPU Nvidia 930mx
supports cuda and tensor-flow
I'm too much confused by this error

curses is not supported on this machine (please install/reinstall curses for an optimal experience)
Traceback (most recent call last):
File "C:\Users\asus\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\layers\recurrent.py", line 9, in
from tensorflow.python.ops.rnn import rnn_cell_impl as _rnn_cell, dynamic_rnn as _drnn, static_rnn as _rnn, bidirectional_dynamic_rnn as _brnn
ImportError: cannot import name 'rnn_cell_impl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\asus\python code\gta5\train_model.py", line 4, in
from alexnet import alexnet
File "C:\Users\asus\python code\gta5\alexnet.py", line 11, in
import tflearn
File "C:\Users\asus\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn_init_.py", line 21, in
from .layers import normalization
File "C:\Users\asus\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\layers_init_.py", line 11, in
from .recurrent import lstm, gru, simple_rnn, bidirectional_rnn,
File "C:\Users\asus\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\layers\recurrent.py", line 13, in
from tensorflow.contrib.rnn.python.ops.core_rnn import static_rnn as _rnn, static_bidirectional_rnn as _brnn
ImportError: No module named 'tensorflow.contrib.rnn.python.ops.core_rnn'

Memory exhausted error

Hello!

While trying to train the data in the latest update i got alot of memory exhausted errors. It didnt even create a model. Is it a problem with the trainingdata that i am giving to it or isnt 4gb x2 not enought anymore? If so is there an easy way to supply it in portions so that i dont fill up the vrm or do i need to go for the slower cpu?
Currently running 2x 980 in sli
Thanks in advance //Frossaren

WindowTools module.

Hi guys, @Sentdex,

I've started to put together a simple (to-be-)cross-platform module to interact with and gather data about GUI windows, in this case, to grab the x-position, y-position, width and height of the "Grand Theft Auto 5" window for use in, for example, the test_model.py script.

My issue is that I don't have a Windows PC available for testing, so if anyone would kindly offer a few minutes to test my changes (below) that'd be great.

https://github.com/OpenPetel/pygta5

Which accuracy do I need to reach for a decent agent?

I ve trained my model by using 200gb of driving data and inception_v3 model on same street with same weather conditions and 8.000 epochs with 300kb batches per epoch and learning rate of 0.000000001.
Accuracy is somewhat about 0.2(smoothed).
But when I test it out it keeps pressing WD with a verage plus of 0.3 of probability.

My question, how much data and which acc. do I need for a decent agent?

Tensorboard:
http://honesthome.ddns.net/s.png

Example Output [W, WA, WD, S, SA, SD, A, D]:
[ 0.1167456 0.16700381 0.27807635 0.05266069 0.08322734 0.1091948
0.11513347 0.07795788]
0.278076350689
forward_right
[ 0.11684461 0.16718809 0.27833453 0.0525772 0.08328342 0.1088759
0.11493194 0.07796425]
0.278334528208
forward_right
[ 0.11666095 0.16800123 0.27791589 0.05253068 0.08310306 0.10876592
0.11503578 0.07798637]
0.277915894985
forward_right
[ 0.116884 0.16714898 0.2784971 0.05255117 0.08334375 0.10875834
0.11481125 0.0780055 ]
0.278497099876
forward_right
[ 0.11672532 0.16719386 0.27897081 0.05252673 0.08339871 0.10860606
0.11471127 0.07786718]
0.278970807791
forward_right
[ 0.11685108 0.16760175 0.27800107 0.05256343 0.08338011 0.10876746
0.11482292 0.07801216]

Not returning `delta_count`

Inside the test_model.py file, there is a line

motion_log.append(delta_count)

delta_count was not initialized before

and I have another line that's

delta_count_last = motion_detection(t_minus, t_now, t_plus)

I'm assuming this is meant to be

delta_count= motion_detection(t_minus, t_now, t_plus)
?

But then under motion.py

the function motion_detection has a line that says

dst = cv2.addWeighted(screen,1.0, delta_view,0.6,0)

But no where is screen defined.

Access Denied Tensorflow Error?

Could not open C:\Users\User1\Desktop\aidrivers: Unknown: NewRandomAccessFile failed to Create/Open: C:\Users\User1\Desktop\aidrivers : Access is denied. ; Input/output error

Why is this happening? All i did was update to the newest pygta5? I tried putting it into another folder but that doesn't seem to fix that error.

A Suggestion on Output of the Network

Currently the output of the network is 9-dimensional with the meaning of:
w = [1,0,0,0,0,0,0,0,0] s = [0,1,0,0,0,0,0,0,0] a = [0,0,1,0,0,0,0,0,0] d = [0,0,0,1,0,0,0,0,0] wa = [0,0,0,0,1,0,0,0,0] wd = [0,0,0,0,0,1,0,0,0] sa = [0,0,0,0,0,0,1,0,0] sd = [0,0,0,0,0,0,0,1,0] nk = [0,0,0,0,0,0,0,0,1]
I believe this could be simplified to just 4 states { w, s, a, d } and keys to be pressed are to detected by a threshold individually. This might speed up the learning and calculation process.

Joystick Emulation

With the idea of hooking up the output weights to the a joystick directly I took a quick look at Joystick emulation options and came across this:
http://vjoystick.sourceforge.net/joomla256.02/index.php/forum/4-Help/1044-getting-started-with-python

Specifically the second last post, by trafiken. It looks like a pretty nice and lightweight solution, I believe it only requires the sdk and driver for the virtual device. If I get the time I'll give an implementation a go myself.
Loving this project so far.

import error

Got this error as i was trying to train my model. Dont know what is causing it and why it suddenly started to happend

Traceback (most recent call last):
File "train_model.py", line 9, in
from models import inception_v3 as googlenet
File "C:\Users\gustav\Desktop\pygta5-master\models.py", line 17, in
import tflearn
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn_init_.py", line 21, in
from .layers import normalization
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\layers_init_.py", line 10, in
from .recurrent import lstm, gru, simple_rnn, bidirectional_rnn,
File "C:\Users\gustav\AppData\Local\Programs\Python\Python35\lib\site-packages\tflearn\layers\recurrent.py", line 8, in
from tensorflow.contrib.rnn.python.ops.core_rnn import static_rnn as _rnn,
ImportError: No module named 'tensorflow.contrib.rnn.python.ops.core_rnn'

Error when running test_model.py

2017-05-29 20:57:50.239519: E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2017-05-29 20:57:50.239596: E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:366] error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
2017-05-29 20:57:50.240658: E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:326] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2017-05-29 20:57:50.242005: F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\kernels\conv_ops.cc:659] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)

Just when i was finished with the last one this popped up. Anyone seen it and know what to do?

Only going straight

When I test my model the car (Rocket League) only chooses straight. If I only train by pressing left and right, rather than only going left or only going right - my model (Inception_v3) chooses to press 'no keys' the whole time. I am assuming the problem has to do something with this line of code, perhaps because I am using a different game and training using 800x600 rather than 1920x1080.

prediction = np.array(prediction) * np.array([4.5, 0.1, 0.1, 0.1, 1.8, 1.8, 0.5, 0.5, 0.2]) 

I don't think I understand the purpose of this line of code. Any help is greatly appreciated.

Grab Screen in other OS

Inspired by your GTA-series, I want to develop an IA able to play other game or able to interact with a android emulator, but I on Mac and I got stuck on the problem to grab the screen with a decent fps. To solve the problem I found a module called MSS (written in pure python, working on windows, mac, linux) capable to grab the screen at ~40 fps at 800x400 resolution.

import numpy as np
import cv2
import time
from mss import mss
from PIL import Image

# Macbook have a bug about box's dimensions, can be just multiple of 16
# but this bug will be fix soon in the next release

box = { "top": 45, "left": 0, "width": 480, "height": 320}
sct = mss()

while True:

    t = time.time()
    sct.get_pixels(box)
    image = Image.frombytes('RGB', (sct.width, sct.height), sct.image)

    # Could be useful in MAC OS    
    # b, g, r = image.split()
    # image = Image.merge("RGB", (r, g, b))

    cv2.imshow('test', np.array(image))
    print('fps: {}'.format(1/(time.time()-t)))

    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break
        

SOLVED grab_screen problem with recorded data

have it set to default but gtav is out of the screen cap
image

screen = grab_screen(region=(0,40,800,640))

got a 4k(3840x2160) display with 100% scaling from window.

got a solution for that
change the grab_screen argument to region=(0,25,800,625)
that will keep the game in full frame

image
used the view_data.py from #20 to see if my frames where fine

Reshape on train_data is mixing the pixels!

The reshape to the train data and also to the screen grabbed on testing model is mixing the pixels in the screen:

image

the solution is to change width by height in every line where is a reshape, as in

X = np.array([i[0] for i in train]).reshape(-1,WIDTH,HEIGHT,3)

X = np.array([i[0] for i in train]).reshape(-1,WIDTH,HEIGHT,3)

by:

X = np.array([i[0] for i in train]).reshape(-1,HEIGHT,WIDTH,3)

and now reshape is OK:

image

Screen not defined in motion

Screen is not defined in motion.py and therefor i cant run test_model.py Anyone know what is supposed to be the screen there?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.