sullychen / autopilot-tensorflow

A TensorFlow implementation of this Nvidia paper: https://arxiv.org/pdf/1604.07316.pdf with some changes

Autopilot-TensorFlow

A TensorFlow implementation of this Nvidia paper with some changes. For a summary of the design process and FAQs, see this medium article I wrote.

IMPORTANT

Absolutely, under NO circumstance, should one ever pilot a car using computer vision software trained with this code (or any home made software for that matter). It is extremely dangerous to use your own self-driving software in a car, even if you think you know what you're doing, not to mention it is quite illegal in most places and any accidents will land you in huge lawsuits.

This code is purely for research and statistics, absolutley NOT for application or testing of any sort.

How to Use

Download the dataset and extract into the repository folder

Use python train.py to train the model

Use python run.py to run the model on a live webcam feed

Use python run_dataset.py to run the model on the dataset

To visualize training using Tensorboard use tensorboard --logdir=./logs, then open http://0.0.0.0:6006/ into your web browser.

autopilot-tensorflow's Issues

issue of steering angle

please share the arduino code to get the steering angle we are not getting how to take it those values

Reporting MSE Loss value

Hey thanks a lot for opensourcing your code and dataset. With a mse loss i know it's difficult to compare accuracy but could you report the mse loss you achieved(on an average) on your validation set?

I am getting error while running "python run.py"

2017-07-07 19:03:25.099477: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
File "run.py", line 19, in
image = scipy.misc.imresize(frame, [66, 200]) / 255.0
File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 480, in imresize
im = toimage(arr, mode=mode)
File "/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py", line 289, in toimage
raise ValueError("'arr' does not have a suitable array shape for "
ValueError: 'arr' does not have a suitable array shape for any mode.

Upgrade

How can we upgrade to tensorflow 2.0?
E.g. placeholders are not supported anymore.
The upgrade-script also fails and adds more compat.v1 code which we shouldn't use anymore.

Image shows swapped color channels during run_dataset

Source & License of the dataset?

Hi,
Can you tell me where did you find the images in the dataset?

Thanks,
Akshay

The angle representation

For the angle you collected, are they in degrees or in radius

Pointer Update

After the first epoch, the train and the validation pointer is updated to maximum and hence to reinitialise the pointers are needed for next epoch. Am I missing something here?

Request for checkpoint file

Could you upload a pre-trained checkpoint file, i.e. .ckpt file.

'NoneType' object is not subscriptable

@SullyChen
Hi，I have encountered such a problem, I can‘t solve, do you have a good solution?
Thank you very much.

TypeError: src data type = 17 is not supported

When I use run_dataset.py to run my model, the error is as follows, how can I solve it?

Predicted steering angle: [[-14.39868]] degrees
Traceback (most recent call last):
File "run_dataset.py", line 78, in
new_img=crop_sky(change_bright(full_image))
File "run_dataset.py", line 25, in change_bright
hsv = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
TypeError: src data type = 17 is not supported

Source of the dataset?

Hi, May I know where did you get the dataset from? I need to know so that I can cite the source. :)

Test model on train data - not corresponding answers

Hi @SullyChen

First of all thank you a lot for letting available your code that is very nice to read, plus the DataSet and pre-trained model. It makes it very handy for experimenting with it :)

Before testing on new data, I wanted to verify I installed all correctly, so I wrote a basic test file, to run the model on few samples from the training data. I was thinking it would output values very close to the ground truth. But it is not the case...

What am I missing? Or are the output from the run actually good?

I hope you can help.. :)
In any case again thank you very much

K.

Images tested from the training data set:
['3252.jpg', '3253.jpg', '3254.jpg', '33255.jpg', '33256.jpg', '33257.jpg', '9501.jpg', '9502.jpg', '9503.jpg']

Corresponding values in data.txt file:
3252.jpg 2.420000
3253.jpg 2.420000
3254.jpg 2.420000

33255.jpg -37.210000
33256.jpg -37.210000
33257.jpg -37.310000

9501.jpg -17.240000
9502.jpg -17.450000
9503.jpg -17.550000

Output from the network:
Predicted steering angle: -61.0882010686 degrees ie -1.06619 rad for image: 3252.jpg
Predicted steering angle: -60.9035537346 degrees ie -1.06297 rad for image: 3253.jpg
Predicted steering angle: -60.0010808395 degrees ie -1.04722 rad for image: 3254.jpg

Predicted steering angle: -27.5703796036 degrees ie -0.481194 rad for image: 33255.jpg
Predicted steering angle: -28.3578594338 degrees ie -0.494938 rad for image: 33256.jpg
Predicted steering angle: -36.3197870436 degrees ie -0.6339 rad for image: 33257.jpg

Predicted steering angle: 46.7075895259 degrees ie 0.815201 rad for image: 9501.jpg
Predicted steering angle: 49.5550373342 degrees ie 0.864899 rad for image: 9502.jpg
Predicted steering angle: 52.0180820961 degrees ie 0.907887 rad for image: 9503.jpg

The code for testing:

import tensorflow as tf
import scipy.misc
import model
import cv2
import glob

sess = tf.InteractiveSession()
saver = tf.train.Saver()
saver.restore(sess, "save/model.ckpt")

all_img = glob.glob("*.jpg")
all_img.sort()
for img_path in all_img:
    img = cv2.imread(img_path)
    image = scipy.misc.imresize(img, [66, 200]) / 255.0
    radian = model.y.eval(feed_dict={model.x: [image], model.keep_prob: 1.0})[0][0]
    degrees = radian * 180 / scipy.pi
    print("Predicted steering angle: " + str(degrees) + " degrees ie "+ str(radian) + " rad for image: "+img_path)

tun.py scipy error

Running run.py. I made a few adjustments to the train and model scripts to make them compatible with tf 1.9. I am running python 3.6 and tesorflow-gpu==1.9

Traceback (most recent call last):
File "run_dataset.py", line 21, in
call("clear")
File "C:\Users\16175\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 287, in call
with Popen(*popenargs, **kwargs) as p:
File "C:\Users\16175\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 729, in init
restore_signals, start_new_session)
File "C:\Users\16175\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 1017, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

P.S. I just realized I spelt run wrong in the title :(

Error occur in run.py line21:call("clear")

I follow Use python run.py to run the model on a live webcam feed .
platform: Win10.
python version:3.5
tesorflow version:1.0
Then it occured the error:

Traceback (most recent call last):
  File "G:/codes/tensorflow2/Autopilot-TensorFlow/run.py", line 21, in <module>
    call("clear")
  File "D:\Program Files (x86)\Miniconda3\lib\subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "D:\Program Files (x86)\Miniconda3\lib\subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "D:\Program Files (x86)\Miniconda3\lib\subprocess.py", line 1224, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

What happen?

Some questions

Hi @SullyChen did you test training your model without the, scaling operation (atan)?
https://github.com/SullyChen/Autopilot-TensorFlow/blob/master/model.py#L88

Also did you do some sort of preprocessing to do image augmentation and to balance the dataset?
I'm asking this because most of the time the data has no steering.

I've created another driving example based on your code (Including a game and a tcp/ip comm class)
https://github.com/leonardoaraujosantos/DLMatFramework/tree/master/virtual/tensorDriver

Here is a video of my current performance
https://www.youtube.com/watch?v=PT8oToyV85w

TypeError: Variables to save should be passed in a dict or a list: ./logs

Traceback (most recent call last):
File "\Autopilot-TensorFlow-master\train.py", line 29, in
summary_writer = tf.train.Saver(logs_path, graph)
File "\python 6\lib\site-packages\tensorflow\python\training\saver.py", line 1281, in init
self.build()
File "\python 6\lib\site-packages\tensorflow\python\training\saver.py", line 1293, in build
self._build(self._filename, build_save=True, build_restore=True)
File "\python 6\lib\site-packages\tensorflow\python\training\saver.py", line 1330, in _build
build_save=build_save, build_restore=build_restore)
File "\python 6\lib\site-packages\tensorflow\python\training\saver.py", line 756, in _build_internal
saveables = self._ValidateAndSliceInputs(names_to_saveables)
File "\python 6\lib\site-packages\tensorflow\python\training\saver.py", line 656, in _ValidateAndSliceInputs
names_to_saveables = BaseSaverBuilder.OpListToDict(names_to_saveables)
File "\python 6\lib\site-packages\tensorflow\python\training\saver.py", line 504, in OpListToDict
"list: %s" % op_list)

Acceleration information

Hello, Chen. Thank you for your excellent job. I'm implementing your work, and I'm wondering whether your dataset has Acceleration information, or just torque degree?

in tensorboard we can find only loss graph how to find accruacy graph also

atan as output

Hi, I have two questions:

1. How did you collect the steering wheel angle data while driving? Tapped into the CAN bus?!

2. I see you used atan as the NN output, why is that? Ground truth (theta * pi/180) can go beyond [-pi/2, pi/2], how is that factored into your model? From dataset, I see angle degrees as big as 500.
   y = tf.mul(tf.atan(tf.matmul(h_fc4_drop, W_fc5) + b_fc5), 2) #scale the atan output

I'm learning the machine learning and try to replicate NVidia paper end-to-end network as well.

Thanks.
-JC

Steering angle PID? Share .ino?

Nice job on the Nvidia-Autopilot-TensorFlow, the code is great!

I saw your YouTube video on collecting the steering angle via the ODBII port with an Arduio. I was looking at ODBII PID's but have not been able to find one for steering angle.

Which ODBII shield did you use for the data collection?
Do you have any plans to share the .ino code? If so, that would be great.

Thanks.

Increasing the number of training images. Error Too many index array

Hi,

I have tried the scripts to train the auto-pilot with your own dataset, and it is working ok. Now, I have created my own dataset installing a webcam and a angle encoder in may car to capture around 30 images for second, and I manage to get for the moment 2 datasets one with 62000 images and their angle and another with 22000 images and angles too.

When I train both new captured datasets individually, they train ok and test ok. But if I merged both datasets to get a total of around 85000 images and angles, after the 100 step in the 1st epoch, the script driving_data.py crashed complaining about too many index array in line 42:

x_out.append(scipy.misc.imresize(scipy.misc.imread(train_xs[(train_batch_pointer + i) % num_train_images])[-150:], [66, 200]) / 255.0)

I changed the batch size from 100 to 200, and the only difference is that it crashed in step 110.

What is the correct number of epoch, batch size, etc for an increase of images in the dataset?

Thanks

Alex

When running run_dataset.py

totalMemory: 3.95GiB freeMemory: 3.74GiB
2018-10-14 15:59:22.345698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-10-14 15:59:23.308077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-14 15:59:23.308142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-10-14 15:59:23.308160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-10-14 15:59:23.308489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3468 MB memory) -> physical GPU (device: 0, name: GeForce 940M, pci bus id: 0000:01:00.0, compute capability: 5.0)

Predicted steering angle: -2.8395388761983194 degrees
: cannot connect to X server

I can't find the paper "End-to-End Learning Driver Policy using Moments Deep Neural Network"

Could you upload the paper or send to my mail ? I can't find the paper.thanks.
mail: [email protected]

No such file error on run_dataset

IOError: [Errno 2] No such file or directory: 'driving_dataset/10000.jpg'

Seems like the linked dataset is numbered up to 7148.jpg.

Would you mind also linking where the dataset is from? Thanks!

No such file or directory: 'driving_dataset/data.txt'

Sorry I tried to run

Python train.py

It says

IOError: [Errno 2] No such file or directory: 'driving_dataset/data.txt'

Where I can download your training data?

Thanks!

Unexpected train result?

Thanks for your sample code, I tried this but couldn't get expected result. I am using wheel rotation degree to train the data, this is my sample train data:

22.jpg 0
23.jpg 0
24.jpg 0
25.jpg 0
26.jpg 0
27.jpg 0
28.jpg 0
29.jpg 0
30.jpg 0
31.jpg 0
32.jpg -3.851463
33.jpg -6.549802
34.jpg -11.0871
35.jpg -14.10705
36.jpg -18.85992
37.jpg -21.56775
38.jpg -20.37253
39.jpg -17.5227
40.jpg -13.01936
41.jpg -9.757766
42.jpg -5.45346
43.jpg -2.566003
44.jpg 0
45.jpg 0
46.jpg 0
47.jpg 0
48.jpg 0
49.jpg 0
50.jpg 0
51.jpg 0
52.jpg 0
53.jpg 0
54.jpg 1.5773

mostly it is 0, but when I eval the data, it outputs:

0.848186871676
0.247460795198
0.585712200104
0.410212570454
0.451668243538
0.933786110727
0.905919685965
0.66789506391
0.497750729456
0.352852214917
0.391191160626
0.959048846033
0.931835024502
0.174691773018
-0.0430651931555
-0.260673075776
-0.258237713131
-0.620764197316
-0.750217252903
-0.715262639247
-0.710381775394
-0.733180519957
-0.473712732446
-0.605725828311
-0.814070557019

There is no 0s and when it is supposed to be 8 degree, it is like 0.x degree, did I feed the right parameter? Should I feed the steer wheel's rotation? Thank you

Hi, where can I find the code for this paper?

Hi, where can I find the code for this paper? thank you!

@Article{pan2017virtual,
title={Virtual to real reinforcement learning for autonomous driving},
author={Pan, Xinlei and You, Yurong and Wang, Ziyan and Lu, Cewu},
journal={arXiv preprint arXiv:1704.03952},
year={2017}
}

How to resume a training. HELP after electrical power down, the PC went down too !!

Hi,

Last night I have a power down at home and I lost 5 bash session of training, several of those were training for 3 days already with about 250000 images.

I have all checkpoint, model.ckpt and model.ckpt.meta from minutes before the power down.

I saw one of the issues were some one requested a checkpoint file, so I presume it is possible to resume a training session with those files, but I dont know how.

I launch a training in the same folder with the same files, but the training start at epoch 0. And I would need the training to resume in the last epoch or checkpoint.

Can anyone help me on that?.

Thanks

Alex

NotFoundError: Key Variable not found in checkpoint

Hello! I'm having the issue that you can find in the image attached.
Can someone help me to find the solution?

Getting same steering angle for "run.mp4"

I have trained your provided model on the same data set as you have used. But when I run "run_dataset" it is giving the same steering angle prediction for the whole video. Although the mse loss value I got was 0.2 approx. But still there is some issue with the output. What could possibly go wrong? If you can help me with this.

Import "tensorflow.compat.v1" could not be resolved

I created Anaconda virtual environment for Tensorflow, tried almost evey number of cases to make Python and Tensorflow versions are compatible with each other.
And as you can see, I already successfully installed Tensorflow, but still I can't get 'tensorflow.compat.v1'.
What should I do to solve this problem?

tensorflow error...

Hi, which version of tensorflow do you use?

My tensorflow is version 0.8

Error occurred：
Traceback (most recent call last):
File "train.py", line 5, in
import model
File "/home/gtx-140601/Autopilot-TensorFlow-master/model.py", line 88, in
y = tf.mul(tf.atan(tf.matmul(h_fc4_drop, W_fc5) + b_fc5), 2) #scale the atan output
AttributeError: 'module' object has no attribute 'atan'

Missing images in data.txt

The data.txt is missing close to 161 images which are present in the original dataset. Did the angle values also get displaced by this?

DataSet not exist

Hi @SullyChen ,

The dataset is not exist on google drive, could you provide?

Thanks!!
LemonAniLabs

Question

This is not issue for your code, but more get help. when I run your code in tensor flow in works perfectly fine. when I use the exact same thing (data, load data ,..) beside the model in keras and I use model.fit or train_batch . However,i don't get the model trained well. since you are experienced I though you can help me with that.( I already implement my whole project in keras and after not getting the proper result after days or weeks i got to this issue) . any help is really appreciated.
model.txt

value nan in loss

Hi!

I change lines in driving_data.py for version compatibility:

-#x_out.append(scipy.misc.imresize(scipy.misc.imread(train_xs[(train_batch_pointer + i) % num_train_images])[-150:], [66, 200]) / 255.0)

+imagen=imageio.imread(train_xs[(train_batch_pointer + i) % num_train_images])
+retamaño2=numpy.array(Image.fromarray(imagen[-150:]).resize([200, 66]))/255
+x_out.append(retamaño2)

But when execute show Loss=nan for all epochs ->Epoch: 0, Step: 0, Loss: nan

Can you help me?
thanks