ljm0 / self_driving_learn

learn deep learning, computer vision and self-driving

Self-driving Simulation

Try to learn some Self-driving and deep learning based on Udacity and Na's work.

Table of Contents

• Dataset
• Usage
  • Training
    • Local machine
    • Google Colab host
    • Kaggle kernel
  • Evaluation
  • Create videos
• Project Overview
  • 1. Prepare training data
  • 2. Project code structure
  • 3. Training neural network
  • 4. Model architecture and hyper-parameters
    • Model architecture
    • Hyper-parameters
• FAQ
• Experimental result
  • Training loss vs Validation loss
  • Training loss vs Validation loss (Smoothed)
  • Demo videos
• References

Dataset

• Kaggle together with starter code and pre-trained models
• Floydhub

Left	Center	Right

Usage

Training

Local machine

usage: main.py [-h] [--dataroot DATAROOT] [--ckptroot CKPTROOT] [--lr LR]
               [--weight_decay WEIGHT_DECAY] [--batch_size BATCH_SIZE]
               [--num_workers NUM_WORKERS] [--train_size TRAIN_SIZE]
               [--shuffle SHUFFLE] [--epochs EPOCHS]
               [--start_epoch START_EPOCH] [--resume RESUME]
               [--model_name MODEL_NAME]

Main pipeline for self-driving vehicles simulation using machine learning.

optional arguments:
  -h, --help            show this help message and exit
  --dataroot DATAROOT   path to dataset
  --ckptroot CKPTROOT   path to checkpoint
  --lr LR               learning rate
  --weight_decay WEIGHT_DECAY
                        weight decay (L2 penalty)
  --batch_size BATCH_SIZE
                        training batch size
  --num_workers NUM_WORKERS
                        # of workers used in dataloader
  --train_size TRAIN_SIZE
                        train validation set split ratio
  --shuffle SHUFFLE     whether shuffle data during training
  --epochs EPOCHS       number of epochs to train
  --start_epoch START_EPOCH
                        pre-trained epochs
  --resume RESUME       whether re-training from ckpt
  --model_name MODEL_NAME
                        model architecture to use [nvidia, lenet]

An example of training usage is shown as follows:

python3 main.py --epochs=50 --resume=True

Google Colab host

if you prefer to use Colab as training platform, feel free to use train.ipynb script. Make sure you have already uploaded training data to Google Drive.

Kaggle kernel

Kaggle also provides GPU support in its own kernel. Feel free to use this as the start code.

Evaluation

Training images are loaded in BGR colorspace using cv2 while drive.py load images in RGB to predict the steering angles.

After training process, use the saved model and drive.py file to test your model performance in the simulator. Remeber to select AUTONOMOUS MODE. Click Allow to accept incoming network connections for python scripts.

usage: drive.py [-h] model [image_folder]

Remote Driving

positional arguments:
  model         Path to model h5 file. Model should be on the same path.
  image_folder  Path to image folder. This is where the images from the run
                will be saved.

optional arguments:
  -h, --help    show this help message and exit

An example of test usage is shown as follows:

python3 drive.py model.h5 runs1/

Tracks

Terrain 1	Terrain 2	Terrain 3

Create videos

usage: video.py [-h] [--fps FPS] image_folder

Create driving video.

positional arguments:
  image_folder  Path to image folder. The video will be created from these
                images.

optional arguments:
  -h, --help    show this help message and exit
  --fps FPS     FPS (Frames per second) setting for the video.

An example of creating video usage is shown as follows:

python3 video.py runs1/ --fps 48

Project Overview

1. Prepare training data

Once you have installed the self-driving car simulator, you will find there are 2 different tracks in the simulator. The second one is much harder to deal with. Please choose the terrain you like and make sure that you select TRAINING MODE.

Click RECORD button on the right corner and select a directory as the folder to save your training image and driving log information.

Click RECORD again and move your car smoothly and carefully. After you have completed recording your move, the training data will be stored in the folder you selected. Here I suggest you record at least 3 laps of the race. The first lap of race, please try best to stay at the center of the road, the rest could be either on the left hand side and right hand side of the road separately.

• /IMG/ - recorded images from cneter, left and right cameras.
• driving_log.csv - saved the image information and associated information like steer angle, current speed, throttle and brake.

2. Project code structure

.
├── RcCarDataset.py         # Customed Dataset for Self-driving car simulator (Simple CNN as well as Nvidia paper's CNN model)
├── drive.py                # Test script
├── main.py                 # Main pipeline
├── model.py                # CNN model declaration
├── train_in_colab.ipynb    # Colab training script
├── trainer.py              # Trainer
├── utils.py                # Helper functions
└── video.py                # Create videos

3. Training neural network

4. Model architecture and hyper-parameters

Model architecture

NVIDIA model used

Image normalization to avoid saturation and make gradients work better.
Convolution: 5x5, filter: 24, strides: 2x2, activation: ELU
Convolution: 5x5, filter: 36, strides: 2x2, activation: ELU
Convolution: 5x5, filter: 48, strides: 2x2, activation: ELU
Convolution: 3x3, filter: 64, strides: 1x1, activation: ELU
Convolution: 3x3, filter: 64, strides: 1x1, activation: ELU
Drop out (0.5)
Fully connected: neurons: 100, activation: ELU
Fully connected: neurons: 50, activation: ELU
Fully connected: neurons: 10, activation: ELU
Fully connected: neurons: 1 (output)

Hyper-parameters

Hyper-parameters	Description
lr=1e-4	learning rate
weight_decay=1e-5	weight decay (L2 penalty)
batch_size=32	training batch size
num_workers=8	# of workers used in dataloader
train_size=0.8	train-validation set split ratio
shuffle=True	whether shuffling data during training

FAQ

AttributeError: Can't get attribute '_rebuild_parameter'

This error means that you are trying to load a newer version of model checkpoint in an older version of PyTorch.

Check PyTorch version in your local machine:

import pytorch
print(torch.__version__)

For Google Colab users, please try to first get your PyTorch version in your local machine and install the same version on Colab virtual machine using the following snippets, PyTorch 0.4.1, for example.

# http://pytorch.org/
from os.path import exists
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision

Experimental result

Training loss vs Validation loss

Training loss vs Validation loss (generalized)

Training loss vs Validation loss (Smoothed)

Applied a Savitzky-Golay filter to the steering angle column. This helps reduce the training loss because original signal for steering angle input is via keyboard which is very dicrete.

Training loss vs Validation loss (denoised)

Demo videos

Track 1	Track 2	Track 3

References

[1] Nvidia research, End to End Learning for Self-Driving Cars
[2] Self-driving car simulator developed by Udacity with Unity