Tutorial for building a model that generates steering angle based on image input. The model is then deployed on to a video game for testing. The complete description can be found on the readme.md document.
Python 1.04%Jupyter Notebook 98.96%
behavioralcloning's Introduction
BehavioralCloning
Tutorial for building a deep learning model that generates steering angle based on image input.
1. Toolchain
Simulator in training mode as data gethering/ data source
Offline Anaconda Python Environment for data pre-processing.
Cloud based ML environment on AWS for Model Training.
Download model to offline system and run the simulator in autonomous mode
Getting started
Offline installation - Install , use
Cloud Environment - This is a written to help you with the cloud based environment setup.
Clone this repository to both the cloud and the local environment.
Simulator Repo -
2. Getting around the repo
Model_Preprocessing.ipynb is the scratchpad that was used to try and experiment with building the model.It helps us extract and preprocess information and combine it.
Model.py is the Keras Model that contains
Model
Image Aug Generators
It obtains data from complete_data.csv which contains the sum total of all images , steering angles and their paths.
The data is obatined by running the simulator / stock data given by udacity itself.
Data/File Structure
The data is of the format ** Drive_Log.csv** which contains the path information where the images are sampled from the video and the actual images are stored in IMG. Each image is timestamped.
3. Workflow
Simulator - Generate Data in Training Mode
Analyze, Augment and PreProcess Data offline
Get more Data if required.
Upload Data to Cloud - AWS Machine Learning system .
Run the Model Training.
Download Model and run simulator in autonomous mode.
Repeat process
4. Summary of Steps
Exploratory Visualization of Dataset
Pandas + SeaBorn + MatPlotLib to create , load and append dataset from dataframe
Visualization to understand the distribution and quality of data.
Distribution Plot to see the spread and quantity of data
Time Series Plot to understand the quality of data. (To see noise to determine if filters are required)
Data Collection based on Shortcomings
Visualizing collected data from driving the simulator shows that the dataset looks entirely different for Keyboard and Mouse
So we pass the data through a Savitzky Golay filter that averages out the samples but maintains the
Area = (Steering_angle x Time) this effectively filters out the noise without destroying the signal.
Based on the histogram distribution plots collecting data by using certain driving styles.
Data from Track 1 - Clock wise and anticlockwise (MAC & Windows)
Data from Track 2 - Clock wise and anticlockwise (MAC & Windows)
Smooth Turn from both Tracks (MAC & Windows)
Recovery driving from both Tracks (MAC & Windows)
Problem Areas in both tracks (MAC & Windows)
Keyboard and Mouse
After initial model save and testing driving and training in problem areas to improve model on subset of data.
Name
Values
Data Sources
Mac, Windows, Linux Sim
Input Sources
Keyboard, Touch Pad, Mouse, Joystick
Tracks
Track-1, Track-2
Direction
Clockwise and Anti-Clockwise
Special Cases
Recovery Driving, Smooth Turns only, Stock Udacity Data
Size of Dataset
~37,000
Data Augmentation
Augmentation using Flipping, Translation from left and right camera images
Reduce the time spent on data gathering through data augmentation techniques
Data Perturbation to Increase Model Robustness
Brightness Perturbation : Random perturbation of brightness of the image.
Gaussian Noise : Blur filter with a random normal distribution across the image.
Adaptive Histogram Equalization : Can greatly help in the model learning the features quickly
Colospace inversion : RBG to BGR colorspace change
Sampling and Image Generators
These steps increase the challenge and generalization capability by creating harder images for the model to train on. Below is an example of augmented and perturbed image batch that is linked with the image generator that generates images during model training on the go. Seen below are the distribution of data and images of one random sample generated by the generator.
Since the distribution is clearly tri modal (peaks around 0 for Center Camera , + 0.25 and -0.25 for left and right cameras respectievely ) it is an unbalanced dataset.
Although significant efforts have been taken gather more data around turns , there is just simply more data around 0 and +/-0.25
Best possible option is to do Balancing through DownSampling
The method used to do downsampling is Weighted Random Normal Sampling .
Why we choose this is because , the dominant characteristic of the system is to stay around 0/.25 so we make sure we don't mess with that.
The steering angles are Discretized i.e made to fall under categories/ Bins
Counts are taken for each group and the weights are given as 1/ Counts in that group.
These weights are then Normalized - When summed up they need to be equal to 1
Then the batch size is used to sample this out of the data frame using the Sampling with weights
Define model architecture
Data Pre-processing steps
Normalization through feature scaling
Cropping region of interest
Resize image to increase model performance
Salient Features of Model
Batch Normalization before every activation
Overfitting prevention Dropouts and batch norm
Dropouts are implemented before the flatten layer and before the output layer with a 50% probability
Tried by adding multiple dropouts but it did need seem to have an effect on improving validation losses.
Pandas Dataframe - Sample is extremely handy in picking a weighted random sample.
Too many dropouts can sometimes be counter-productive.
Wasted a lot of time in trying to figure out why the model wasn't performing well and this was due to the gaussian blur perturbation that converted images to float.
The keyboard data though noisy didnt need filtration. Filtering the steering values lead to slower responses in sharp turns.
Image data generators were a life saver when it came to handling data in batches, sharing load between GPU and CPU.
Recovery driving training is only partial as we don't control the direction, throttle or speed just the steering. However that is a topic for the future. To plug in the steering and throttle based closed loop PI based speed controller.
7. Further aspirations
Optimize the learning process - Use the model driving the vehicle to iteratively generate data for further model training.
Increase the challenge in driving - Single lane driving based on direction, adversarial agents, pedestrians and traffic to understand model performance.