Learn How to Train a Self Driving Car with Keras and Video Game Data

with machinehorizon


Computer Vision and Deep Learning with Simulated Driving Data

Predicting steering angles is an a critical task for any self-driving machine. Whether it be an actual car, a Roomba vacuum, or a video game car - all must be able to anticipate steering angles. 

The Dataset

In this tutorial, I am collecting data via Udacity's self driving car simulator. This is a simple driving program developed in the Unity framework. I am assuming you have access to this application, or a similar pipeline for predicting steering angles in real time. 

Any driving game that is mod-able could hypothetically be used as a data soruce for this lesson. GTA V would most likely yield the most valuable and robust data due to its realistic traffic simulation. The link to download the core dataset can be found on the DeepDrive.io homepage. It is 80GB in size and contains 600,000 images. This is the dataset used for this example, but you may consider building your own dataset from scratch. 

Packages you will Need

  • Image Frame + Steering Angle Data (From Udacity, or Mod-able Video Game)
  • Python 3.5
  • OpenCV 3
  • Keras (TensorFlow 0.12 Backend)

Watch the Self Driving Car in Action

Here's the model in action. It stays mostly straight, but making the sharp turns without going off-road is the sign of a well balanced and generalized model. Your car should drive like this when you finish the lesson!

Reading the Data

The first step is to read the data. There is no need to perform extensive image preprocessing, in my opinion, because the images are not very noisy.  We are going to use all three image channels, but reverse the order becuase OpenCV reads as BGR, while Numpy reads as RGB. In the code below, we are simply importing the necessary python libraries and creating a few helper methods to use later on. 

import os, cv2, random, json
import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

from keras.models import Sequential
from keras.layers import Input, Dropout, Flatten, Convolution2D, MaxPooling2D, BatchNormalization, Dense
from keras.optimizers import RMSprop
from keras.callbacks import ModelCheckpoint, Callback, EarlyStopping
from keras import backend as K

ROWS = 160
COLS = 320
DIR = 'data/IMG/'

def img_id(path):
    return path.split('/IMG/')[1]

def read_image(path):
    """Read image and reverse channels"""
    img = cv2.imread(path, cv2.IMREAD_COLOR)
    return img[:,:,::-1]

#### Use to load ALL data into memory (See next section for python generator) 

X_all = np.ndarray((n_samples, ROWS, COLS, CHANNELS), dtype=np.uint8)

for i, path in enumerate(image_paths):
    img = read_image(DIR+path)
    X_all[i] = img

### Create steering angle labels

data = pd.read_csv('data/driving_log.csv', header=None, 
                   names=['center', 'left', 'right', 'angle', 'throttle', 'break', 'speed'])

y_all = data.angle.values

Use Keras fit_generator to Yield Images in Batches

This Section is Locked!

Unlock this lesson for $7 to view all sections.

Keras Convolutional Neural Network for Steering Angles

Our model is closely related to the Nvidia model developed in End to End Learning for Self-Driving Cars. It's a pretty basic convolutional neural network that gets flattened down to two fully connected layers. Let's create a function that holds the layers for our model and compiles it when called. 

def get_model():
    """Define hyperparameters and compile model"""
    lr = 0.0001
    opt = RMSprop(lr)
    loss = 'mean_squared_error'

    model = Sequential()
    model.add(BatchNormalization(mode=2, axis=1, input_shape=(ROWS, COLS, CHANNELS)))
    model.add(Convolution2D(3, 3, 3, init=weight_init, border_mode='valid', activation='relu', input_shape=(ROWS, COLS, CHANNELS)))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Convolution2D(9, 3, 3, init=weight_init, border_mode='valid', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Convolution2D(18, 3, 3, init=weight_init, border_mode='valid', activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Convolution2D(32, 3, 3, init=weight_init, border_mode='valid',  activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Dense(80, activation='relu', init=weight_init))
    model.add(Dense(15, activation='relu', init=weight_init))
    model.add(Dense(1, init=weight_init, activation='linear'))

    model.compile(optimizer=opt, loss=loss)

    return model

model = get_model()

Model Summary from Keras

If you run model.summary(), you should get the following output from Keras.

Fitting the Model, Saving Weights, and Early Stopping

Because we are use a python generator to read the data, we need to preload some validation data. We could also use a python generator for this, but for the sake of simplicity, we will just split the data with SKLearn. 

Also, make note of the callbacks being passed to the fit_generator function. The ModelCheckpoint will save the weights from the best epoch, while the EarlyStopping callback will end the model if the validation loss stops improving for X number of epochs.

nb_epoch = 30
batch_size = 64

### Creating Validation Data
X_train, X_test, y_train, y_test = train_test_split(
    X_all, y_all, test_size=0.20, random_state=23)

# Callbacks
early_stopping = EarlyStopping(monitor='val_loss', patience=8, verbose=1, mode='auto')   
save_weights = ModelCheckpoint('model.h5', monitor='val_loss', save_best_only=True)

model.fit_generator(fit_gen(data, batch_size),
        samples_per_epoch=data.shape[0], nb_epoch=nb_epoch, 
        validation_data=(X_test, y_test), callbacks=[save_weights, early_stopping])

preds = model.predict(X_test, verbose=1)

print( "Test MSE: {}".format(mean_squared_error(y_test, preds)))
print( "Test RMSE: {}".format(np.sqrt(mean_squared_error(y_test, preds))))

Extra Tips for Models that are Just Not Working

This Section is Locked!

Unlock this lesson for $7 to view all sections.

Saving the Keras Model to JSON

Now that your model is trained, its time to save it to JSON so it can be recompiled with the saved weights for real-time predictions. 

Piece of cake!

with open('model.json', 'w') as outfile:
    json.dump(model.to_json(), outfile)

If you have any questions, or need help building this model, feel free to message me. Remember, never drink and train self-driving cars!

Signup and Unlock for $7


Nobody has graded this lesson yet.

n 0.0%
Technology Data Analysis

  • 15 Unlocks
  • 1051 Total Reads
  • 36 minutes Est. Learning Time
Top Seller