An augmentation based deep neural network approach to learn human driving behavior
Overview
In this post, we will go over the work I did for project 3 of Udacity’s self-driving car project, behavior cloning for driving. The main task is to drive a car around in a simulator on a race track, and then use deep learning to mimic the behavior of human. This is a very interesting problem because it is not possible to drive under all possible scenarios on the track, so the deep learning algorithm will have to learn general rules for driving. We must be very careful while using deep learning models, because they have a tendency to overfit the data. Overfitting refers to the condition where the model is very sensitive to the training data itself, and the model’s behavior does not generalize to new/unseen data. One way to avoid overfitting is to collect a lot of data. A typical convolutional neural network can have up to a million parameters, and tuning these parameters requires millions of training instances of uncorrelated data, which may not always be possible and in some cases cost prohibitive. For our car example, this will require us to drive the car under different weather, lighting, traffic and road conditions. One way to avoid overfitting is to use augmentation.
Augmentation refers to the process of generating new training data from a smaller data set such that the new data set represents the real world data one may see in practice. As we are generating thousands of new training instances from each image, it is not possible to generate and store all these data on the disk. We will therefore utilize keras generators to read data from the file, augment on the fly and use it to train the model. We will utilize images from the left and right cameras so we can generate additional training data to simulate recovery. Keras generator is set up such that in the initial phases of learning, the model drops data with lower steering angles with higher probability. This removes any potential for bias towards driving at zero angle. After setting up the image augmentation pipeline, we can proceed to train the model. The training was performed using simple adam learning algorithm with learning rate of 0.0001. After this training, the model was able to drive the car by itself on the first track for hours and generalized to the second track.
All the training was based on driving data of about 4 laps using ps4 controller on track 1 in one direction alone. The model never saw track 2 in training, but with image augmentation (flipping, darkening, shifting, etc) and using data from all the cameras (left, right and center) the model was able to learn general rules of driving that helped translate this learning to a different track.
IMPORTANT: These results were obtained on Titan X GPU machine I built earlier. Full specifications of the computer can be found here. Please note that computers with different performance will provide a different performance of the network.
- Augmentation:
Augmentation helps us extract as much information from data as possible. We will generate additional data using the following data augmentation techniques. Augmentation is a technique of manipulating the incoming training data to generate more instances of training data. This technique has been used to develop powerful classifiers with little data. https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html . However, augmentation is very specific to the objective of the neural network.
Brightness augmentation
Changing brightness to simulate day and night conditions. We will generate images with different brightness by first converting images to HSV, scaling up or down the V channel and converting back to the RGB channel.
def augment_brightness_camera_images(image):
image1 = cv2.cvtColor(image,cv2.COLOR_RGB2HSV)
image1 = np.array(image1, dtype = np.float64)
random_bright = .5+np.random.uniform()
image1[:,:,2] = image1[:,:,2]*random_bright
image1[:,:,2][image1[:,:,2]>255] = 255
image1 = np.array(image1, dtype = np.uint8)
image1 = cv2.cvtColor(image1,cv2.COLOR_HSV2RGB)
return image1
Using left and right camera images
Using left and right camera images to simulate the effect of car wandering off to the side, and recovering. We will add a small angle .25 to the left camera and subtract a small angle of 0.25 from the right camera. The main idea being the left camera has to move right to get to center, and right camera has to move left.
Horizontal and vertical shifts
We will shift the camera images horizontally to simulate the effect of car being at different positions on the road, and add an offset corresponding to the shift to the steering angle. We added 0.004 steering angle units per pixel shift to the right, and subtracted 0.004 steering angle units per pixel shift to the left. We will also shift the images vertically by a random number to simulate the effect of driving up or down the slope.
def trans_image(image,steer,trans_range):
# Translation
tr_x = trans_range*np.random.uniform()-trans_range/2
steer_ang = steer + tr_x/trans_range*2*.2
tr_y = 40*np.random.uniform()-40/2
#tr_y = 0
Trans_M = np.float32([[1,0,tr_x],[0,1,tr_y]])
image_tr = cv2.warpAffine(image,Trans_M,(cols,rows))
return image_tr,steer_ang
Shadow augmentation
The next augmentation we will add is shadow augmentation where random shadows are cast across the image. This is implemented by choosing random points and shading all points on one side (chosen randomly) of the image. The code for this augmentation is presented below.
def add_random_shadow(image):
top_y = 320*np.random.uniform()
top_x = 0
bot_x = 160
bot_y = 320*np.random.uniform()
image_hls = cv2.cvtColor(image,cv2.COLOR_RGB2HLS)
shadow_mask = 0*image_hls[:,:,1]
X_m = np.mgrid[0:image.shape[0],0:image.shape[1]][0]
Y_m = np.mgrid[0:image.shape[0],0:image.shape[1]][1]shadow_mask[((X_m-top_x)*(bot_y-top_y) -(bot_x - top_x)*(Y_m-top_y) >=0)]=1
#random_bright = .25+.7*np.random.uniform()
if np.random.randint(2)==1:
random_bright = .5
cond1 = shadow_mask==1
cond0 = shadow_mask==0
if np.random.randint(2)==1:
image_hls[:,:,1][cond1] = image_hls[:,:,1][cond1]*random_bright
else:
image_hls[:,:,1][cond0] = image_hls[:,:,1][cond0]*random_bright
image = cv2.cvtColor(image_hls,cv2.COLOR_HLS2RGB)return image
Flipping
In addition to the transformations above, we will also flip images at random and change the sign of the predicted angle to simulate driving in the opposite direction.
2. Preprocessing
After augmenting the image as above, we will crop the top 1/5 of the image to remove the horizon and the bottom 25 pixels to remove the car’s hood. Originally 1/3 of the top of car image was removed, but later it was changed to 1/5 to include images for cases when the car may be driving up or down a slope. We will next rescale the image to a 64X64 square image. After augmentation, the augmented images looks as follows. These images are generated using kera’s generator, and unlimited number of images can be generated from one image. I used Lambda layer in keras to normalize intensities between -.5 and .5.
new_size_col,new_size_row = 64, 64def preprocessImage(image):
shape = image.shape
# note: numpy arrays are (row, col)!
image = image[math.floor(shape[0]/5):shape[0]-25, 0:shape[1]]
image = cv2.resize(image,(new_size_col,new_size_row), interpolation=cv2.INTER_AREA)
#image = image/255.-.5
return imagedef preprocess_image_file_train(line_data):
i_lrc = np.random.randint(3)
if (i_lrc == 0):
path_file = line_data['left'][0].strip()
shift_ang = .25
if (i_lrc == 1):
path_file = line_data['center'][0].strip()
shift_ang = 0.
if (i_lrc == 2):
path_file = line_data['right'][0].strip()
shift_ang = -.25
y_steer = line_data['steer_sm'][0] + shift_ang
image = cv2.imread(path_file)
image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
image,y_steer,tr_x = trans_image(image,y_steer,100)
image = augment_brightness_camera_images(image)
image = preprocessImage(image)
image = np.array(image)
ind_flip = np.random.randint(2)
if ind_flip==0:
image = cv2.flip(image,1)
y_steer = -y_steer
return image,y_steer
3. Keras generator for subsampling
Get Vivek Yadav’s stories in your inbox
Join Medium for free to get updates from this writer.
As there was limited data and we are generating thousands of training examples from the same image, it is not possible to store all the images apriori into memory. We will utilize kera’s generator function to sample images such that images with lower angles have lower probability of getting represented in the data set. This alleviates any problems we may ecounter due to model having a bias towards driving straight. Panel below shows multiple training samples generated from one image. The keras generator is presented below. The ‘pr_threshold’ variable is a threshold that determines if a data with small angle will be dropped or not.
def generate_train_from_PD_batch(data,batch_size = 32):
batch_images = np.zeros((batch_size, new_size_row, new_size_col, 3))
batch_steering = np.zeros(batch_size)
while 1:
for i_batch in range(batch_size):
i_line = np.random.randint(len(data))
line_data = data.iloc[[i_line]].reset_index()
keep_pr = 0
#x,y = preprocess_image_file_train(line_data)
while keep_pr == 0:
x,y = preprocess_image_file_train(line_data)
pr_unif = np.random
if abs(y)<.1:
pr_val = np.random.uniform()
if pr_val>pr_threshold:
keep_pr = 1
else:
keep_pr = 1
#x = x.reshape(1, x.shape[0], x.shape[1], x.shape[2])
#y = np.array([[y]])
batch_images[i_batch] = x
batch_steering[i_batch] = y
yield batch_images, batch_steering4. Model Architecture and training
I implemented the model architecture above for training the data. The first layer is 3 1X1 filters, this has the effect of transforming the color space of the images. Research has shown that different color spaces are better suited for different applications. As we do not know the best color space apriori, using 3 1X1 filters allows the model to choose its best color space. This is followed by 3 convolutional blocks each comprised of 32, 64 and 128 filters of size 3X3. These convolution layers were followed by 3 fully connected layers. All the convolution blocks and the 2 following fully connected layers had exponential relu (ELU) as activation function. I chose leaky relu to make transition between angles smoother.
Training:
I trained the model using the keras generator with batch size of 256 for 8 epochs. In each epoch, I generated 20000 images. I started with pr_threshold, the chance of dropping data with small angles as 1, and reduced the probability by dividing it by the iteration number after each epoch. The entire training took about 5 minutes. However, it too more than 20 hours to arrive at the right architecture and training parameters. Snippet below presents the result of training.
5. Model performance:
Video below shows the performance of algorithm on the track 1 on which the original data was collected. The car is able to drive around for hours, we will next look into the case where either the camera resolution, video size or tracks are changed.
Generalization from one image size to another
Video below presents generalization from one image size to another. I used the same pretrained model and tested it on all the other image sizes and found that the deep learning neural network was able to drive the car around for all image sizes.
Generalization from one image resolution to another
Video below presents generalization from one image resolution to another. I used the same pretrained model and tested it on all the other image resolutions and found that the deep learning neural network was able to drive the car around for all image resolutions.
I also tested different combinations of image size and image resolutions, and on track 1 the deep learning algorithm was able to drive the car around for all combinations of image resolution and sizes.
Generalization from one track to another
Figure below presents generalization from one track to another. This was perhaps the toughest test for the deep learning algorithm. In the second track, there were more right turns and u-turns, it was darker, and the road had slopes. All of which were absent in the original track. However, all these effects were artificially included into the model via image augmentation.
6. Future Directions
This project is far from over. This project opened more questions than it answered. A few more things to try are,
- Check how pretrained models perform compared to the current model.
- Include a parallel network to learn any track-specific behavior. John Chen implemented an agile trainer for training the car on the fly. Rationale being instead of recording recovery data, we let the car make mistakes and train on a need-basis. Its a very powerful tool to have, but its best to avoid using it until absolutely necessary. I specifically did not choose this approach based on my past control theory knowledge. These schemes do not learn if the error is zero, and have a potential to learn features specific to the training data itself, instead of learning general driving rules. This is analogous to dynamic richness in adaptive control. The parameter update in adaptive control are need-based and do not generalize to other control tasks. A good alternative is to use linear models of nonlinear functions that are updated on the fly, and are discarded once we move to a different task.
- Another interesting thing to try is to test how well this model performs on real-world data. This is on my things to do for future.
- Try additional augmentation schemes, such as placing ‘fake’ cars on the road ahead, occlusion, etc.
6. Reflections
This was perhaps the weirdest project I did. This project challenged all the previous knowledge I had about deep learning. In general large epoch size and training with more data results in better performance, but in this case any time I got beyond 10 epochs, the car simply drove off the track. Although all the image augmentation and tweaks seem reasonable n0w, I did not think of them apriori. I hope others find this post useful, and get inspried to try novel things. I havent used on-the fly training agile trainer by John Chen yet. I wanted to try and stretch the data as much as possible. Next thing to try is to experiment with parallel network using John’s trainer.
Acknowledgements:
I am very thankful to Udacity for selecting me for the first cohort, this allowed me to connect with many like-minded individuals. As always, learned a lot from discussions with Henrik Tünnermann and John Chen. I am also thankful for getting the NVIDA’s GPU grant. Although, its for work, but I use it for Udacity too.
