Protodriver: The Story
My hands-on dive into the world of autonomous racing
All I wanted was to make a video game racing driver that drives faster than me using nothing but the screen as input. How hard could it be?
Pretty hard, it turns out.
Image by Father of Lies (Luna) is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0
Getting up and running
The software engineering part wasn’t too difficult. The basic setup is few dozen lines of code and can be explained in two short tutorials (see Protodriver Part 1 and Part 2). The rest of the engineering went into infrastructure to make the development workflow go smoother. This included things like training from multiple save files and being able to pause mid-training.
Getting it to drive in a reasonable manner was fairly straightforward. It’s just a supervised classifier that predicts which way to move based on the training data. The models trained this way were able to get around the track, mostly stay in lanes, and even get unstuck using reverse after hitting a wall. Getting beyond this point, however, was where the difficulty lied. Using supervised learning, I’d only be able to get Protodriver to drive as fast as the data it was trained on. For a faster driver, I’d have to use a physics-based approach or reinforcement learning (RL). I opted for RL because it opens the door for essentially unlimited learning compared to a more traditional-robotics-eque model.
Picking up speed
Before this point, I had never implemented an RL model. I spent some time researching the history of RL and looking into different models/architectures and python implementations. I decided on deep Q learning (DQN) based on Reinforcement Learning w/ Keras + OpenAI: DQNs, a tutorial by Yash Patel. It was simple enough to copy the code over and adjust based on Protodriver’s needs.
Get Andrew Washington’s stories in your inbox
Join Medium for free to get updates from this writer.
Any RL model needs a reward function, and the obvious thing to maximize is speed. I wanted something that had little dependence on the specific game being played, so taking the speed or location from the game itself was out of the question. Instead I’d pull something from the screen itself. I could read the speedometer itself, but that would be too easy. The human brain estimates speed by calculating “optical flow”, so I figured I could do something in the same vein. As it turns out, there are functions to calculate optical flow already implemented in OpenCV, which I was already using to capture the screen. I made the first reward function optimize average optical flow over the last few frames.
The wall
This optical flow method started out strong as the agent learned, but it quickly devolved. The issue was made clear after the car hit the wall for the first time. Protodriver liked hitting walls. It became addicted. You see, every time the car hit the wall, the game’s camera shook. That camera shake caused an optical flow rate larger than taking the car to it’s top speed. I was able to clean the signal up with some pretty standard signal processing. First I calculated an expected range and removed outliers. I also measured the optical flow rate of hitting a wall and clipped all values over that threshold.
But even with the reward function cleaned up, the agent didn’t learn much. After a full night of training, the agent had only completed a single lap, and that was pretty much by random chance.
Trying things out
At this point, I realized this was bigger task than just a weekend project. I started looking into how other autonomous drivers work to see if there was a quick fix. I watched videos upon videos, read papers and blog posts, and went through the documentation of DonkeyCar and F1Tenth. I decided to see what I could do with some quick fixes, and after that I’d take a break from this project.
I looked back for places of improvement. The first was the reward function. I noticed there were issues getting a steady read on optical flow. Even the valid values were noisy and the range from slow to fast speed had a small range of optical flow values. I decided to try a different reward function. I used Tesseract to read the speed values from the on-screen speedometer. This worked, but didn’t have a major improvement on actual speed.
Next I started on a baseline. I figured I could learn something by getting a model to just get around the track. Plus it’d be nice to see what types of lap times a simple model could put down. I implemented a simple Ralph clone heavily based on Welch Labs videos and code, but after spending some time tweaking parameters, I realized it was probably time for a break…
What’s next?
And this brings me to now. I’ve taken a break and am ready to get started on this or some other project. I think I’m going to start out with a different project. I’ve been playing quite a bit of music lately, so that might be the next theme hmm… But if you’re reading this and have any ideas on where to pick this project up, feel free to comment below or reach out to me directly by email, LinkedIn, or GitHub.
Did you enjoy this? Feel free to contact me at awashington07@gmail.com for questions and comments
