The use of AI in real world and robotics have always been a wonder to me. Inspired by YouTube-famous maker Mark Rober (go check him out if you haven’t already), I always enjoy the process of getting my hands dirty and building engineering side projects that can be fun & learning something new along the way. What if using approaches in reinforcement learning (RL), that can train an agent make decisions adaptive to changes in the environment, and a donkey car, a hobbyist self driving car toolkit, to build something cool, say a trash collecting robot?

So that’s what I decided to do, to embark on this multi months project involving many moving parts, as an opportunity to learn about latest RL research, robotics and how papers in academia can be applied to our physical world.

This article serves the purpose for me to bounce off some ideas and track my progress along the way.

Preparation — the car

Donkey car — the hardware. I ordered one with Nvidia GPU (Jetson Nano) onboard from their online store, and begin assembling the car. This reminds me of the days taking electronic courses during undergrad.

For details refer to the donkey car website, but basically the steps are:

1. Assemble the chassis, attach the computer board to the platform and to the RC car securely via lots of screwdrivers.
2. connect the camera module to the chassis slot, attach the cables carefully to the board via the latch under the heatsink of the GPU
3. Attach the WIFI antennas to the board
4. Connect the power cables from battery to a power controller to convert to appropriate voltages to drive the motor electronic controller & the board, make sure not to burn your board
5. Attach the PWM controller signals to the brushless motors, make sure to connect the correct output pin to the controller — pins corresponding to motor speed, spinning direction (forward & backward), steer angle of the front wheels

These are steps at a high level and should arrive at this futuristic looking car below:

Donkey car — the software. Next flash a ubuntu image into a micro SD card and insert it to the card slot of the Jetson Nano. The image can be downloaded from the official Nvidia site, but the donkey car team also provide the image with additional binaries & runtimes for running CUDA, python & other ML libraries etc. which saves a lot of time. Also make sure to setup the WIFI connection on boot-up, allow ssh to connect to the board and that the OS can recognise & talk to the camera module. With that done, you’re all set with a full fledge computer, connected to the internet and can stream YouTube videos smoothly.

Donkey car — finetune. With the software & hardware taken care of, it’s time to link it up so that the car can be controlled with code. Donkey car provides a command line tool and library that can bootstrap the project code, think of something like a create- react-app, which makes life so much easier. With a command a small web server can be booted up which provides a simple web console to control the car , and a small window showing what the camera sees. At this point we can finetune the frequencies for PWM and other configs by driving the car a few times which in turn control the maximum speed and steering angles signals provided to the electronic controller of the motor.

A nice to have (which is strong recommended) is to pair a PS4 controller to the car via Bluetooth if you have one. Controlling with the joystick instead of through the web console makes adjusting the steering and throttle much more precise. The car will begin logging control inputs as well as camera feed frame by frame once you start driving. This data can then be used as training data to the machine learning models. Seeing the car come to life for the very first time is so rewarding, after all the work of assembling the car and setting things up. This also gives a small boost in confidence that I’m on the right path. At this point the car is fully functioning as a RC car that you can bring to a car race,

Next are the possiblity (still to be figured out) that laid out the ground work for training the car to be self driving:

• training environment: one of the contributors have made a donkey car environment (built with Unity I think) that simulate different race tracks & scenarios, this can be potentially use to train self-driving model and even modify on top to generate trash objects to actually train the trash collecting robot that I initially set out to do.

Reinforcement Learning — the model for controlling the car

Given the many subfields in machine learning — one that suits this setting the most is reinforcement learning — which an agent can learn by interacting with the environment in a feedback loop, seems to be most viable solution, framing trash collecting robot as an optimal control problem. Other related studies that focus on robotics such as meta-learning & multi-task learning might provide insights as well.

At this point my understanding towards RL is pretty much at the surface level, despite reading a few key papers in the field and have trained neural nets in the past (object recognition, transfer learning of representation from vision to audio (see Zamplfiy project)). After all what better way to understand the latest research than to try replicating one of the key papers?

After months of learning (and still going on), I can’t help but to be amazed by the brilliance and amount of work people have put into the field. This reminds me of the AlphaGo & DQN playing Atari games moment that first got me interested in AI. I wonder if one day I can contribute to this body of research.

By following the line of research / inspirations from one paper to the other, one gets to compare the different approaches researches took, the key contributions, the limitations, the trade-offs, the technical difficulty that are trying to overcome (sparse reward, reward shaping, sample efficiency, exploration vs exploitation etc) and different environment settings where the model can perform well (e.g. DQN for discrete action space, PPO & SAC for continuous action space), even can compare how some techniques get borrowed over for different purpose (KL divergence in PPO to regularize the policy to not deviate too much from the past iteration vs Entropy in SAC to encourage exploration of the agent), the use of some techniques that have been helpful for computer vision & NLP might not be useful for RL setting (batch norm, drop out) while some techniques are specifically useful for RL (target networks for critics, experience replay)

At the end I decided to try replicating Soft Actor Critic published around 2018, a model that is fairly robust & sample efficient comparing to its model-free peers. The 2nd version of the paper have even done more experiments in the real world and is able to train a Minitaur robot to walk on different ‘terrains’. Code is available here (https://github.com/MarShin/mycar). I use PyTorch for development and Weights & Bias to track the experiments hyperparams used and results (https://wandb.ai/marshin/trashbot-sac). Training SAC on a Nvidia GTX 1080 TI GPU machine is like a minimum requirement these days for RL tasks and so far the model seems to be able to converge for Hopper-v2 gym environment in 40 mins. But for env such as ant and half-cheetah with a larger action space the model seems to struggle and the episode reward plateaus even I’ve been training it for 700 steps. Or maybe I just have to be a little more patient given the papers report a good performance for these tasks only after training for >1 million steps. This is where discussing with fellow researchers & some guidance would be really helpful. I do understand why machine learning & especially RL type of problems are particularly hard to debug, as even the code runs fine and the agent seems to be learning something, there’s little way to tell if its suboptimal or get stuck by just looking at the gradient plots.

Next steps

• RL side: Now that SAC seems to work at some level, I would like to learn more about other multi-task or meta-learning approaches such as Hindsight Experience Replay (HER) and Model-Agnostic Meta-Learning (MAML) and their successors and maybe those approaches can work well even with less compute like i have right now
• RL side: Use pytorch-lightning to better organize the research code from the engineering code & hopefully save time by writing less boilerplate
• Donkey car Unity environment: So far I’ve only play around with the environment with my keyboard controls (like a car game) and yet to test out my model on this environment. Would be fun to see if its able to perform basic line following with the SAC model. And if possible to enhance the environment to add trash objects and change the terrain to see how it will perform on beach.