The post covers the following:

• What are MobileNets?
• How to build a custom dataset to train a MobileNet with TensorFlow
• How to train a MobileNet that’s pretrained on ImageNet with TensorFlow
• How MobileNets perform against Inception V3
• How to use your retrained MobileNet to classify images

Read the post at Hacker Noon.

We built insanely fast image classifiers with MobileNets in TensorFlow was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

In part 1, Creating Insanely Fast Image Classifiers with MobileNet in TensorFlow, we covered how to retrain a MobileNet on a new dataset. Specifically, we trained a classifier to detect Road or Not Road at more than 400 frames per second on a laptop.

MobileNets are made for — wait for it — mobile devices. So, let’s move our road not road model to an Android app so we can see it in action.

Goals and Plan

Let’s set some constraints so we have something specific to shoot for. We’ll attempt to:

• Retrain a MobileNet on a very small amount of purpose-built data
• Achieve 95% classification accuracy on a hold out test set
• Use less than 5% of a $300 device’s CPU while running inference

To do that, we’ll follow these steps:

1. Generate a new training dataset
2. Train several MobileNet configurations to find the smallest net that will hit our accuracy target
3. Get benchmarks by running Inception V3 on Android
4. Update the TensorFlow Android example app to use our MobileNet model
5. Try it in the wild
6. Tune it to get below 5% CPU usage

Building the Dataset

In the previous post, we were classifying road/not road generally, so we pulled images from several sources. Now we’re going to drill down on the problem a bit more. If you recall, the purpose for this project is for user privacy: At Coastline, we’re building driving safety features for mobile devices that make use of the camera. So when someone turns on the app, we want to validate that what we’re looking at is a road. If we aren’t, we’ll disable recording.

So to build our training set, I’m going to walk around doing everyday things while recording video: Around my house, outside my car, inside my car fiddling with the radio, petting the cats, etc. This will be our “not road” training data.

For our “road” data, I’m going to sample randomly from the Coastline driving dataset, which is from a camera facing out the front of a car.

With 3,000 examples from each set, we’re ready to train.

Training MobileNet on our custom dataset

The next step is to see what sort of accuracy we can get from the different MobileNet configurations. We’ll start with training the widest one: MoileNet 1.0 @ 128. And because we’re going to put this on a mobile device, we’ll use quantized weights, which will reduce the model memory footprint even further.

For the details on how to retrain MobileNet on your own data, be sure to check out my previous post.

From the root TensorFlow folder, we’ll run:

python tensorflow/examples/image_retraining/retrain.py \
  --image_dir ~/ml/blogs/road-not-road/data/ \
  --learning_rate=0.0005 \
  --testing_percentage=15 \
  --validation_percentage=15 \
  --train_batch_size=32 \
  --validation_batch_size=-1 \
  --flip_left_right True \
  --random_scale=30 \
  --random_brightness=30 \
  --eval_step_interval=100 \
  --how_many_training_steps=1000 \
  --architecture mobilenet_1.0_128_quantized

After 1,000 training steps, we achieve 99.7% accuracy on our hold out set. Wow! Apparently MobileNets are pretty good at classifying roads.

Here are a couple images it miss-classified:

Now let’s do the same thing, but with the smallest MobileNet: 0.25 @ 128, quantized. After 1,000 training steps, we get to 92.6%. Doesn’t satisfy our accuracy target.

How about something a little wider, say 0.5?

95.0%! And the final model is just 1.6 MB. Looks like our money shot, if just barely. (It should be noted that this entire model is trained on just 10 minutes of video captured at 10fps. There is a lot of room for improvement by piling on the data.)

Let’s give it a quick try to make sure it’s working as expected:

python tensorflow/examples/label_image/label_image.py \ 
  --graph=/tmp/output_graph.pb \
  --labels=/tmp/output_labels.txt \
  --image=/home/harvitronix/ml/blogs/road-not-road/test-image.jpg \
  --input_layer=input \
  --output_layer=final_result \
  --input_mean=128 \
  --input_std=128 \
  --input_width=128 \
  --input_height=128

And since our headline has the words “insanely fast” in it, how fast can we run this on an NVIDIA GeForce 960m GPU on our laptop? It runs through 1,000 images in just 3.36 seconds. That’s 297.6 frames per second!

Using our MobileNet model in an Android app

Now that we have a model that’s tiny, fast and accurate enough for our use case, let’s load it up in an Android app so we can test it in the real world.

Don’t have a model trained yet? Download the model I trained on the data described above. It contains both the .pb and label files. Extract and follow the instructions below.

Sticking with our theme of using tools provided by TensorFlow (how awesome is that, btw?), we’ll make use of the Android example project to get this thing running in no time flat.

Getting and building the project

If you haven’t already, go ahead and clone the TensorFlow repo:

git clone https://github.com/tensorflow/tensorflow.git --depth 1

You’ll find an Android project ready-made for doing this kind of task in the tesnorflow/examples/android folder. Open the folder in Android Studio, build it, load the APK on your phone, and you’ve got an image classifier that uses the Inception V3 model trained on ImageNet, ready to tell apart your cat from a platypus.

If you have trouble building the app, be sure to take a look at the instructions in the TensorFlow Android ReadMe. My biggest challenge was the NDK version. Downgrading to r12b solved my problems.

Inception speed benchmark

Since we’ve already got Inception running on the app, let’s take some measurements so we can compare it against our MobileNet.

The Inception model that comes with the example project is 53.9 mb. That’s versus our MobileNet that’s just 1.6 mb! It runs at about 240ms per inference, or 4 frames per second (fps). And it uses about 40% of the CPU.

Let’s try it at 1fps:

Still up over 35%. Let’s hope our MobileNet can do better than that, or we’re not going to get anywhere near our goal of max 5% usage.

Switching to MobileNet

Now let’s make a couple minor changes to the Android project to use our custom MobileNet model.

First, copy your model and labels into the project’s assets folder. Mine were at /tmp/output_graph.pb and /tmp/output_labels.txt.

Next, open up ClassifierActivity, which can be found in:

tensorflow/examples/android/src/org/tensorflow/demo/ClassifierActivity.java

You’ll want to update the constants at the top of the file to define the settings for our new model. It looks like this when you first open it:

private static final int INPUT_SIZE = 224;
private static final int IMAGE_MEAN = 117;
private static final float IMAGE_STD = 1;
private static final String INPUT_NAME = "input";
private static final String OUTPUT_NAME = "output";

private static final String MODEL_FILE = "file:///android_asset/tensorflow_inception_graph.pb";
private static final String LABEL_FILE =
    "file:///android_asset/imagenet_comp_graph_label_strings.txt";

Change it to:

private static final int INPUT_SIZE = 128;
private static final int IMAGE_MEAN = 128;
private static final float IMAGE_STD = 128;
private static final String INPUT_NAME = "input";
private static final String OUTPUT_NAME = "final_result";

private static final String MODEL_FILE = "file:///android_asset/output_graph.pb";
private static final String LABEL_FILE =
    "file:///android_asset/output_labels.txt";

Hit run to build the project and load the APK on your device, and you’ve got your very own road / not road classifier!

The Results

Here’s a video of my road / not road app in action. I made a few tweaks to the UI to make it easier to see what’s going on:

So how fast is it, and what about CPU usage?

On my Xiaomi Mi5, this thing flies. It does inference in about 55ms, which is 18 frames per second! On a $300 Xiaomi Mi5s!

It is quite CPU intensive, though, using 25–30% when running at full-throttle. Makes sense, since we’re running it as fast as it will go.

We want to get down to under 5%, which we’ll do by reducing the frequency at which it runs, since our use case doesn’t need to run inference continuously to achieve our privacy objective discussed above. Changing it to classify once every 18 frames (so once a second, roughly) brings the average usage down to about 5.5%!

So our MobileNet model is 1/30th as large as Inception. It runs more than 3x faster per frame. And it uses far fewer CPU resources.

It’s safe to say me and MobileNets are going to be like like:

Enjoyed this post? Help others find it by hitting the little ❤. Thanks for reading!

Building an insanely fast image classifier on Android with MobileNets in TensorFlow was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

MobileNets are a new family of convolutional neural networks that are set to blow your mind, and today we’re going to train one on a custom dataset.

There are a few things that make MobileNets awesome:

1. They’re insanely small
2. They’re insanely fast
3. They’re remarkably accurate
4. They’re easy to tune for resources vs. accuracy

Why is this important? Many mobile deep learning tasks are actually performed in the cloud. When you want to classify an image, that image is sent to a web service, it’s classified on a remote server, and the result is sent back to your phone.

That’s changing quickly. The computational power on your phone is increasing rapidly, and the network complexity required for computer vision is shrinking (thanks to architectures like SqueezeNet and MobileNet).

Besides the clear benefit of AI without an internet connection, sending images to the cloud is impractical in speed-hungry situations, like in the vehicle safety apps we’re developing at Coastline.

With that, let’s learn the following:

• What are MobileNets?
• How to build a custom dataset to train a MobileNet with TensorFlow
• How to train a MobileNet that’s pretrained on ImageNet with TensorFlow
• How MobileNets perform against Inception V3
• How to use your retrained MobileNet to classify images

What are MobileNets?

MobileNets are a class of convolutional neural network designed by researches at Google. They are coined “mobile-first” in that they’re architected from the ground up to be resource-friendly and run quickly, right on your phone.

The main difference between the MobileNet architecture and a “traditional” CNN’s is instead of a single 3x3 convolution layer followed by batch norm and ReLU, MobileNets split the convolution into a 3x3 depthwise conv and a 1x1 pointwise conv. The details of why this is so significant can be found in the MobileNet paper, which I strongly encourage you to read.

So what’s the catch? Accuracy. MobileNets are not usually as accurate as the bigger, more resource-intensive networks we’ve come to know and love. But finding that resource/accuracy trade-off is where MobileNets really shine.

MobileNets surface two parameters that we can tune to fit the resource/accuracy trade-off of our exact problem: width multiplier and resolution multiplier. The width multiplier allows us to thin the network, while the resolution multiplier changes the input dimensions of the image, reducing the internal representation at every layer.

Google open-sourced the MobileNet architecture and released 16 ImageNet checkpoints, each corresponding to a different parameter configuration. This gives us an excellent starting point for training our own classifiers that are insanely small and insanely fast.

To learn more about how MobileNets work, read MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.

How to build your own dataset to train a MobileNet

Our challenge today is to build an image classifier that can detect if an image is a road or not a road. It’s like hot dog, not hot dog, but for roads.

Why road, not road? At Coastline, we’re developing safety features for your car with mobile apps that use computer vision. As with all vision problems, user privacy is critical. So one of the first checks we do when a user turns on the camera in our app is check if it sees a road. If it doesn’t, we turn off the camera. We want to be able to do this fast and at as little computational cost to the user as possible.

Since we’re tackling a custom problem, we need to start with creating our dataset. Our target is to collect 10,000 images, split roughly evenly 50/50 road/not road.

We’ll get data from a few different places:

• 4,000 obviously road images sampled randomly from the Coastline driving dataset
• 2,000 obviously not road images sampled randomly from the ImageNet dataset
• 3,000 less obviously not road scenics sampled from the internet to make sure the classifier doesn’t just learn sky, not sky
• 1,000 less obviously road scenics also sampled from the internet to make sure the classifier doesn’t recognize windshield reflections or other commonalities in the Coastline data

We’ll place each image into one of two folders, each representing the class of that image: road and not-road. That’s all we have to do to prepare our images for retraining!

However, while grabbing CC images from the internet is a great place to add diversity to your dataset, it comes with a drawback: the labels are noisy. For instance, an image found by searching “road landscape” could have a road front and center with a nice scene in the background, or it could be a mountain scene with a tiny road off in the distance.

We could solve for this by going through each image and hand-labeling it, but why do that when we have deep learning?! Instead, we retrain a big network (like Inception V3) on all our data, paying special attention not to over-fit on our training data by early stopping and heavy data augmentation. Then we run every image of our dataset (even those images we just used to train!) through the network and keep track of the images it classified incorrectly or with little confidence. Then we go through each of those images and move them to their proper classes, if applicable. This reduces the number of images we have to manually clean up significantly. And doing multiple passes of this technique helped us increase our accuracy by seven percentage points on Inception.

Now that we have 5,000 road images and 5,000 not road images, in a structure like this…

data/
  road/[images...]
  not-road/[images...]

…we’ll use TensorFlow and transfer learning to fine-tune MobileNets on our custom dataset.

How to retrain a MobileNet that’s pretrained on ImageNet

TensorFlow comes packaged with great tools that you can use to retrain MobileNets without having to actually write any code. This stuff is fresh off the presses: Retraining support for MobileNet was added less than a week ago!

If you don’t have it downloaded already, fork and/or clone the TensorFlow repo:

git clone https://github.com/tensorflow/tensorflow.git

Now you can use the scripts in the example folder to retrain MobileNet on your own data.

But wait! Which MobileNet should you use? That’s a good question. Let’s retrain a small assortment and see how they perform. To kick off training, we’ll run the following command from the root of the TensorFlow repo:

python tensorflow/examples/image_retraining/retrain.py \
    --image_dir ~/ml/blogs/road-not-road/data/ \
    --learning_rate=0.0001 \
    --testing_percentage=20 \
    --validation_percentage=20 \
    --train_batch_size=32 \
    --validation_batch_size=-1 \
    --flip_left_right True \
    --random_scale=30 \
    --random_brightness=30 \
    --eval_step_interval=100 \
    --how_many_training_steps=600 \
    --architecture mobilenet_1.0_224

The architecture flag is where we tell the retraining script which version of MobileNet we want to use. The 1.0 corresponds to the width multiplier, and can be 1.0, 0.75, 0.50 or 0.25. The 224 corresponds to image resolution, and can be 224, 192, 160 or 128. For example, to train the smallest version, you’d use --architecture mobilenet_0.25_128.

Some other important parameters:

• learning_rate: This is something you’ll want to play with. I found 0.0001 to work well.
• testing and validation percentage: The script will split your data into train/val/test for you. It will use train to train, val to give performance updates every “eval_step_interval”, and test will run after “how_many_training_steps” to give you your final score.
• validation_batch_size: Setting this to -1 tells the script to use all your data to validate on. When you don’t have a lot of data (like only 10,000 images), it’s a good idea to use -1 here to reduce variance between evaluation steps.

After retraining on several model architectures, let’s see how they compare.

Comparing MobileNet parameters and their performance against Inception

After just 600 steps on training Inception to get a baseline (by setting the — architecture flag to inception_v3), we hit 95.9%. Training took 18 minutes. (There is a lot of room for improvement here, but we don’t have all day!) The resulting checkpoint landed at 84mb. And doing a quick speed test by running 1,000 images through it shows it can classify images on an NVIDIA GeForce 960m GPU at ~19fps.

Aside: Why “only” 95.9% and not 100%? It seems like a pretty simple problem, right? Well, besides the ample tuning we could do to the training parameters (we actually achieved 98.9% with the same data using a different configuration in another go), it turns out the distinction between classes is a bit more subtle than it seems on the surface. Take these cases:

• The image is a one-lane dirt road in the woods: Is it a road or a trail? I don’t even know.
• It’s a landscape scenic with a road in the distance. Is it a picture or a road, or is there just a road in the picture? At what point does the scenic move classes?
• It’s an artsy tilt-shift photo of a couple holding hands in the foreground and a time-lapsed street in the background. Road or not road? Flip a coin.

So, how do the MobileNets perform? Not surprisingly, not quite as well. However, the tradeoff benefit is astounding.

Using the biggest MobileNet (1.0, 224), we were able to achieve 95.5% accuracy with just 4 minutes of training. The resulting model size was just 17mb, and it can run on the same GPU at ~135fps.

For those keeping score, that’s 7 times faster and a quarter the size. All for just 0.4% loss in accuracy.

How about the smallest MobileNet (0.24, 128), using quantized weights? Big accuracy tradeoff, achieving just 89.2%. But get this: 450 frames per second, and the model takes just 930kb of memory. That’s kilobytes!

Using your retrained MobileNet to classify images

Now that you have your MobileNet retrained on your custom dataset, it’s time to give it a try. Not surprisingly, TensorFlow comes with a script to do that, too.

python3 tensorflow/examples/label_image/label_image.py \ 
  --graph=/tmp/mobilenet_0.50_192.pb \
  --labels=/tmp/output_labels.txt \
  --image=/home/harvitronix/ml/blogs/road-not-road/test-image.jpg \
  --input_layer=input \
  --output_layer=final_result \
  --input_mean=128 \
  --input_std=128 \
  --input_width=192 \
  --input_height=192

Aside: It should be noted that in our fairly simple two-class problem, the accuracy trade-off is not that big. In the case of ImageNet with its 1,001 classes, the accuracy tradeoffs are much more significant. See the table here.

Next steps

Okay, so the whole point of MobileNets is to run on mobile, right? Stay tuned! In our next post, we’ll create some purpose-built training data, fine-tune again, and load our retrained MobileNet into an Android app. We’ll see how fast it can run on a mobile device and how accurate it is in the real world.

UPDATE: Part 2 is now live! Don’t miss Building an insanely fast image classifier on Android with MobileNets in TensorFlow.

Creating insanely fast image classifiers with MobileNet in TensorFlow was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

Training and evaluating these networks in series is fine if the dataset or parameter space are small. But what if you’re evolving a neural network with a genetic algorithm, like we did in our previous post? Or what if one train/eval step takes a day to complete? We need parallelism!

Here’s a super simple way to achieve distributed hyperparameter tuning using MongoDB as a quasi pub/sub, with a controller defining jobs to process, and N workers to do the training and evaluating.

It looks like this:

All the code is available on GitHub here: Super Simple Distributed Hyperparameter Tuning

The Controller

The system starts with a controller that creates the jobs. How you choose to create jobs is up to you. In our simple example, we just randomly create a two-layer MLP with a random number of units per layer. It goes something like this:

def main():
    """Toy example of adding jobs to be processed."""
    db = Database('blog-test')

    while True:
        print("Creating job.")
        network = gen_network(784, 10)  # mnist settings
        add_job('mnist', network, db)

        sleep_time = random.randint(60, 120)
        print("Waiting %d seconds." % (sleep_time))
        time.sleep(sleep_time)

The Database

Jobs are stored in a Mongo DB. We create a Database class with the following methods:

• insert_job — Add the job to the DB. The job document includes the dataset we want to train on (in our case, just mnist or cifar10), the Keras-generated JSON network definition, processing and processed flags, and an empty dictionary to hold the metrics
• find_job — Returns the first available job that isn’t currently being processed and sets a flag that indicates the job is being worked on
• score_job — Update the job to indicate it has been processed and add the resulting metrics

The DB needs to be remotely accessible to all the workers if you’re running this truly distributed. In its current state, the code only accepts a host argument since I run this on AWS instances that share a security group. If you want to connect to a remote DB, you’ll need to implement username/password (etc) arguments as well. (Pull requests welcome!)

The Workers

As in life, the workers in this setup do all the hard work. You can run as many workers as you’d like, though generally you’d want to limit to one worker per instance or GPU. When I use this to train my own models on my data, I use a cluster of eight AWS P2 instances, with one of the instances also running the controller and database.

The workflow is as follows:

1. Check the jobs collection to see if there’s anything to do
2. Once we have a job, get the dataset, compile the model, fit and score
3. Update the job in the collection with the new score
4. Repeat from step 1

Super simple, right? 👍

Here’s the code.

Want more posts like this? Give me a follow below and take a look at my other posts. Thanks for reading!

Super simple distributed hyperparameter tuning with Keras and Mongo was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building the perfect deep learning network involves a hefty amount of art to accompany sound science. One way to go about finding the right hyperparameters is through brute force trial and error: Try every combination of sensible parameters, send them to your Spark cluster, go about your daily jive, and come back when you have an answer.

But there’s gotta be a better way!

Here, we try to improve upon the brute force method by applying a genetic algorithm to evolve a network with the goal of achieving optimal hyperparameters in a fraction the time of a brute force search.

How much faster?

Let’s say it takes five minutes to train and evaluate a network on your dataset. And let’s say we have four parameters with five possible settings each. To try them all would take (5**4) * 5 minutes, or 3,125 minutes, or about 52 hours.

Now let’s say we use a genetic algorithm to evolve 10 generations with a population of 20 (more on what this means below), with a plan to keep the top 25% plus a few more, so ~8 per generation. This means that in our first generation we score 20 networks (20 * 5 = 100 minutes). Every generation after that only requires around 12 runs, since we don’t have the score the ones we keep. That’s 100 + (9 generations * 5 minutes * 12 networks) = 640 minutes, or 11 hours.

We’ve just reduced our parameter tuning time by almost 80%! That is, assuming it finds the best parameters…

Ok Google, what’s a genetic algorithm?

Genetic algorithms are commonly used to generate high-quality solutions to optimization and search problems by relying on bio-inspired operators such as mutation, crossover and selection. — Wikipedia

How do genetic algorithms work?

At its core, a genetic algorithm…

1. Creates a population of (randomly generated) members
2. Scores each member of the population based on some goal. This score is called a fitness function.
3. Selects and breeds the best members of the population to produce more like them
4. Mutates some members randomly to attempt to find even better candidates
5. Kills off the rest (survival of the fittest and all), and
6. Repeats from step 2. Each iteration through these steps is called a generation.

Repeat this process enough times and you should be left with the very best possible members of a population. Sounds like a lot evolution, right? Same deal.

Applying genetic algorithms to Neural Networks

We’ll attempt to evolve a fully connected network (MLP). Our goal is to find the best parameters for an image classification task.

We’ll tune four parameters:

• Number of layers (or the network depth)
• Neurons per layer (or the network width)
• Dense layer activation function
• Network optimizer

The steps we’ll take to evolve the network, similar to those described above, are:

1. Initialize N random networks to create our population.
2. Score each network. This takes some time: We have to train the weights of each network and then see how well it performs at classifying the test set. Since this will be an image classification task, we’ll use classification accuracy as our fitness function.
3. Sort all the networks in our population by score (accuracy). We’ll keep some percentage of the top networks to become part of the next generation and to breed children.
4. We’ll also randomly keep a few of the non-top networks. This helps find potentially lucky combinations between worse-performers and top performers, and also helps keep us from getting stuck in a local maximum.
5. Now that we’ve decided which networks to keep, we randomly mutate some of the parameters on some of the networks.
6. Here comes the fun part: Let’s say we started with a population of 20 networks, we kept the top 25% (5 nets), randomly kept 3 more loser networks, and mutated a few of them. We let the other 12 networks die. In an effort to keep our population at 20 networks, we need to fill 12 open spots. It’s time to breed!

Breeding

Breeding is where we take two members of a population and generate one or more child, where that child represents a combination of its parents.

In our neural network case, each child is a combination of a random assortment of parameters from its parents. For instance, one child might have the same number of layers as its mother and the rest of its parameters from its father. A second child of the same parents may have the opposite. You can see how this mirrors real-world biology and how it can lead to an optimized network quickly.

Now that we understand how it works, let’s actually do it.

Dataset

We’ll use the relatively simple but not easy CIFAR10 dataset for our experiment. This dataset gives us a big enough challenge that most parameters won’t do well, while also being easy enough for an MLP to get a decent accuracy score.

Note: A convolutional neural network is certainly the better choice for a 10-class image classification problem like CIFAR10. But a fully connected network will do just fine for illustrating the effectiveness of using a genetic algorithm for hyperparameter tuning.

Code explained

Hopefully most of the code is self-explanatory and well-documented. (Famous last words, I know.) Here are parts of the optimizer.py module, which holds the meat of the genetic algorithm code. Note that this code is heavily inspired by an excellent post by Will Larson, Genetic Algorithms: Cool Name & Damn Simple.

Results

We’ll start by running the brute force algorithm to find the best network. That is, we’ll train and test every possible combination of network parameters we provided.

Our top result using brute force achieved 56.03% accuracy with the following parameters:

• Layers: 2
• Neurons: 768
• Activation: elu
• Optimizer: adamax

This took 63 hours to run. Yikes.

Now we’ll run our genetic algorithm, starting with a population of 20 randomly initialized networks, and we’ll run it for 10 generations.

Final score? 56.56% accuracy! Time? Seven hours. And the resulting network? Almost identical:

• Layers: 2
• Neurons: 512
• Activation: elu
• Optimizer: adamax

The only difference is the genetic algorithm preferred 512 to 768 neurons. (In the brute force run, the 512 network achieved 55.65%. Should’ve set a random seed.)

So what’s the big deal? The genetic algorithm gave us the same result in 1/9th the time! Seven hours instead of 63. And it’s likely that as the parameter complexity increases, the genetic algorithm provides exponential speed benefit.

What’s next?

I’m looking forward to applying this type of hyperparameter tuning to a much more complex problem and network. That will probably require distributed training, so perhaps we tackle that next!

Want the code? It’s all right here on GitHub.

Let’s evolve a neural network with a genetic algorithm—code included was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Classifying video presents unique challenges for machine learning models. As I’ve covered in my previous posts, video has the added (and interesting) property of temporal features in addition to the spatial features present in 2D images. While this additional information provides us more to work with, it also requires different network architectures and, often, adds larger memory and computational demands.

Today, we’ll take a look at different video action recognition strategies in Keras with the TensorFlow backend. We’ll attempt to learn how to apply five deep learning models to the challenging and well-studied UCF101 dataset.

Want the code? It’s all available on GitHub: Five Video Classification Methods. Pull requests encouraged!

This is part 3 in my series about video classification. If you missed the first two posts (gasp!), see here:

• Continuous online video classification with TensorFlow, Inception and a Raspberry Pi
• Continuous video classification with TensorFlow, Inception and Recurrent Nets

The video classification methods

We’ll look at each of our five methods in turn to see which one achieves the best top 1 and top 5 accuracy on UCF101. We’ll explore:

1. Classifying one frame at a time with a ConvNet
2. Using a time-distributed ConvNet and passing the features to an RNN, in one network
3. Using a 3D convolutional network
4. Extracting features from each frame with a ConvNet and passing the sequence to a separate RNN
5. Extracting features from each frame with a ConvNet and passing the sequence to a separate MLP

Constraints

Each of these methods could be its own blog post (or ten), so we’ll impose a few constraints to help simplify things and also to keep down computational complexity for future applications in real-time systems:

1. We won’t use any optical flow images. This reduces model complexity, training time, and a whole whackload of hyperparemeters we don’t have to worry about.
2. Every video will be subsampled down to 40 frames. So a 41-frame video and a 500 frame video will both be reduced to 40 frames, with the 500-frame video essentially being fast-forwarded.
3. We won’t do much preprocessing. A common preprocessing step for video classification is subtracting the mean, but we’ll keep the frames pretty raw from start to finish.
4. Every model has to fit into the 12 GiB of memory provided to us in the GPU in the new AWS p2.xlarge instances.

With these constraints, we know we won’t hit the ~94% state-of-the-art accuracy, but we’ll see if we can go in that direction.

Dataset

We’re going to use the popular UCF101 dataset. I find this dataset to have a great balance of classes and training data, as well as a lot of well-documented benchmarks for us to judge ourselves against. And unlike some of the newer video datasets (see YouTube-8M), the amount of data is manageable on modern systems.

UCF summarizes their dataset well:

With 13,320 videos from 101 action categories, UCF101 gives the largest diversity in terms of actions and with the presence of large variations in camera motion, object appearance and pose, object scale, viewpoint, cluttered background, illumination conditions, etc, it is the most challenging data set to date.

Challenge accepted!

Data preparation

The first thing we need to do is get the data in a format we can train on. We accomplish this in three steps:

1. Split all the videos into train/test folders
2. Extract jpegs of each frame for each video
3. Summarize the videos, their class, train/test status and frame count in a CSV we’ll reference throughout our training.

One important note is that the training set has many videos from the same “group”. This means there may be multiple videos of the same person from the same angle in the same setting performing the same action. It’s crucial that we don’t end up with videos from the same group in both the train and test groups, as we’d score unrealistically high on these classes.

UCF provides three train/test split recommendations that we can follow. For the sake of time, we use just split #1 for all of our experiments.

A note about the graphs below

Each graph includes three series:

1. The CNN-only top 1 accuracy in red, used as a baseline.
2. The top 5 categorical accuracy in green.
3. The top 1 categorical accuracy in blue.

I apologize for the lack of legend and the fugliness of the Matplotlib charts!

Method #0: Randomly guess or always choose the most common

This isn’t a real classification method, but if our model can’t beat random or the most used, we’re definitely not on the right track!

Trying to randomly guess the best result gives us ~0.9% accuracy. This makes sense since there are 101 classes, and, well… math.

Always guessing the most common class, “TennisSwing”, yields 1.32%. Also makes sense since TennisSwing labels are ~1.32% of our dataset. Okay, we’re on the right track, and we have something to beat. Let’s build some models.

Method #1: Classify one frame at a time with a CNN

For our first method, we’ll ignore the temporal features of video and attempt to classify each clip by looking at a single frame. We’ll do this by using a CNN, AKA ConvNet. More specifically, we’ll use Inception V3, pre-trained on ImageNet.

We’ll use transfer learning to retrain Inception on our data. This takes two steps.

First, we fine-tune the top dense layers for 10 epochs (at 10,240 images per epoch) in an attempt to retain as much of the previous learning as possible. (Updated Sep 26, ’07: A reader pointed out a bug in the train_cnn.py file that was causing the fine tuning to not update weights. I previously wrote that this step oddly did not yield meaningful results. The below chart is not updated to reflect this.) Fine tuning the top dense layers get us to ~52% top-1 validation accuracy, so it’s a great shortcut!

Next, we retrain the top two inception blocks. Coming up on 70 epochs, we’re looking really good, achieving a top 1 test accuracy of about 65%!

It’s worth noting that we’re literally looking at each frame independently, and classifying the entire video based solely on that one frame. We aren’t looking at all the frames and doing any sort of averaging or max-ing.

Let’s spot check a random sample of images from the test set to see how we did:

Final test accuracy: ~65% top 1, ~90% top 5

Method #2: Use a time-distributed CNN, passing the features to an RNN, in one network

Now that we have a great baseline with Inception to try to beat, we’ll move on to models that take the temporal features of video into consideration. For our first such net, we’ll use Kera’s awesome TimeDistributed wrapper, which allows us to distribute layers of a CNN across an extra dimension — time. Obviously.

For the ConvNet part of the model, we’ll use a very small VGG16-style network. Ideally we’d use a deeper network for this part, but given that we have to load the whole thing into GPU memory, our options are fairly limited.

For the RNN part of the net, we’ll use a three-layer GRU, each consisting of 128 nodes, and a 0.2 dropout between each layer.

Unlike with method #1, where we got to use the pre-trained ImageNet weights, we’ll have to train the whole model on our data from scratch here. This could mean that we’ll require too much data or a much bigger network to achieve the same level of accuracy as the Inception whopper produced. However, it also means that our CNN weights will be updated on each backprop pass along with the RNN. Let’s see how it does!

Yikes. How disappointing. Looks like we may need a more complex CNN to do the heavy lifting.

Final test accuracy: 20% top 1, 41% top 5

Method #3: Use a 3D convolutional network

Okay so training a CNN and an LSTM together from scratch didn’t work out too well for us. How about 3D convolutional networks?

3D ConvNets are an obvious choice for video classification since they inherently apply convolutions (and max poolings) in the 3D space, where the third dimension in our case is time. (Technically speaking it’s 4D, since our 2D images are represented as 3D vectors, but the net result is the same.)

However, they have the same drawback we ran into with method #2: memory! In Learning Spatiotemporal Features with 3D Convolutional Networks, the authors propose a network they call C3D that achieves 52.8% accuracy on UCF101. I was excited to attempt to reproduce these results, but I was stalled out with memory limitations of the 12 GiB GPU in the P2. The C3D simply wouldn’t run, even as I hacked off layer after layer.

As a plan B, I designed a smaller derivative, consisting of just three 3D convolutions, growing in size from 32 to 64 to 128 nodes. How’d we do?

After 28 epochs, we aren’t even close to hitting the benchmark we set with Inception. I did reduce the learning rate from 5e-5 to 1e-6 and trained for another 30 epochs (not graphed), which got us a little better, but still not in the ballpark.

Maybe training from top-to-bottom isn’t the way to go. Let’s go another direction.

Final test accuracy: 28% top 1, 51% top 5

Method #4: Extract features with a CNN, pass the sequence to a separate RNN

Given how well Inception did at classifying our images, why don’t we try to leverage that learning? In this method, we’ll use the Inception network to extract features from our videos, and then pass those to a separate RNN.

This takes a few steps.

First, we run every frame from every video through Inception, saving the output from the final pool layer of the network. So we effectively chop off the top classification part of the network so that we end up with a 2,048-d vector of features that we can pass to our RNN. For more info on this strategy, see my previous blog post on continuous video classification.

Second, we convert those extracted features into sequences of extracted features. If you recall from our constraints, we want to turn each video into a 40-frame sequence. So we stitch the sampled 40 frames together, save that to disk, and now we’re ready to train different RNN models without needing to continuously pass our images through the CNN every time we read the same sample or train a new network architecture.

For the RNN, we use a single, 4096-wide LSTM layer, followed by a 1024 Dense layer, with some dropout in between. This relatively shallow network outperformed all variants where I tried multiple stacked LSTMs. Let’s take a look at the results:

Hey that’s pretty good! Our first temporally-aware network that achieves better than CNN-only results. And it does so by a significant margin.

Final test accuracy: 74% top 1, 91% top 5

Method #5: Extract features from each frame with a CNN and pass the sequence to an MLP

Let’s apply the same CNN extraction process as in the previous method, but instead of sending each piece of the sequence to an RNN, we’ll flatten the sequence and pass the new (2,048 x 40) input vector into a fully connected network, AKA a multilayer perceptron (MLP). The hypothesis is that the MLP will be able to infer the temporal features from the sequence organically, without it having to know it’s a sequence at all.

(Plus, “multilayer perceptron” is one of the coolest terms in data science.)

After trying quite a few deep, shallow, wide and narrow networks, we find that the most performant MLP is a simple two-layer net with 512 neurons per layer:

Another method that beats the CNN-only benchmark! But not nearly as impressively as the RNN did. My gut tells me there’s room for parameter tuning on this to do better.

For most of our methods, I’m only showing results for our top performing network. However, with the MLP, something interesting happened when we tried deeper and wider networks: The top 1 accuracy floundered, but the top 5 accuracy went off the charts! Here’s a four-layer, 2,048-wide MLP:

That is basically perfect top 5 classification accuracy. But I digress…

Final test accuracy: 70% top 1, 88% top 5

And the winner is…

…the Inception ConvNet to extract features followed by a single-layer LSTM RNN!

Now it’s time to make a concession: 74% accuracy on UCF101 is nowhere near state-of-the-art, which stands at over 94% last I checked. However, with a single stream of images (no optical flow), minimal preprocessing, limited memory and very little parameter tuning, I think we’ve outlined some great jumping off points for diving deeper into each of these five classification methods.

My primary takeaway from this research is how powerful convolutional networks are at a wide range of tasks. Classifying “out in the wild” moving images with 101 classes while only looking at individual images at 65% accuracy is just astounding. And using it to extract features for other models turns out to be key to achieving satisfactory results with low memory availability.

My second takeaway is that there is a lot of work to be done here. This excites me greatly, and I hope this post helps kick start ideas and motivates others to explore the important world of video classification as well!

Want the code? It’s all available on GitHub: Five Video Classification Methods. Pull requests encouraged!

Five video classification methods implemented in Keras and TensorFlow was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deep learning in 2017 is magical. We get to apply immensely complex algorithms to equally complex problems without having to spend all our time writing the algorithms ourselves. Instead, thanks to libraries like TensorFlow and Keras, we get to focus on the fun stuff: model architecture, parameter tuning and data augmentation.

Today, we’ll explore one such application of deep learning. We’ll use the Udacity self-driving car nanodegree program simulator to train a generalized steering model in under 100 lines of code. (As a point of reference, WordPress’ get_permalink() method, which retrieves the URL of a WordPress post, is 124 lines!)

Note: Quite a few people have written about their solutions to this Udacity simulator steering problem on Medium. I highly recommend Vivek Yadav’s post, An augmentation based deep neural network approach to learn human driving behavior, which presents a wealth of information on the problem and the power of data augmentation.

Without further ado, here’s our entire solution: 99 lines including comments and white space:

Note: Shoutout to Whoopska on Reddit for the code review!

Setup

The Udacity simulator has two modes, training and autonomous, and two tracks. The challenge is to train only on track 1 and come up with a model that can drive itself around track 2, which has significantly different features.

To do this, we first drive the car around in training mode on track 1 for five laps to generate our data, doing our best to keep the car in the center. From there, the above code takes over. Let’s go through each part:

The Model

The model is a ConvNet that draws inspiration from a slew of sources including NVIDIA, VGG16 and Mr. Yadav’s post, linked above. We stack five convolution layers, which grow in width each layer, with max pooling between them for spatial reduction (and memory friendliness). Then we top it off with a couple dense layers, and use a linear activation to output our continuous steering angle. We use “elu” activations across the board the adam optimizer, both of which were found to help the model converge quickly.

Processing the log

The Udacity simulator spits out the data it records into a CSV. It stores the path to three images for each frame: left image, center image and right image. This is where the NVIDIA influence really comes in strong:

We actually throw out the center images entirely because the left and right images hold so much information. For each sample, we add 0.4 steering angle to the left image, and -0.4 angle for the right image. The intuition here is that if our center image (as we’d see when actually driving with the model) looks like the right image when we’re going straight, then we probably need to veer to the left a little, and vice versa.

The 0.4 value is pretty aggressive and was found through trial and error. Others who have solved this problem report using 0.2 or 0.25. Although those values worked great for track 1, generalizing for track 2 required us to go big or go home, and 0.4 worked best.

Processing the images

We have to pass our images to our neural net as numpy arrays for training, so that is taken care of in the image processing function (and we normalize the values between -0.5 and 0.5).

We also do some data augmentation here to help generalize our model, which is critical for getting it to work well on track 2. For each image, we randomly shift it vertically between -20 and +20%. This vertical shift helps the model perform significantly better on both tracks. We also randomly apply a darkened area to each image. One of the challenges with only training on track 1 is that track 2 has a lot of harsh shadows, which really threw off the model’s ability to stay on course. Applying a random darkened box to each image solved this issue effectively.

Last, we double the number of training samples we have by randomly performing a horizontal flip on the images, with a corresponding inversion of the steering angle. This has the added effect of neutralizing steering bias from imbalanced training data.

Training & Generator

Since we’re dealing with image data, we can’t load it all up into memory at the same time, so we use Kera’s awesome fit_generator function. This function accepts a Python generator as an argument, which yields the data.

Our generator randomly chooses batch_size samples from our X/y pairs, passes the image through our processor, and returns the batch for training. We used a batch size of 64, which performed significantly better than smaller batch sizes (32) and much larger ones (256/512). More exploration should be done here to better understand why.

Steering Performance

The model was able to drive itself around track 1 flawlessly after just a single epoch of training on 20,224 samples, which takes all of 64 seconds on my GeForce 960m. It’s really quite amazing how quickly the model is able to perform on this track.

Generalizing for track 2 took longer, going 8 epochs before circumnavigating the whole track. It took a while for the model to learn how to deal with shadows and other very dark areas, relative to the training data.

So there you have it, a generalized steering model in under 100 lines of code. Now, a few caveats and notes:

• I’m not generally for attempting to solve problems under some arbitrary code limit. The purpose of this post is to show how current libraries allow us to focus on the application of machine learning, instead of getting caught in the weeds writing sophisticated algorithms.
• Although it didn’t take much code, this project did take quite a lot of experimenting and research to find good network parameters and data augmentation techniques. But that’s the beauty of it: I was able to spend all my time there, rather than on the nitty gritty.
• This is only possible because of libraries like TensorFlow, TFLearn and Keras. The biggest hat tip to those who spend their time developing and open sourcing these projects.
• I’m not in the Udacity self-driving car program, but my Medium feed is flooded with posts by those who are, so I got inspired. :)

Have suggestions for how I could improve my solution? I’d love to hear it. Thanks for reading!

Training a deep learning model to steer a car in 99 lines of code was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

A video is a sequence of images. In our previous post, we explored a method for continuous online video classification that treated each frame as discrete, as if its context relative to previous frames was unimportant. Today, we’re going to stop treating our video as individual photos and start treating it like the video that it is by looking at our images in a sequence. We’ll process these sequences by harnessing the magic of recurrent neural networks (RNNs).

To restate the problem we outlined in our previous post: We’re attempting to continually classify video as it’s streamed, in an online system. Specifically, we’re classifying whether what’s streaming on a TV is a football game or an advertisement.

Background

Convolutional neural networks, which we used exclusively in our previous post, do an amazing job at taking in a fixed-size vector, like an image of an animal, and generating a fixed-size label, like the class of animal in the image. What CNNs cannot do (without computationally intensive 3D convolution layers) is accept a sequence of vectors. That’s where RNNs come in.

RNNs allow us to understand the context of a video frame, relative to the frames that came before it. They do this by passing the output of one training step to the input of the next training step, along with the new frames. Andrej Karpathy describes this eloquently in his popular blog post, “The Unreasonable Effectiveness of Recurrent Neural Networks”:

At the core, RNNs have a deceptively simple API: They accept an input vector x and give you an output vector y. However, crucially this output vector’s contents are influenced not only by the input you just fed in, but also on the entire history of inputs you’ve fed in in the past.

We’re using a special type of RNN here, called an LSTM, that allows our network to learn long-term dependencies. Christopher Olah writes in his outstanding essay about LSTMs: “Almost all exciting results based on recurrent neural networks are achieved with [LSTMs].”

Sold! Let’s get to it.

Methods

Our aim is to use the power of CNNs to detect spatial features and RNNs for the temporal features, effectively building a CNN->RNN network, or CRNN. For the sake of time, rather than building and training a new network from scratch, we’ll…

1. Use transfer learning to retrain the Inception v3 network on our training data, like we did in our previous post
2. Pass our train and test data through the CNN, saving the output of a couple different layers to disk (see below)
3. Convert the output of individual frames into sequences of frames
4. Train the new RNN on the train set
5. Evaluate the RNN on the test set of data and see if it performs better than the 93.3% benchmark we set in the previous post.

Step 2 is unique so we’ll expand on it a bit. There are two interesting paths that come to mind when adding a recurrent net to the end of our convolutional net:

1. We can pass the actual label predictions generated by the softmax layer of the CNN to the RNN. This gives us the probability that the frame is each of our classes, football or ad, which is the prediction we used in our previous post.
2. We can pass the output of the pool layer, before it’s made into a prediction, to the RNN. The pool layer gives us a 2,048-d vector that represents the convoluted features of the image, but not a class prediction.

Softmax and pool layers?

Let’s say you’re baking a cake. You have at your disposal all of the ingredients in the world. We’ll say that this assortment of ingredients is our image to be classified. By looking at a recipe, you see that all of the possible things you could use to make a cake (flour, whisky, another cake) have been reduced down to ingredients and measurements that will make a good cake. The person who created the recipe out of all possible ingredients is the convolutional network, and the resulting instructions are the output of our pool layer. Now you make the cake and it’s ready to eat. You’re the softmax layer, and the finished product is our class prediction.

Code

I’ve made the code to explore these methods available on GitHub. I’ll pull out a couple interesting bits here:

Frames to sequences

In order to turn our discrete predictions or features into a sequence, we loop through each frame in chronological order, add it to a queue of size N, and pop off the first frame we previously added. Here’s the gist:

N represents the length of our sequence that we’ll pass to the RNN. We could choose any length for N, but I settled on 40. At 10fps, which is the framerate of our video, that gives us 4 seconds of video to process at a time. This seems like a good balance of memory usage and information.

The RNN itself

The architecture of the network is a single LSTM layer with 256 nodes. This is followed by a dropout of 0.2 to help prevent over-fitting and a fully-connected softmax layer to generate our predictions. I also experimented with wider and deeper networks, but neither performed as well as this one. It’s likely that with a larger training set, a deeper network would perform best.

Note: I’m using the incredible TFLearn library, a higher-level API for TensorFlow, to construct our network, which saves us from having to write a lot of code.

Training and evaluating

Once we have our sequence of features and our network, training with TFLearn is a breeze.

Evaluating is even easier.

Now, let’s evaluate each of the methods we outlined above for adding an RNN to our CNN.

Softmax layer method

Intuitively, if one frame is an ad and the next is a football game, it’s essentially impossible that the next will be an ad again. (I wish commercials were only 1/10th of a second long!)

This is why it could be interesting to examine the temporal dependencies of the probabilities of each label before we look at the more raw output of the pool layer. We convert our individual predictions into sequences using the code above, and then feed the sequences to our RNN.

After training the RNN on our first batch of data, we then evaluate the predictions on both the batch we used for training and a holdout set that the RNN has never seen. No surprise, evaluating the same data we used to train gives us an accuracy of 99.55%! Good sanity check that we’re on the right path.

Now the fun part. We run the holdout set through the same network and get… 95.4%! Better than our 93.3% we got without the LSTM, and not a bad result, given we’re using the full output of the CNN, and thus not giving the RNN much responsibility. Let’s change that.

Pool layer method

Here we’ll go a little deeper. (See what I did there?) Instead of letting the CNN do all the hard work, we’ll give more responsibility to the RNN by using output of the CNN’s pool layer, which gives us the feature representation (not a prediction) of our images. We again build sequences with this data to feed into our RNN.

Running our training data through the network to make sure we get high accuracy succeeds at 99.89%! Sanity checked.

How about our holdout set?

96.58%! That’s an error reduction of 3.28 percentage points (or 49%!) from our CNN-only benchmark. Awesome!

Next steps

We have shown that taking both spatial and temporal features into consideration improves our accuracy significantly.

Next, we’ll want to try this method on a more complex dataset, perhaps using multiple classes of TV programming, and with a whole whackload more data to train on. (Remember, we’re only using 20 minutes of TV here.)

Once we feel comfortable there, we’ll go ahead and combine the RNN and CNN into one network so we can more easily deploy it in an online system. That’s going to be fun.

Part 3 is now available: Five video classification methods implemented in Keras and TensorFlow

Continuous video classification with TensorFlow, Inception and Recurrent Nets was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Much has been written about using deep learning to classify prerecorded video clips. These papers and projects impressive tag, classify and even caption each clip, with each comprising a single action or subject.

Today, we’re going to explore a way to continuously classify video as it’s captured, in an online system. Continuous classification allows us to solve all sorts of interesting problems in real-time, like understanding what’s in front of a car for autonomous driving applications to understanding what’s streaming on a TV. We’ll attempt to do the latter using only open source software and uber-cheap hardware. Specifically, TensorFlow on a Raspberry Pi with a PiCamera.

We’ll use a “naive” classification approach in this post (see next section), which will give us a relatively straightforward path to solving our problem and will form the basis for more advanced systems to explore later.

By the time we’re done today, we should be able to classify what we see on our TV as either a football game or an advertisement, running on our Raspberry Pi.

Let’s get to it!

The intuition

Video is an interesting classification problem because it includes both temporal and spatial features. That is, at each frame within a video, the frame itself holds important information (spatial), as does the context of that frame relative to the frames before it in time (temporal).

We hypothesize that for many applications, using only spatial features is sufficient for achieving high accuracy. This approach has the benefit of being relatively simple, or at least minimal. It’s naive because it ignores the information encoded between multiple frames of the video.

Since football games have rather distinct spatial features, we believe this method should work wonderfully for the task at hand.

Method

We’re going to collect data for offline training with a Raspberry Pi and a PiCamera. We’ll point the camera at a TV and record 10 frames per second, or more specifically, save 10 jpegs every second, which will comprise our “video”.

Here’s the code for capturing our images:

Once we have our data, we’ll use a convolutional neural network (CNN) to classify each frame with one of our labels: ad or football.

Offline training and exploration

TensorFlow and Inception

CNNs are the state-of-the-art for image classification. And in 2016, it’s essentially a solved problem. It feels crazy to say that, but it really is: Thanks in large part to Google→TensorFlow→Inception and the many researchers who came before it, there’s very little low-level coding required for us when it comes to training a CNN for our continuous video classification problem.

Pete Warden at Google wrote an awesome blog post called TensorFlow for Poets that shows how to retrain the last layer of Inception with new images and classes. This is called transfer learning, and it lets us take advantage of weeks of previous training without having to train a complex CNN from scratch. Put another way, it lets us train an image classifier with a relatively small training set.

Our training data

We collected 20 minutes of footage at 10 jpegs per second, which amounted to 4,146 ad frames and 7,899 football frames. The next step is to sort each frame into two folders: football and ad. The name of the folders represent the labels of each frame, which will be the classes our network will learn to predict on when we retrain the top layer of the Inception v3 CNN.

This is essentially using the flowers method described in TensorFlow for Poets, applied to video frames.

To retrain the final layer of the CNN on our new data, we checkout the r0.11 tag from the TensorFlow repo and run the following command:

python3 tensorflow/examples/image_retraining/retrain.py \     
    --bottleneck_dir=/home/harvitronix/blog/inception/bottlenecks \
    --model_dir=/home/harvitronix/blog/inception/ \
    --output_graph=/home/harvitronix/blog/retrained_graph.pb \
    --output_labels=/home/harvitronix/blog/retrained_labels.txt \
    --image_dir /home/harvitronix/blog/images/classifications/

Retraining the final layer of the network on this data takes about 30 minutes on my laptop with a GeForce GTX 960m GPU. At the completion of 4,000 training steps, our model reports an incredible 98.8% accuracy on the held out validation set! I’m not sure I could do much better using my eyes on the same data. As a point of reference, if the network had classified each frame as football, it would have achieved about 66% accuracy. So it seems to be working!

Spot checking

It’s always a good idea to run some known data through a trained network to sanity check the results, so we’ll do that here.

Here’s the code we use to classify a single image manually through our retrained model:

And here are the results of spot checking individual frames:

Simulated online classification

Before we transfer everything to our Pi and do this in real-time, let’s use a different batch of recorded data and see how well we do on that set.

To get this dataset, and to make sure we don’t have any data leakage into our training set, we separately record another 19 minutes of the football broadcast. This dataset amounted to 2,639 ad frames and 8,524 football frames.

We run each frame of this set through our classifier and achieve a true holdout accuracy score of 93.3%. Awesome!

Looks like we’ve validated our hypothesis that we can achieve high levels of accuracy while only considering spatial features. Impressive results, considering that we only used 20 minutes of training data! Thank you, Google, Pete, TensorFlow and all the folks who have developed CNNs over the years for your incredible work and contributions.

Online classification

Great, so now we have our CNN trained and we know that we can classify each frame of our video with relatively high accuracy. How does it do on live TV, with always changing context?

For this, we load up our Raspberry Pi 3 with our newly trained model weights, turn on the PiCamera at 10 fps, and instead of saving the image, send it through our CNN to be classified.

We have to make some modifications to the code to classify in real time. The final result looks like this:

We also have to get TensorFlow running on the Pi. Sam Abrahams wrote up excellent instructions for doing this, so I won’t cover them again here.

After we install our dependencies, we run the program and… crap! Inception on the Raspberry Pi 3 can only classify one image every four seconds.

Okay, so we don’t quite have the hardware yet to do 10fps, but this still feels like magic, so let’s see how we do.

Flipping on Sunday Night Football and pointing our camera at the TV shows a remarkable job at classifying each moment as football or ad, once every few seconds. For the vast majority of the broadcast, we see our prediction come out true to life. So cool.

Next steps

In all, our naive method worked remarkably well at continuous online video classification for this particular use case. But we know that we’re only considering part of the information provided to us inherently in video, and so there must be room for improvement, especially as our datasets become more complex.

For that, we’ll have to dive deeper. So in the next post, we’ll explore feeding the output of our CNN (both the final softmax layer and the pool layer, which gives us a 2,048-d feature vector of each image) to an LSTM RNN to see if we can increase our accuracy.

Spoiler alert: We can!

Update: Part 2 is now live!

The code that accompanies this post is available on GitHub.

Further reading

• This is an amazing reference that will get you caught up with the state of CNNs for video: “Deep Learning for Video Classification and Captioning”
• This is a creative network that uses a hybrid approach: “Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification”
• Another great paper, this one focused on CNN+LSTMs, similar to what we’ll explore in the next post: “Beyond Short Snippets: Deep Networks for Video Classification”

Continuous online video classification with TensorFlow, Inception and a Raspberry Pi was originally published in Coastline Automation on Medium, where people are continuing the conversation by highlighting and responding to this story.

The way I’ve done this for years is to send myself an e-mail. But because I’m a terrible sys admin I find it a pain to get e-mail to reliably send from Python+Ubuntu+some SMTP server. I will never admit how long I’ve spent fighting this battle…

Enter: Slack teams, Python and push notifications.

Here’s how to do it:

Step 1: Set yourself up with a personal Slack team by registering a team of one at slack.com.

Step 2: Get your Slack token here: https://api.slack.com/docs/oauth-test-tokens

Step 3: Install the official Slack Python package:

pip install slackclient

Step 4: Write a little function to send yourself a message:

from slackclient import SlackClient

def slack_message(message, channel):
    token = '[YOUR TOKEN]'
    sc = SlackClient(token)

    sc.api_call('chat.postMessage', channel=channel, 
                text=message, username='My Sweet Bot',
                icon_emoji=':robot_face:')

That’s it that’s all! Free, quick and easy native push notifications for your next personal project.