The new Create ML app just announced at WWDC 2019, is an incredibly easy way to train your own personalized machine learning models. All that’s required is dragging a folder containing your training data into the tool and Create ML does the rest of the heavy lifting.

So how do we prepare our data?

When doing image or sound classification we just need to organize the data into folders, but if we want to do object detection the task becomes a bit more complicated. With object detection, we need to specify some additional information.

In addition to our images, we need an annotations.json with the coordinates of where the objects are. The annotations need to match the following format:

[
    {
        "image": "image1.jpg",
        "annotations": [
            {
                "label": "carrots",
                "coordinates": {
                    "x": 120
                    "y": 164
                    "width": 230
                    "height": 119
                }
            },
            {
                "label": "orange",
                "coordinates": {
                    "x": 230
                    "y": 321
                    "width": 50
                    "height": 50
                }
            }
        ]
    },
    ...
]

Note: The x and y coordinates are the centers of the bounding rectangles and all coordinates are in pixels

How do we generate this json? (I definitely don’t want to do it all by hand)

Apple tells us

“You can download tools from the web to help you build these [annotations]”

So what tool should we use? Cloud Annotations!

Cloud Annotations is a tool that I built exactly for this purpose. It lets us quickly draw boxes on our images and gives us an annotations.json in the format required by Apple.

Creating an object storage instance

To use Cloud Annotations we need to create a cloud object storage instance. Creating a Cloud Object Storage instance gives us a reliable place to keep our training data. It also opens up the potential for data collection and collaboration, letting us collect user data and allowing a team of specialists to easily label it.

IBM Cloud offers a lite tier of object storage, which includes 25 GB of storage for free. (this is what we will be using throughout the tutorial)

To create an instance we first need to log in or sign up for IBM Cloud.

Once logged in you should find your IBM Cloud dashboard. This is where we can create and manage IBM Cloud resources.

We want to create a new Cloud Object Storage instance so click the Create resource button.

Locate and choose the Object Storage option.

Choose a pricing plan and click Create, then Confirm on the following popup.

Credentials

Once we have an object storage instance we need a way to access our data from outside of IBM Cloud. To do be able to do this we need to create a set credentials for our resource.

We can do this by navigating to the Service credentials tab and clicking the New credential button.

You can leave most of the options blank, but make sure the credential has the role of Writer and add the following inline configuration parameters:

{"HMAC": true}

Once added, click the View credentials ▾ dropdown and take note of your apikey, access_key_id, secret_access_key and resource_instance_id.

{
  "apikey": "...",
  "cos_hmac_keys": {
    "access_key_id": "...",
    "secret_access_key": "..."
  },
  "endpoints": "...",
  "iam_apikey_description": "...",
  "iam_apikey_name": "...",
  "iam_role_crn": "...",
  "iam_serviceid_crn": "...",
  "resource_instance_id": "..."
}

Cloud Annotations

To use the tool just navigate to the Cloud Annotations Tool and add your object storage credentials.

We will be storing our files and annotations in something called a bucket, we can create one by clicking Create bucket.

After we create and name our bucket, it will prompt us to choose an annotation type. We need to choose Localization. This allows us to draw bounding box rectangles on our images.

Training data best practices

• The model we will be training is optimized for photographs of objects in the real world. They are unlikely to work well for x-rays, hand drawings, scanned documents, receipts, etc.
• The training data should be as close as possible to the data on which predictions are to be made. For example, if your use case involves blurry and low-resolution images (such as from a security camera), your training data should be composed of blurry, low-resolution images. In general, you should also consider providing multiple angles, resolutions, and backgrounds for your training images.
• The model we will be training can’t generally predict labels that humans can’t assign. So, if a human can’t be trained to assign labels by looking at the image for 1–2 seconds, the model likely can’t be trained to do it either.
• We recommend at least 50 training images per label for a usable model, but using 100s or 1000s would provide better results.
• The model we will be training resizes the image to 300x300 pixels, so keep that is mind when training the model with images where one dimension is much longer than the other.

Labeling the data

To label images:

1. Upload a video or many images
2. Create the desired labels
3. Start drawing bounding boxes

Getting the annotations

After we have collected and labeled our first round of images, we are ready to start training our model!

Installation

To access our annotations we need to install the Cloud Annotation CLI:

npm install -g cloud-annotations

Note: You’ll need to have Node 10.13.0 or later installed. You can use nvm (macOS/Linux) or nvm-windows to easily switch Node versions between different projects.

Download the annotations

To download the annotations all we need to do is run the following command

cacli export --create-ml

Once finished there should be a folder named exported_buckets with your bucket inside. All you need to do is drag this folder into the Create ML app and you’re good to go!

Thanks for reading! If you have any questions, feel free to reach out atbourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

How to Label Data — Create ML for Object Detection was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last article I showed you how to do image classification in the browser.

Image classification can be a very useful tool, it can give us an idea of what’s in an image. However, sometimes we want more. It can be a little counterintuitive, but just because a machine learning model can tell what’s in an image, doesn’t mean it can tell us where it is in the image. We need a different architecture for that.

That’s where object detection comes into play.

Object detection opens up the capability of counting how many objects are in a scene, tracking motion and simply just locating an object’s position.

By the end of this tutorial we’ll have a fully functional real-time object detection web app that will track objects via our webcam.

Detecting Objects

To make object detection predictions, all we need to do is import the TensorFlow model, coco-ssd, which can be installed with a package manager like NPM or simply imported in a <script> tag. We can then load the model, and make a prediction.

Note: Loading the model can take several seconds. It is best to load the model once and save a reference to it.

The image we pass into the detection function is just a reference to the html <img> tag:

<img id="image" src="image_url">

After we get our prediction, we need a way to display it to the screen. Our prediction response will be a list of bounding boxes and their confidence scores. It should look something like this:

[{
  bbox: [x, y, width, height],
  class: "cat",
  score: 0.8380282521247864
}]

To draw the bounding boxes we are going to take advantage of a <canvas> element:

<canvas id="canvas">

The canvas element allows us to use the strokeRect function, which maps perfectly with our prediction results:

Simple Detection Demo

Streaming from the Webcam

To run real-time detection on a webcam stream is almost as easy as changing from an <img> tag, to a <video> tag …with the simple exception of this giant blob of code to start up the webcam:

We can then just pass our video element to our model for detection. However, this time we are going to call requestAnimationFrame which will call our detection function over and over in an infinite loop as fast as it can, skipping frames when it can’t keep up.

Real-Time Detection Demo

React App

Note: You must use Google Chrome or a browser that supports webcam streaming (not Safari).

Link to the full code: https://codesandbox.io/s/z364noozrm.

Final Thoughts

I’ve been having a lot of fun using TensorFlow.js. They have made it extremely easy to play with machine learning with minimal headache. A similar project would have taken me days, but this only took hours. I think my next project will be to experiment with training some of my own models 🎉

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

TensorFlow.js — Real-Time Object Detection in 10 Lines of Code was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the largest obstacles for beginners getting experience with artificial intelligence and machine learning can honestly be the setup.

I’m not going to lie, there are still plenty of days that completely slip away, just trying to get Python, TensorFlow and my GPU to cooperate. Does this make me question my abilities as a competent software engineer? Yes, yes it does.

I digress.

So what is TensorFlow.js and how can it help us? From the official page, TensorFlow.js is, “A JavaScript library for training and deploying ML models in the browser and on Node.js.”

What does that mean for us? We can try it out right from this Medium article!

The Demo

In this demo we are using a deep learning model called “MobileNet”. MobileNet is a type of Convolution Neural Network, a model architecture that is good at image classification.

Note: To learn more about what artificial intelligence actually sees, check out my other article.

The Code

Using TensorFlow.js to classify an image with MobileNet is as easy as 3 lines of code:

mobilenet.load()
  .then(model => model.classify(myImage))
  .then(predictions => // Use predictions)

Final Thoughts

At the time of writing this post TensorFlow provides 5 official models, that are just as easy to use:

• mobilenet: Classify images with labels from the ImageNet database.
• posenet: Realtime pose detection. Blog post here.
• coco-ssd: Object detection based on the TensorFlow object detection API.
• speech-commands: Classify 1 second audio snippets from the speech commands dataset.
• knn-classifier: Create a custom k-nearest neighbors classifier. Can be used for transfer learning.

However, if none of these models suffice, you can also create/train your own, but that’s for another article.

For fun, I challenge you to look at the PoseNet documentation and try to match the results of the cover photo. But, if you get stuck here’s the code and demo:

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Start Using TensorFlow.js — Without Leaving This Article was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

Many call artificial intelligence (AI) a “black box”, and it kinda is. One of the biggest problems of AI is that it’s incredibly difficult to understand how the data is being interpreted.

Before we get our hands dirty and dive deeper, let’s play a little game.

I’m going to show you a series of abstract images that are either in category A or B.

Do you think the following image belongs to category A or B?

Hint: There’s no C.

We’ll get back to this later.

Let’s look at some more examples first.

Now can you tell if it belongs to A or B?

⚠️ Spoiler Alert

The answer is… A!

If you choose B, don’t be embarassed, you’re not alone. When asked to a room full of engineers and developers, the split is always 50/50. So… why is the answer A…?

Because I said so.

The answer is A, there’s no debating it, but if you don’t agree with me, then it was my fault as the trainer.

As the trainer, I know that A is a red circle. So anything with a red circle in it is A. I also know that B is an orange circle. The rest of the image is irrelevant. It’s all about trying to find a pattern between the set of images.

But it’s hard.

In an AI system, I can’t explain with words what makes the image A. All I can do is show you more pictures and hope it starts to click.

And you, the AI, can’t tell me why you think it’s B. It’s up to me to blindly feed you data until you get the answer right.

Here’s the same set of images, but less abstract. If I were to ask you the same question, everyone would know right away that A is an apple and B is an orange. This is almost so easy that many people think it’s a trick question. We all know that the hand and background are all irrelevant information, because we’re humans and grew up learning these things, but for AI it’s not a given. It sees images as more abstract and doesn’t know what you want it to focus on.

A Miscommunication

Let’s take a look at another toy scenario that shows how we might accidentally communicate the wrong signals to the AI system.

We have a few samples of oak trees. (It’s a bit cloudy where I live)

Here are some palm trees. (It was really sunny on the beach)

This next example is a palm tree, but the lighting is much closer to the oak trees. Which pattern should we focus on? The lighting? Or the shape of the tree? It might be difficult for the model to tell.

Confidence:
 - Palm 0.75
 - Oak 0.60

With this example, it might be pretty obvious that we left behind an unintended pattern for the AI to pick up. However, in reality, it’s normally something much more inconspicuous.

Peeking Under the Curtain

So how can we get more insight into what the AI is focussing on?

What if we passed a rectangle over the image and recorded the changes in confidence? If the confidence drops, then that’s probably an important part of the image.

Which picture makes it easier to tell that this cable is a USB?

The first image completely obscures the connector, making it nearly impossible to guess, so we can denote the region the rectangle covers as important. However, the rectangle in the second image doesn’t hinder our ability to determine the cable type. We can safely mark the location as insignificant.

We can continue to pass the rectangle over image to establish a heat map of importance.

We can see that the model’s focus is on the tip of the connector, which is great. It’s looking where we want it to.

Let’s look at a model that wasn’t trained well.

Confidence:
 - USB 0.76

The model correctly predicted that the cable was a USB with a confidence of 0.76. We might say that’s acceptable, especially since the photo is far away and isn’t great quality.

However, upon closer inspection, the model seems to be focusing on the wrong area, not the ends of the cable like we would expect.

What does this tell us?

The model appears to rely too heavily on the wire and fingers. To improve accuracy and clear up the confusion, we can include more examples of wire and hands in a negative training set.

We don’t need to train on piles and piles of generic data until our model starts performing better. We can tactfully use this information as an aid in retraining the model, saving us time and money.

Using the Tool

Wow! This is great, but I don’t want to put in the effort to actually implement this

Good news! You can find the fully functional iOS app on my GitHub 😘

Final Thoughts

Creating your own model is easy, but that doesn’t mean the work stops there. The hardest part of machine learning is always producing good data.

We can use the basic guidelines of having similar pose, lighting and a consistent mix of stock and natural photos across our training images to gain a foothold in our quest toward a good model. After that, we are left using tools and our intuition to try and gain insight into the thought process of AI.

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Understanding What Artificial Intelligence Actually Sees was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

IBM Watson just announced the ability to run Visual Recognition models locally on iOS as Core ML models. I’m very excited.

Before now, it was fairly easy to integrate a visual recognition system into your iOS app by just downloading a model from Apple. However, the models you can use are very cookie cutter and specific to a standard set of items that it can recognize (cars, people, animals, fruit, etc).

But what if you wanted to have a model that could recognize items for a use case specific to you?

Before now, if you wanted to do this and you weren’t familiar with the ins and outs of AI, this could be a fairly difficult task. You would need to familiarize yourself with a machine learning framework such as TensorFlow, Caffe, or Keras. However, with Watson we can train our own custom model without having to touch any code.

Before, if you trained a model with Watson

• It was stuck in the cloud
• You needed an internet connection to access it
• You were charged to use it

Now

• You can download your model as a Core ML model
• You can use it locally like any other Core ML model
• No strings attached

Collecting Data

To build a custom model we need to collect data to train it on. If we want to train our model on different brands of shoes we need to collect a bunch of pictures of each brand.

I went ahead and downloaded from Google:

• 100 pictures of Nike shoes
• 100 pictures of Adidas shoes
• 100 pictures of New Balance shoes

This is a very lazy approach to collecting data. It won’t produce great results unless our use case happens to be classifying Google image search results. A much better approach (given this is for an iOS app) is to take pictures of the different shoes with your iPhone camera. However, I only happen to own 2 pairs of shoes and they both fall into the category of “Nike”.

We do what we can 🤷‍♂️

Note: As expected the model didn’t work as well as I wanted, so I ended up just using 10 pictures of my Nikes, 10 pics of my dad’s New Balances and 10 pics of my friend’s Adidas shoes (instead of the Google photos).

After you collect your image data, we need each category to be zipped up in its own .zip file.

Getting Started with Watson

To use Watson we need to create an IBM Watson Studio account. Sign in or sign up and then make your way to the main page:

Note: If you don’t see New project, Refine data, New notebook, etc... Click the Get started dropdown in the top right corner.

Click on New project and then select Visual Recognition:

Give your project a name and continue:

Browse for your .zip files that you created:

Then add them to the model by clicking the 3-dot menu and choosing Add to model:

Then click to train the model:

Downloading the Core ML Model

Once the model finishes training you will be greeted with a small notification. Click to view and test the model:

That will bring you to the page that will allow you to download the model:

Choose Implementation followed by Core ML. Then download the model file:

Important Note: Safari downloads the file with a.dms extension. Just rename it to have a .mlmodel extension.

Running the App

For the simplicity of this post I’m going to assume that you have a working knownledge of Xcode 😬

I built a sample app that you can clone from my GitHub.

All you need to do is open up the project. Import your Core ML model file that you downloaded. Followed by changing the name of the model to match yours in the CameraViewController file:

Update: After writing this, I found out about the Watson Swift SDK. It makes it extremely easy to make classifications as well as download any updates to your model after retraining.

Classify

Update Local Model

That’s all! You should be able to run the app, take pictures and get classifications back (please excuse my extremely dirty shoes):

Final Thoughts

I know I work for IBM, but hopefully I’m not the only one excited about this 😝 I’ve been waiting for the ability to run my custom Watson models locally for a while now. No more need for an internet connection and the models run fast no matter where you are.

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Simplify Adding AI to Your Apps — Core ML Say Hello to Watson was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

Training your model is hands down the most time consuming and expensive part of machine learning. Training your model on a GPU can give you speed gains close to 40x, taking 2 days and turning it into a few hours. However, this normally comes at a cost to your wallet.

The other day I stumbled upon a great tool called Google Colab. I would describe Colab as the google docs equivalent of Jupyter notebooks. Colab is aimed at being an education and research tool for collaborating on machine learning projects. The great part is, that it’s completely free forever.

There is no setup to use it. I didn’t even need to log in. (I was already logged into my google account)

The best part is that you get an unlimited supply of 12 hours of continuous access to a k80 GPU, which is pretty powerful stuff. (You get disconnected after 12 hours, but you can use it as many times as you want)

I want our focus to be training on a GPU and Colab specific so the notebook is extremely bare bones.

The first step is to download the notebook (or another notebook of your choice)

Then, head over to Google Colab, sign into your google account (or create one if you somehow made it this far through life without one)

Choose File > Upload notebook...:

Upload the notebook you downloaded:

Choose Runtime > Change runtime type:

Then choose GPU:

Now you should be able to run your notebooks how you normally would. The only difference is the very last part at the end. If you want to download your model or any other files via the browser, you can use their python library:

from google.colab import files
files.download("PATH/TO/FILE")

Final Thoughts

This was a pretty short post, but hopefully it ends the painful days of training your models on your poor little old laptop for days at a time or dropping a ton of 💰 on AWS bills.

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Train Your Machine Learning Models on Google’s GPUs for Free — Forever was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

Convolutional neural networks have done an amazing job, but are rooted in problems. It’s time we started thinking about new solutions or improvements — and now, enter capsules.

Previously, I briefly discussed how capsule networks combat some of these traditional problems. For the past for few months, I’ve been submerging myself in all things capsules. I think it’s time we all try to get a deeper understanding of how capsules actually work.

In order to make it easier to follow along, I have built a visualization tool that allows you to see what is happening at each layer. This is paired with a simple implementation of the network. All of it can be found on GitHub here.

This is the CapsNet architecture. Don’t worry if you don’t understand what any of it means yet. I’ll be going through it layer by layer, with as much detail as I can possibly conjure up.

Part 0: The Input

The input into CapsNet is the actual image supplied to the neural net. In this example the input image is 28 pixels high and 28 pixels wide. But images are actually 3 dimensions, and the 3rd dimension contains the color channels.

The image in our example only has one color channel, because it’s black and white. Most images you are familiar with have 3 or 4 channels, for Red-Green-Blue and possibly an additional channel for Alpha, or transparency.

Each one of these pixels is represented as a value from 0 to 255 and stored in a 28x28x1 matrix [28, 28, 1]. The brighter the pixel, the larger the value.

Part 1a: Convolutions

The first part of CapsNet is a traditional convolutional layer. What is a convolutional layer, how does it work, and what is its purpose?

The goal is to extract some extremely basic features from the input image, like edges or curves.

How can we do this?

Let’s think about an edge:

If we look at a few points on the image, we can start to pick up a pattern. Focus on the colors to the left and right of the point we are looking at:

You might notice that they have a larger difference if the point is an edge:

255 - 114 = 141
114 - 153 = -39
153 - 153 = 0
255 - 255 = 0

What if we went through each pixel in the image and replaced its value with the value of the difference of the pixels to the left and right of it? In theory, the image should become all black except for the edges.

We could do this by looping through every pixel in the image:

for pixel in image {
  result[pixel] = image[pixel - 1] - image[pixel + 1]
}

But this isn’t very efficient. We can instead use something called a “convolution.” Technically speaking, it’s a “cross-correlation,” but everyone likes to call them convolutions.

A convolution is essentially doing the same thing as our loop, but it takes advantage of matrix math.

A convolution is done by lining up a small “window” in the corner of the image that only lets us see the pixels in that area. We then slide the window across all the pixels in the image, multiplying each pixel by a set of weights and then adding up all the values that are in that window.

This window is a matrix of weights, called a “kernel.”

We only care about 2 pixels, but when we wrap the window around them it will encapsulate the pixel between them.

Window:
┌─────────────────────────────────────┐
│ left_pixel middle_pixel right_pixel │
└─────────────────────────────────────┘

Can you think of a set of weights that we can multiply these pixels by so that their sum adds up to the value we are looking for?

Window:
┌─────────────────────────────────────┐
│ left_pixel middle_pixel right_pixel │
└─────────────────────────────────────┘

(w1 * 255) + (w2 * 255) + (w3 * 114) = 141

Spoilers below!

 │            │            │
 │            │            │
 │            │            │
 │            │            │
 │            │            │
\│/          \│/          \│/
 V            V            V

We can do something like this:

Window:
┌─────────────────────────────────────┐
│ left_pixel middle_pixel right_pixel │
└─────────────────────────────────────┘

(1 * 255) + (0 * 255) + (-1 * 114) = 141

With these weights, our kernel will look like this:

kernel = [1  0 -1]

However, kernels are generally square — so we can pad it with more zeros to look like this:

kernel = [
  [0  0  0]
  [1  0 -1]
  [0  0  0]
]

Here’s a nice gif to see a convolution in action:

Note: The dimension of the output is reduced by the size of the kernel plus 1. For example:(7 — 3) + 1 = 5 (more on this in the next section)

Here’s what the original image looks like after doing a convolution with the kernel we crafted:

You might notice that a couple edges are missing. Specifically, the horizontal ones. In order to highlight those, we would need another kernel that looks at pixels above and below. Like this:

kernel = [
  [0  1  0]
  [0  0  0]
  [0 -1  0]
]

Also, both of these kernels won’t work well with edges of other angles or edges that are blurred. For that reason, we use many kernels (in our CapsNet implementation, we use 256 kernels). And the kernels are normally larger to allow for more wiggle room (our kernels will be 9x9).

This is what one of the kernels looked like after training the model. It’s not very obvious, but this is just a larger version of our edge detector that is more robust and only finds edges that go from bright to dark.

kernel = [
  [ 0.02 -0.01  0.01 -0.05 -0.08 -0.14 -0.16 -0.22 -0.02]
  [ 0.01  0.02  0.03  0.02  0.00 -0.06 -0.14 -0.28  0.03]
  [ 0.03  0.01  0.02  0.01  0.03  0.01 -0.11 -0.22 -0.08]
  [ 0.03 -0.01 -0.02  0.01  0.04  0.07 -0.11 -0.24 -0.05]
  [-0.01 -0.02 -0.02  0.01  0.06  0.12 -0.13 -0.31  0.04]
  [-0.05 -0.02  0.00  0.05  0.08  0.14 -0.17 -0.29  0.08]
  [-0.06  0.02  0.00  0.07  0.07  0.04 -0.18 -0.10  0.05]
  [-0.06  0.01  0.04  0.05  0.03 -0.01 -0.10 -0.07  0.00]
  [-0.04  0.00  0.04  0.05  0.02 -0.04 -0.02 -0.05  0.04]
]

Note: I’ve rounded the values because they are quite large, for example 0.01783941

Luckily, we don’t have to hand-pick this collection of kernels. That is what training does. The kernels all start off empty (or in a random state) and keep getting tweaked in the direction that makes the output closer to what we want.

This is what the 256 kernels ended up looking like (I colored them as pixels so it’s easier to digest). The more negative the numbers, the bluer they are. 0 is green and positive is yellow:

After we filter the image with all of these kernels, we end up with a fat stack of 256 output images.

Part 1b: ReLU

ReLU (formally known as Rectified Linear Unit) may sound complicated, but it’s actually quite simple. ReLU is an activation function that takes in a value. If it’s negative it becomes zero, and if it’s positive it stays the same.

In code:

x = max(0, x)

And as a graph:

We apply this function to all of the outputs of our convolutions.

Why do we do this? If we don’t apply some sort of activation function to the output of our layers, then the entire neural net could be described as a linear function. This would mean that all this stuff we are doing is kind of pointless.

Adding a non-linearity allows us to describe all kinds of functions. There are many different types of function we could apply, but ReLU is the most popular because it’s very cheap to perform.

Here are the outputs of ReLU Conv1 layer:

Part 2a: PrimaryCaps

The PrimaryCaps layer starts off as a normal convolution layer, but this time we are convolving over the stack of 256 outputs from the previous convolutions. So instead of having a 9x9 kernel, we have a 9x9x256 kernel.

So what exactly are we looking for?

In the first layer of convolutions we were looking for simple edges and curves. Now we are looking for slightly more complex shapes from the edges we found earlier.

This time our “stride” is 2. That means instead of moving 1 pixel at a time, we take steps of 2. A larger stride is chosen so that we can reduce the size of our input more rapidly:

Note: The dimension of the output would normally be 12, but we divide it by 2, because of the stride. For example: ((20 — 9) + 1) / 2 = 6

We will convolve over the outputs another 256 times. So we will end up with a stack of 256 6x6 outputs.

But this time we aren’t satisfied with just some lousy plain old numbers.

We’re going to cut the stack up into 32 decks with 8 cards each deck.

We can call this deck a “capsule layer.”

Each capsule layer has 36 “capsules.”

If you’re keeping up (and are a math wiz), that means each capsule has an array of 8 values. This is what we can call a “vector.”

Here’s what I’m talking about:

These “capsules” are our new pixel.

With a single pixel, we could only store the confidence of whether or not we found an edge in that spot. The higher the number, the higher the confidence.

With a capsule we can store 8 values per location! That gives us the opportunity to store more information than just whether or not we found a shape in that spot. But what other kinds of information would we want to store?

When looking at the shape below, what can you tell me about it? If you had to tell someone else how to redraw it, and they couldn’t look at it, what would you say?

This image is extremely basic, so there are only a few details we need to describe the shape:

• Type of shape
• Position
• Rotation
• Color
• Size

We can call these “instantiation parameters.” With more complex images we will end up needing more details. They can include pose (position, size, orientation), deformation, velocity, albedo, hue, texture, and so on.

You might remember that when we made a kernel for edge detection, it only worked on a specific angle. We needed a kernel for each angle. We could get away with it when dealing with edges because there are very few ways to describe an edge. Once we get up to the level of shapes, we don’t want to have a kernel for every angle of rectangles, ovals, triangles, and so on. It would get unwieldy, and would become even worse when dealing with more complicated shapes that have 3 dimensional rotations and features like lighting.

That’s one of the reasons why traditional neural nets don’t handle unseen rotations very well:

As we go from edges to shapes and from shapes to objects, it would be nice if we had more room to store this extra useful information.

Here is a simplified comparison of 2 capsule layers (one for rectangles and the other for triangles) vs 2 traditional pixel outputs:

Like a traditional 2D or 3D vector, this vector has an angle and a length. The length describes the probability, and the angle describes the instantiation parameters. In the example above, the angle actually matches the angle of the shape, but that’s not normally the case.

In reality it’s not really feasible (or at least easy) to visualize the vectors like above, because these vectors are 8 dimensional.

Since we have all this extra information in a capsule, the idea is that we should be able to recreate the image from them.

Sounds great, but how do we coax the network into actually wanting to learn these things?

When training a traditional CNN, we only care about whether or not the model predicts the right classification. With a capsule network, we have something called a “reconstruction.” A reconstruction takes the vector we created and tries to recreate the original input image, given only this vector. We then grade the model based on how close the reconstruction matches the original image.

I will go into more detail on this in the coming sections, but here is a simple example:

Part 2b: Squashing

After we have our capsules, we are going to perform another non-linearity function on it (like ReLU), but this time the equation is a bit more involved. The function scales the values of the vector so that only the length of the vector changes, not the angle. This way we can make the vector between 0 and 1 so it’s an actual probability.

This is what lengths of the capsule vectors look like after squashing. At this point it’s almost impossible to guess what each capsule is looking for.

Part 3: Routing by Agreement

The next step is to decide what information to send to the next level. In traditional networks, we would probably do something like “max pooling.” Max pooling is a way to reduce size by only passing on the highest activated pixel in the region to the next layer.

However, with capsule networks we are going to do something called routing by agreement. The best example of this is the boat and house example illustrated by Aurélien Géron in this excellent video. Each capsule tries to predict the next layer’s activations based on itself:

Looking at these predictions, which object would you choose to pass on to the next layer (not knowing the input)? Probably the boat, right? both the rectangle capsule and the triangle capsule agree on what the boat would look like. But they don’t agree on how the house would look, so it’s not very likely that the object is a house.

With routing by agreement, we only pass on the useful information and throw away the data that would just add noise to the results. This gives us a much smarter selection than just choosing the largest number, like in max pooling.

With traditional networks, misplaced features don’t faze it:

With capsule networks, the features wouldn’t agree with each other:

Hopefully, that works intuitively. However, how does the math work?

We have 10 different digit classes that we are predicting:

0, 1, 2, 3, 4, 5, 6, 7, 8, 9

Note: In the boat and house example we were predicting 2 objects, but now we are predicting 10.

Unlike in the boat and the house example, the predictions aren’t actually images. Instead, we are trying to predict the vector that describes the image.

The capsule’s predictions for each class are made by multiplying it’s vector by a matrix of weights for each class that we are trying to predict.

Remember that we have 32 capsule layers, and each capsule layer has 36 capsules. That means we have a total of 1,152 capsules.

cap_1 * weight_for_0 = prediction
cap_1 * weight_for_1 = prediction
cap_1 * weight_for_2 = prediction
cap_1 * ...
cap_1 * weight_for_9 = prediction

cap_2 * weight_for_0 = prediction
cap_2 * weight_for_1 = prediction
cap_2 * weight_for_2 = prediction
cap_2 * ...
cap_2 * weight_for_9 = prediction

...

cap_1152 * weight_for_0 = prediction
cap_1152 * weight_for_1 = prediction
cap_1152 * weight_for_2 = prediction
cap_1152 * ...
cap_1152 * weight_for_9 = prediction

You will end up with a list of 11,520 predictions.

Each weight is actually a 16x8 matrix, so each prediction is a matrix multiplication between the capsule vector and this weight matrix:

As you can see, our prediction is a 16 degree vector.

Where does the 16 come from? It’s an arbitrary choice, just like 8 was for our original capsules.

But it should be noted that we want to increase the number of dimensions of our capsules the deeper we get into the network. This should make sense intuitively, because the deeper we go the more complex our features become and the more parameters we need to recreate them. For example, you will need more information to describe an entire face than just a person’s eye.

The next step is to figure out which of these 11,520 predictions agree with each other the most.

It can be difficult to visualize a solution to this when we think in terms of high dimensional vectors. For the sake of sanity, let’s start off by pretending our vectors are just points in 2 dimensional space:

We start off by calculating the mean of all of the points. Each point starts out with equal importance:

We then can measure the distance between every point from the mean. The further the point is away from the mean, the less important that point becomes:

We then recalculate the mean, this time taking into account the point’s importance:

We end up going through this cycle 3 times:

As you can see, as we go through this cycle, the points that don’t agree with the others start to disappear. The highest agreeing points end up getting passed on to the next layer with the highest activations.

Part 4: DigitCaps

After agreement, we end up with ten 16 dimensional vectors, one vector for each digit. This matrix is our final prediction. The length of the vector is the confidence of the digit being found — the longer the better. The vector can also be used to generate a reconstruction of the input image.

This is what the lengths of the vectors look like with the input of 4:

The fifth block is the brightest, which means high confidence. Remember that 0 is the first class, meaning 4 is our predicted class.

Part 5: Reconstruction

The reconstruction portion of the implementation isn’t very interesting. It’s just a few fully connected layers. But the reconstruction itself is very cool and fun to play around with.

If we reconstruct our 4 input from its vector, this is what we get:

If we manipulate the sliders (the vector), we can see how each dimension affects the 4:

I recommend cloning the visualization repo to play around with different inputs and see how the sliders affect the reconstruction:

git clone https://github.com/bourdakos1/CapsNet-Visualization.git
cd CapsNet-Visualization
pip install -r requirements.txt

Run the tool:

python run_visualization.py

Then point your browser to: http://localhost:5000

Final Thoughts

I think that the reconstructions from capsule networks are stunning. Even though the current model is only trained on simple digits, it makes my mind run with the possibilities that a matured architecture trained on a larger dataset could achieve.

I’m very curious to see how manipulating the reconstruction vectors of a more complicated image would affect it. For that reason, my next project is to get capsule networks to work with the CIFAR and smallNORB datasets.

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Understanding Capsule Networks — AI’s Alluring New Architecture was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.

“How can I draw a more detailed outline around an object?”

This is probably one of the most frequently asked questions I get after someone reads my previous article on how to do object detection using TensorFlow. There is good news, I finally have the answer.

Facebook AI Research (FAIR) just open sourced their Detectron platform. This means that the software that FAIR uses for object detection research is now available to all of us developers. One of the many things that this new platform can do is object masking. Object masking takes objected detection a step further and instead of just drawing a bounding box around the image, it can actually draw a complex polygon.

Detectron makes it incredibly simple to get object masking running out of the box. Here’s how:

Install Caffe2

The first step is to install Caffe2:

git clone --recursive https://github.com/caffe2/caffe2.git && cd caffe2
make && cd build && sudo make install
python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"

Note: Currently, you MUST have an NVIDIA GPU with CUDA in order to use Detectron.

Install the COCO API

git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
make install

Clone the Repo

After we successfully install Caffe2 all we need to do is clone the Detectron repo:

git clone https://github.com/facebookresearch/Detectron.git && cd Detectron

Install the requirements:

pip install \
numpy \
pyyaml \
matplotlib \
opencv-python>=3.0 \
setuptools \
Cython \
mock \
scipy

And build the python modules:

cd lib && make && cd ..

Making Inferences

Now all we have to do is run the following command to run object masking on the demo images:

python tools/infer_simple.py \
--cfg configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml \
--output-dir demo/output \
--image-ext jpg \
--wts \
https://s3-us-west-2.amazonaws.com/detectron/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.02_32_51.SgT4y1cO/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl \
demo

You’ll be able to find the output as PDFs in the Detectron/demo/output directory.

Here are some of my own test examples, most of them worked pretty well.

But as always, the Kardashians continue to stump AI, because are they real?

Note: If you want to test your own images just add JPG files to the Detectron/demo directory. For best results keep the images around 600–800px.

Final Thoughts

This model does a great job at masking very general objects, however a lot of us need something more specific to our use case. For that, keep a look out for my next tutorial on how to build your own custom object masking model!

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

How to use Detectron — Facebook’s Free Platform for Object Detection was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you follow AI you might have heard about the advent of the potentially revolutionary Capsule Networks. I will show you how you can start using them today.

Geoffrey Hinton is known as the father of “deep learning.” Back in the 50s the idea of deep neural networks began to surface and, in theory, could solve a vast amount of problems. However, nobody was able to figure out how to train them and people started to give up. Hinton didn’t give up and in 1986 showed that the idea of backpropagation could train these deep nets. However, it wasn’t until 5 years ago in 2012 that Hinton was able to demostrate his breakthrough, because of the lack of computational power of the time. This breakthrough set the stage for this decade’s progress in AI.

And now, on October 26, 2017, he has released a paper on a new groundbreaking concept, Capsule Networks.

Note: For a deeper explanation on how capsules work check out part 2. Also, the research papers can be found here and here.

Problems with Traditional Neural Networks

Up until now Convolutional Neural Networks (CNNs) have been the state-of-the-art approach to classifying images.

CNNs work by accumulating sets of features at each layer. It starts of by finding edges, then shapes, then actual objects. However, the spatial relationship information of all these features is lost.

This is a gross oversimplification, but you can think of a CNN like this:

if (2 eyes && 1 nose && 1 mouth) {
  It's a face!
}

You might be thinking that this sounds pretty good, it makes sense, and it does. Although, we might run into a few problems, take this picture of Kim Kardashian for example:

Yikes! There’s definitely two eyes, a nose and a mouth, but something is wrong, can you spot it? We can easily tell that an eye and her mouth are in the wrong place and that this isn’t what a person is supposed to look like. However, a well trained CNN has difficulty with this concept:

In addition to being easily fooled by images with features in the wrong place a CNN is also easily confused when viewing an image in a different orientation. One way to combat this is with excessive training of all possible angles, but this takes a lot of time and seems counter intuitive. We can see here the massive drop in performance by simply flipping Kim upside down:

Finally, convolutional neural networks can be susceptible to white box adversarial attacks. Which is essentially embedding a secret pattern into an object to make it look like something else.

“Convolutional neural networks are doomed” — Geoffrey Hinton

Capsule Networks to the Rescue!

The introduction of Capsule Networks gives us the ability to take full advantage of spatial relationship, so we can start to see things more like:

if (2 adjacent eyes && nose under eyes && mouth under nose) {
  It's a face!
}

You should be able to see that with this definition our neural net shouldn’t be as easily fooled by our misshapen Kardashian.

This new architecture also achieves significantly better accuracy on the following data set. This data set was carefully designed to be a pure shape recognition task that shows the ability to recognize the objects even from different points of view. It beat out the state-of-the-art CNN, reducing the number of errors by 45%.

Further more, in their most recent paper, they found that Capsules show far more resistance to white box adversarial attack than a baseline convolutional neural network.

Training CapsNet

I have pieced together a repo that is an implementation of Hinton’s paper (many thanks to naturomics). In order to use the Capsule Network model you first need to train it.

The following guide will get you a model trained on the MNIST data set. For those of you who don’t know, MNIST is a data set of handwritten digits and is a good baseline for testing out machine learning algorithms.

Start by cloning the repo:

git clone https://github.com/bourdakos1/capsule-networks.git

Then install the requirements and download the dataset:

pip install -r requirements.txt
python download_data.py

Start training!

python main.py

The MNIST data set is 60,000 training images. By default the model will be trained for 50 epochs at a batch size of 128. An epoch is one full run through the training set. Since the batch size is 128 it will do about 468 batches per epoch.

Note: Training might take a very long time if you don’t have a GPU. You can read this article on how to speed up training time.

Making Inferences

Once our model is fully trained we can test it by running the following command:

python main.py --is_training False

Final Thoughts

Capsule Networks seem awesome, but they are still babies. We could see problems in the future when training huge datasets, but I have faith.

If you enjoyed reading this and want more, there’s a part 2!

P.S. Here is a great video that I recommend taking the time to watch.

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Capsule Networks Are Shaking up AI — Here’s How to Use Them was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

At the time of writing this post, most of the big tech companies (such as IBM, Google, Microsoft, and Amazon) have easy-to-use visual recognition APIs. Some smaller companies also provide similar offerings, such as Clarifai. But none of them offer object detection.

Update: IBM and Microsoft now have customizable object detection APIs.

The following images were both tagged using the same Watson Visual Recognition default classifier. The first one, though, has been run through an object detection model first.

Object detection can be far superior to visual recognition on its own. But if you want object detection, you’re going to have to get your hands a little dirty.

Depending on your use case, you may not need a custom object detection model. TensorFlow’s object detection API provides a few models of varying speed and accuracy, that are based on the COCO dataset.

For your convenience, I have put together a complete list of objects that are detectable with the COCO models:

If you wanted to detect logos or something not on this list, you’d have to build your own custom object detector. I wanted to be able to detect the Millennium Falcon and some Tie Fighters. This is obviously an extremely important use case, because you never know…

Annotate your images

Training your own model is a lot of work. At this point, you may be thinking, “Whoa, whoa, whoa! I don’t want to do a lot of work!” If so, you can check out my other article about using the provided model. It’s a much smoother ride.

You need to collect a lot of images and annotate them all. Annotations include specifying the object coordinates and a corresponding label. For an image with two Tie Fighters, an annotation might look something like this:

<annotation>
    <folder>images</folder>
    <filename>image1.jpg</filename>
    <size>
        <width>1000</width>
        <height>563</height>
    </size>
    <segmented>0</segmented>
    <object>
        <name>Tie Fighter</name>
        <bndbox>
            <xmin>112</xmin>
            <ymin>281</ymin>
            <xmax>122</xmax>
            <ymax>291</ymax>
        </bndbox>
    </object>
    <object>
        <name>Tie Fighter</name>
        <bndbox>
            <xmin>87</xmin>
            <ymin>260</ymin>
            <xmax>95</xmax>
            <ymax>268</ymax>
        </bndbox>
    </object>
</annotation>

For my Star Wars model, I collected 308 images including two or three objects in each. I’d recommend trying to find 200–300 examples of each object.

“Wow,” you might be thinking, “I have to go through hundreds of images and write a bunch of XML for each one?”

Of course not! There are plenty of annotation tools out there, such as labelImg and RectLabel. I used RectLabel, but it’s only for macOS. It’s still a lot of work, trust me. It took me about three or four hours of nonstop work to annotate my entire dataset.

Update: I ended up building my own tool to annotate images and video frames. It’s a free online tool called Cloud Annotations that you can check out here.

If you have the money, you can pay somebody else, like an intern, to do it. Or you can use something like Mechanical Turk. If you are a broke college student like me and/or find doing hours of monotonous work fun, you’re on your own.

We will need to do a little setup before we can run the script to prepare the data for TensorFlow.

Clone the repo

Start by cloning my repo here.

Update: This repo is a bit out of date, I recommend checking out this one for a much better time :)

Note: The following instructions are also out of date, I urge you to check out the new walkthrough.

The directory structure will need to look like this:

models
|-- annotations
|   |-- label_map.pbtxt
|   |-- trainval.txt
|   `-- xmls
|       |-- 1.xml
|       |-- 2.xml
|       |-- 3.xml
|       `-- ...
|-- images
|   |-- 1.jpg
|   |-- 2.jpg
|   |-- 3.jpg
|   `-- ...
|-- object_detection
|   `-- ...
`-- ...

I’ve included my training data, so you should be able to run this out of the box. But if you want to create a model with your own data, you’ll need to add your training images to images, add your XML annotations to annotations/xmls, update trainval.txt, and label_map.pbtxt.

trainval.txt is a list of file names that allows us to find and correlate the JPG and XML files. The following trainval.txt list would let us to find abc.jpg, abc.xml, 123.jpg, 123.xml, xyz.jpg and xyz.xml:

abc
123
xyz

Note: Make sure your JPG and XML file names match, minus the extension.

label_map.pbtxt is our list of objects that we are trying to detect. It should look something like this:

item {
  id: 1
  name: 'Millennium Falcon'
}

item {
  id: 2
  name: 'Tie Fighter'
}

Running the script

First, with Python and pip installed, install the scripts requirements:

pip install -r requirements.txt

Add models and models/slim to your PYTHONPATH:

export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Important Note: This must be run every time you open the terminal, or added to your ~/.bashrc file.

Run the script:

python object_detection/create_tf_record.py

Once the script finishes running, you will end up with a train.record and a val.record file. This is what we will use to train the model.

Downloading a base model

Training an object detector from scratch can take days, even when using multiple GPUs. To speed up training, we’ll take an object detector trained on a different dataset and reuse some of it’s parameters to initialize our new model.

You can download a model from this model zoo. Each model varies in accuracy and speed. I used faster_rcnn_resnet101_coco.

Extract and move all the model.ckpt files to our repo’s root directory.

You should see a file named faster_rcnn_resnet101.config. It’s set to work with the faster_rcnn_resnet101_coco model. If you used another model, you can find a corresponding config file here.

Ready to train

Run the following script, and it should start to train!

python object_detection/train.py \
        --logtostderr \
        --train_dir=train \
        --pipeline_config_path=faster_rcnn_resnet101.config

Note: Replace pipeline_config_path with the location of your config file.

global step 1:
global step 2:
global step 3:
global step 4:
...

Yay! It’s working!

10 minutes later.

global step 41:
global step 42:
global step 43:
global step 44:
...

Computer starts smoking.

global step 71:
global step 72:
global step 73:
global step 74:
...

How long is this thing supposed to run?

The model that I used in the video ran for about 22,000 steps.

Wait, what?!

I use a specked-out MacBook Pro. If you’re running this on something similar, I’ll assume you’re getting about one step every 15 seconds or so. At that rate it will take about three to four days of nonstop running to get a decent model.

Well, this is dumb. I don’t have time for this 😵

PowerAI to the rescue!

PowerAI

PowerAI lets us train our model on IBM Power Systems with P100 GPUs fast!

It only took about an hour to train for 10,000 steps. However, this was just with one GPU. The real power in PowerAI comes from the the ability to do distributed deep learning across hundreds of GPUs with up to 95% efficiency.

With the help of PowerAI, IBM just set a new image recognition record of 33.8% accuracy in 7 hours. It surpassed the previous industry record set by Microsoft — 29.9% accuracy in 10 days.

WAYYY fast!

Since I’m not training on millions of images, I definitely didn’t need those kind of resources. One GPU will do.

Creating a Nimbix account

Nimbix provides developers a trial account with ten hours of free processing time on the PowerAI platform. You can register here.

Note: This process is not automated, so it may take up to 24 hours to be reviewed and approved.

Once approved, you should receive an email with instructions on confirming and creating your account. It will ask you for a promotional code, but leave it blank.

You should now be able to log in here.

Deploy the PowerAI Notebooks application

Start by searching for PowerAI Notebooks.

Click on it, and then choose TensorFlow.

Choose the machine type of 32 thread POWER8, 128GB RAM, 1x P100 GPU w/NVLink (np8g1).

Once started, the following dashboard panel will be displayed. When the server Status turns to Processing, the server is ready to be accessed.

Get the password by clicking on (click to show).

Then, click Click here to connect to launch the Notebook.

Log-in using the user name nimbix and the previously supplied password.

Start training

Get a new terminal window by clicking on the New pull-down and selecting Terminal.

You should be greeted with a familiar face:

Note: Terminal may not work in Safari.

The steps for training are the same as they were when we ran this locally. If you’re using my training data then you can just clone my repo by running (if not, just clone your own repo):

git clone https://github.com/bourdakos1/Custom-Object-Detection.git

Then cd into the root directory:

cd Custom-Object-Detection

Run this snippet, which will download the pre-trained faster_rcnn_resnet101_coco model we downloaded earlier.

wget http://storage.googleapis.com/download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_coco_11_06_2017.tar.gz
tar -xvf faster_rcnn_resnet101_coco_11_06_2017.tar.gz
mv faster_rcnn_resnet101_coco_11_06_2017/model.ckpt.* .

Then we need to update our PYTHONPATH again, because this in a new terminal:

export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Then we can finally run the training command again:

python object_detection/train.py \
        --logtostderr \
        --train_dir=train \
        --pipeline_config_path=faster_rcnn_resnet101.config

Downloading your model

When is my model ready? It depends on your training data. The more data, the more steps you’ll need. My model was pretty solid at nearly 4,500 steps. Then, at about 20,000 steps, it peaked. I even went on and trained it for 200,000 steps, but it didn’t get any better.

I recommend downloading your model every 5,000 steps or so and evaluating it to make sure you’re on the right path.

Click on the Jupyter logo in the top left corner. Then, navigate the file tree to Custom-Object-Detection/train.

Download all the model.ckpt files with the highest number.

• model.ckpt-STEP_NUMBER.data-00000-of-00001
• model.ckpt-STEP_NUMBER.index
• model.ckpt-STEP_NUMBER.meta

Note: You can only download one at a time.

Note: Be sure to click the red power button on your machine when finished. Otherwise, the clock will keep on ticking indefinitely.

Export the inference graph

To use the model in our code, we need to convert the checkpoint files (model.ckpt-STEP_NUMBER.*) into a frozen inference graph.

Move the checkpoint files you just downloaded into the root folder of the repo you’ve been using.

Then run this command:

python object_detection/export_inference_graph.py \
        --input_type image_tensor \
        --pipeline_config_path faster_rcnn_resnet101.config \
        --trained_checkpoint_prefix model.ckpt-STEP_NUMBER \
        --output_directory output_inference_graph

Remember export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim .

You should see a new output_inference_graph directory with a frozen_inference_graph.pb file. This is the file we need.

Test the model

Now, run the following command:

python object_detection/object_detection_runner.py

It will run your object detection model found at output_inference_graph/frozen_inference_graph.pb on all the images in the test_images directory and output the results in the output/test_images directory.

The results

Here’s what we get when we run our model over all the frames in this clip from Star Wars: The Force Awakens.

Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.

If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.

Tracking the Millennium Falcon with TensorFlow was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.