Let’s continue our journey to explore the best machine learning frameworks in computer vision.

In the first article we explored object detection with the official Tensorflow APIs. The second article was dedicated to an excellent framework for instance segmentation, Matterport Mask R-CNN based on Keras.

In this article we examine Keras implementation of RetinaNet object detection developed by Fizyr. RetinaNet, as described in Focal Loss for Dense Object Detection, is the state of the art for object detection.
The object to detect with the trained model will be my little goat Rosa.

The colab notebook and dataset are available in my Github repo.

In this article, we go through all the steps in a single Google Colab netebook to train a model starting from a custom dataset.

We will keep in mind these principles:

• illustrate how to make the annotation dataset
• describe all the steps in a single Notebook
• use free software, Google Colab and Google Drive, so it’s based exclusively on free cloud resources

At the end of the article you will be surprised by the simplicity of use and the good results we will obtain through this object detection framework.

Despite its ease of use, Fizyr is a great framework, also used by the winner of the Kaggle competition “RSNA Pneumonia Detection Challenge”.

Making the dataset

We start by creating annotations for the training and validation dataset, using the tool LabelImg. This excellent annotation tool let you quickly annotate the bounding boxes of the objects to train the machine learning model.

LabelImg creates annotations in PascalVoc format, so we need to convert annotations to Fizyr format:

• create a zip file containing training dataset images and annotations with the same filename (check my example dataset in Github)

objdet_dataset.zip
|- img1.jpg
|- img1.xml
|- img2.jpg
|- img2.xml
...

• Upload zip file in Google Drive, get Drive file id, and substitute the DATASET_DRIVEID value
• Run cell that iterates over the xml files and creates annotations.csv file

Note: you can see my answer on Stackoverflow to get the Drive file id.

Model training

Model training is the core of the notebook. Fizyr offers various parameters, described in Github, to run and optimize this step.

It’s a good option to start from a pretrained model instead of training a model from scratch. Fizyr released a model based on ResNet50 architecture, pretrained on Coco dataset.

URL_MODEL = 'https://github.com/fizyr/keras-retinanet/releases/download/0.5.0/resnet50_coco_best_v2.1.0.h5'

We can even use our pretrained model, and continue the training from it. This option is particularly useful to train for some epochs, so save it in Google Drive, and later restart the training from the saved model. In this way we can bypass the 12-hour execution limit in Colab, and we can train the model for many epochs.

From my tests, a high value of batch_size and steps offers better results, but they greatly increase the execution time of each epoch.

We can start training from our custom dataset with:

!keras_retinanet/bin/train.py --freeze-backbone --random-transform --weights {PRETRAINED_MODEL} --batch-size 8 --steps 500 --epochs 10 csv annotations.csv classes.csv

Let’s analyze each argument passed to the script train.py.

• freeze-backbone: freeze the backbone layers, particularly useful when we use a small dataset, to avoid overfitting
• random-transform: randomly transform the dataset to get data augmentation
• weights: initialize the model with a pretrained model (your own model or one released by Fizyr)
• batch-size: training batch size, higher value gives smoother learning curve
• steps: number of steps for epochs
• epochs: number of epochs to train
• csv: annotations files generated by the script above

The training process output contains a description of layers and loss metrics during training, and as you can see, loss metrics decrease during each epoch:

Using TensorFlow backend.
...
Layer (type)                    Output Shape         Param #     Connected to
input_1 (InputLayer)            (None, None, None, 3 0
padding_conv1 (ZeroPadding2D)   (None, None, None, 3 0           input_1[0][0]                    
...
Total params: 36,382,957
Trainable params: 12,821,805
Non-trainable params: 23,561,152
None
Epoch 1/10
500/500 [==============================] - 1314s 3s/step - loss: 1.0659 - regression_loss: 0.6996 - classification_loss: 0.3663
Epoch 2/10
500/500 [==============================] - 1296s 3s/step - loss: 0.6747 - regression_loss: 0.5698 - classification_loss: 0.1048
Epoch 3/10
500/500 [==============================] - 1304s 3s/step - loss: 0.5763 - regression_loss: 0.5010 - classification_loss: 0.0753

Epoch 3/10
500/500 [==============================] - 1257s 3s/step - loss: 0.5705 - regression_loss: 0.4974 - classification_loss: 0.0732

Inference

The last step performs inference of test images with the trained model.
The Fizyr framework allows us to perform inference using CPU, even if you trained the model with GPU. This feature is important in typical production environments, where people usually opt for less expensive hardware infrastructures for inference, without GPUs.

Let’s examine the following lines in detail:

model_path = os.path.join('snapshots', sorted(os.listdir('snapshots'), reverse=True)[0])
print(model_path)

# load retinanet model
model = models.load_model(model_path, backbone_name='resnet50')
model = models.convert_model(model)

The first line sets the model file as the last model generated by the training process in /snapshots directory. Then the model is loaded from the filesystem and converted to run inference.

You can change the values of THRES_SCORE, which represents the confidence threshold to show an object detection.

Conclusions

We went through the complete journey to make object detection with Fizyr implementation of RetinaNet. We created a dataset, trained a model, and ran inference (here is my Github repo for the notebook and dataset).

I was impressed by the following aspects of this excellent framework:

• this framework is easy to use to get good inference, even without much customization
• it was simple to transform annotations to Fizyr’s dataset format, compared to other frameworks.

In general Fizyr is a good choice to start an object detection project, in particular if you need to quickly get good results.

If you enjoyed this article, leave a few claps, it will encourage me to explore further machine learning opportunities :)

Object Detection in Colab with Fizyr Retinanet was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.

Previous article was about Object Detection in Google Colab with Custom Dataset, where I trained a model to infer bounding box of my dog in pictures. The protagonist of my article is again my dog: in this case we take a step forward, we identify not only the bounding box, we make even pixel wise classification.

Compared to previous article, we hold the same characteristics:

• Only requirement is the dataset, created with annotation tool
• A single Google Colab notebook contains all the steps: it starts from the dataset, executes the model’s training and shows inference
• It runs in Google Colab (GPU enabled environment) and Google Drive storage, so it’s based exclusively on free cloud resources

These features allow anybody following this tutorial to create an instance segmentation model, and test it in Google Colab or export the model to run in a local machine.

Source code of this article, including the sample dataset, is available in my Github repo.

Choosing Framework

There are various open source frameworks to implement instance segmentation, you can find an overview in this presentation of Stanford University.

We discard solutions that are not based on Tensorflow, such as Facebook Detectron based on Caffe2, because we decided to train the model in Google Colab, that is already integrated with Tensorflow.

One of the most popular frameworks, easy to use and well documented, is Matterport Mask R-CNN. From my tests it’s one of the simplest and most robust implementations available.

In addition, a big effort I faced with other implementations is to convert the annotations output file to framework input format. To be clear, once you create the pixel annotations of dataset with a graphical tool, you should convert it to input format defined by training framework.

Matterport developed this task in a clear article, demonstrating how to transform annotations file to Matterport Mask R-CNN format.

Making Dataset

In previous article we created bounding box annotations to obtain object detection model, now we are going to train instance segmentation model, therefore we create pixel level mask annotations to define the boundaries of the objects in dataset. Among various available tools, I chose an intuitive and well done tool: VGG Image Annotator (VIA) by University of Oxford, you can see documentation in the official page of the project. Furthermore, it’s easy to integrate VIA with Matterport framework.

This tool doesn’t need any installation, you just download the package and open the via.htmlfile with a modern browser.

It’s important to create a good dataset to achieve a well performing trained model. Taking pictures of objects with different lighting conditions, from various angles and in different contexts, are good principles to obtain a well generalized model, and avoid overfitting.

At the end of the annotation process, I created “images.zip” file with the following structure:

images.zip
|- "train" directory
  |- jpg image files of training data
  |- "via_region_data.json" annotations file of training data
|- "val" directory
  |- jpg image files of validation data
  |- "via_region_data.json" annotations file of validation data

Lastly I uploaded zip file into Google Drive, to use it during the training and test process. I included the dataset file in my Gitub repo, having pixel wise annotations of dog images.

Training Model

All the steps are in Google Colab notebook included in my repo. In my example, training process last about half an hour for 5 epochs, to get a more accurate model you can increase the number of epochs and the dataset size.

I selected Python3 GPU enabled environment, to use up to 12 hours of Tesla K80 GPU offered in Google Colab. Next steps in notebook are:

Install required packages: install packages, repositories and environment variables for Matterport instance segmentation with Tensorflow.

Download and extract dataset: download images.zip dataset in Google Colab filesystem, previously uploaded in Google Drive. Update fileId variable with Google Drive id of your image.zip dataset.

Edit settings file: code in my repo is inspired by Matterport Splash of Color sample, to run with a different dataset you should replace occurrences of “balloon” and “Balloon” with the name of object.

Train model: use pretrained weights to apply transfer learning in training process. Options are COCO and ImageNet.

Training process outputs the structure of neural network and various parameters, like the network architecture (Resnet50 or Resnet101).

Using TensorFlow backend. 
Weights:  coco 
Dataset:  dataset/ 
Logs:  /logs

Configurations: 
BACKBONE                       resnet101 
BACKBONE_STRIDES               [4, 8, 16, 32, 64] 
BATCH_SIZE                     2
...
GPU_COUNT                      1 
GRADIENT_CLIP_NORM             5.0 
IMAGES_PER_GPU                 2
...
Selecting layers to train 
fpn_c5p5               (Conv2D) 
fpn_c4p4               (Conv2D) 
fpn_c3p3               (Conv2D) 
fpn_c2p2               (Conv2D) 
...

Below the Tensorboard charts of training process:

Inference

Finally we can run test dataset inference with trained model.

Output includes inference data (image resolution, anchors shapes, …), and test images with bounding box, segmentation mask and confidence score.

Conclusions

If you want to run instance segmentation on a single object class, you can make a few minor changes to my Github code and adapt it to your dataset.

I hope you liked this article, in case leave some claps, it will encourage me to write other practical articles about machine learning for computer vision :)

Instance Segmentation in Google Colab with Custom Dataset was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

All the code and dataset used in this article is available in my Github repo.

Peculiarities of this proposal are:

• Only requirement is the dataset created with LabelImg
• A single Google Colab notebook contains all the steps: it starts from the dataset, executes the model’s training and shows inference
• It runs in Google Colab’s GPU enabled and Google Drive storage, so it’s based exclusively on free cloud resources

Furthermore, important changes have recently been made to Tensorflow’s Object Detection api, that made obsolete other available tutorials.

Making dataset

The only step not included in the Google Colab notebook is the process to create the dataset.

With an appropriate number of photos (my example have 50 photos of dog), I created the annotations. The tool I used is LabelImg. For the sake of simplicity I identified a single object class, my dog. It’s possible to extend it to obtain models that perform object detection on multiple object classes.

I renamed the image files in the format objectclass_id.jpg (i.e. dog_001.jpg, dog_002.jpg). Then in LabelImg, I defined the bounding box where the object is located, and I saved annotations in Pascal Voc format.

Finally I uploaded annotations files in my Google Drive account, using a single zip file with the following structure:

.zip file
|-images directory
  |-image files (filename format: objectclass_id.jpg)
|-annotations directory
  |-xmls directory
    |-annotations files (filename format: objectclass_id.xml)

Check my dataset file in Github to see an example.

Model training

All the next steps are included in the Google Colab notebook. I execute cells in sequence to train the model and run inference:

Install required packages: install packages, repositories and environment variables for object detection in Tensorflow, then run a test.

Download and extract dataset: download in the filesystem the dataset created. It’s important that the zip file has the structure explained above.

Empty png files: this is a cell to avoid error in create_pet_tf_record.py, it has not any effect in training process.

Create TFRecord: from the dataset it creates the TFRecord. In this simplified version, algorithm will train model only for one class.

Download pretrained model: download pretrained model from ModelZoo as initial checkpoint for transfer learning. In the example we download the model faster_rcnn_inception_v2_coco, to use another model from ModelZoo change MODEL var.

MODEL = 'faster_rcnn_inception_v2_coco_2018_01_28'
MODEL_FILE = MODEL + '.tar.gz'
DOWNLOAD_BASE = 'http://download.tensorflow.org/models/object_detection/'
DEST_DIR = 'pretrained_model'

Edit model config file: set the fields of the config file, identified by PATH_TO_BE_CONFIGURED. If you choose a different initial checkpoint model, update accordingly filename var and re.sub functions in the cell.

Train model: this is the main step, it performs the train of the model with the data and the configurations so far created. It is possible to change the number of steps in train and in validation.

Below the Tensorboard charts resulting from training process:

Inference

Export trained model: export model to run inference. The cell converts last trained model to the format to run inference.

Upload image for inference: browser upload of test image file to run inference in the next step.

Run inference: finally it performs inference of the uploaded image, and shows the result below.

Next goals

Thanks a lot for reading my article. In this article we easily trained an object detection model in Google Colab with custom dataset, using Tensorflow framework. If you liked, leave some claps, I will be happy to write more about machine learning.

In next articles we will extend the Google Colab notebook to:

• Include multiple classes of object detection
• View Tensorboard in a different browser tab during model training
• Perform instance segmentation to do pixel wise classification

Object Detection in Google Colab with Custom Dataset was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.