Don’t know how to run Tensorflow Object Detection? In this tutorial, I will show you 10 simple steps to run it on your own machine! We will use Tensorflow version 1.8. Are you ready to start detecting objects?

This guide will help you install Tensorflow on GPU enabled host. You will need Nvidia GPU with Compute Capability equal to 3.0 or greater. You can check CC here. If you don’t have such GPU you can omit CUDA and CUDNN installation and just install tensorflow without GPU support.

You can try to use these steps to install Tensorflow on Windows, but you will probably need some additional steps to accomplish it.

0. Operating system

I recommend Ubuntu 16.04 / 18.04. You can try Windows but Tensorflow doesn’t provide GPU enabled version for this OS. In Apple, there is no GPU anymore. In this tutorial, I will focus on Ubuntu and I will show you some tricks which will help to install.

1. Install CUDA Toolkit 9.0

Wow, why should I install 9.0 and not 9.1 or 9.2? Because Tensorflow 1.8 is built with 9.0, so you can just download the prebuild version. If you really need the bleeding edge not-stable version you can waste few days and compile it.

From the CUDA Toolkit Archive download version specific for your OS. In case of Ubuntu don’t use run-file installer — it destroys graphics driver and you will be stuck in the login loop. Use deb remote or local.

Download from here: https://developer.nvidia.com/cuda-90-download-archive

2. Install CUDNN 7.0.5 for CUDA 9.0

CUDNN provides functions specific to neural networks that are used by Tensorflow. It is extremely important to download the correct version for your CUDA and OS.

You will need to register. Registration is free.

https://developer.nvidia.com/rdp/cudnn-archive

3. Install python

If you are already using python just go to next step. There are many ways to install python. I will show you three of them, namely Anaconda, Virtualenv and native.

Anaconda and Virtualenv allow you to have multiple environments with different sets of libraries.

Anaconda

You can download it from here: https://www.anaconda.com/download/

Virtualenv

Here you will find instructions to download Virtualenv: https://virtualenv.pypa.io/en/stable/installation/

Native python

You will need to install python and package management system pip. Here is simple way to install python3 on Ubuntu.

sudo apt-get update
sudo apt-get install -y build-essential libssl-dev libffi-dev python3-dev python3 python3-pip

4. Install Tensorflow

When you prepared your python environment, you are ready to install Tensorflow:

Install Tensorflow with GPU support:

pip install tensorflow-gpu

Install Tensorflow without GPU support:

pip install tensorflow

5. Download Tensorflow repository

If you don’t have git installed, install git with following command:

sudo apt install -y git

Now we will clone Tensorflow and Tensorflow models repositories:

git clone https://github.com/tensorflow/tensorflow
cd tensorflow
git clone https://github.com/tensorflow/models

These repositories have tags matching versions of Tensorflow. You can checkout to a particular version, but from my experience you can leave it on branch master with latest commit.

6. Install Object Detection API

Now we will install the APU using this guide:https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md

So let’s install necessary libraries:

sudo apt-get install protobuf-compiler python-pil python-lxml python-tk
pip install --user Cython
pip install --user contextlib2
pip install --user jupyter
pip install --user matplotlib

7. Install COCO API

We will need to for model evaluation, but it is a good idea to install it just now. Change <path_to_tensorflow> to path to your Tensorflow repository.

git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
make
cp -r pycocotools <path_to_tensorflow>/models/research/

8. Install Protoc

You can download it here: https://github.com/google/protobuf/releases

For 64-bit Ubuntu you will need: protoc-3.6.0-linux-x86_64.zip

After downloading run following command:

# From tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.

Troubleshooting

1. protoc not found
2. Install protoc in /usr/bin/local
3. Run protoc by providing the absolute path to protoc executable
4. Protoc returns anything (shouldn’t print any message)
5. Install the newest protoc
6. Make sure that you haven’t installed protoc from Ubuntu repository using apt.

9. Install object detection

If you want to run object detection from any directory, I guess you want, you will need to add the path to your ~/.bashrc file. Obviously change <path_to_tensorflow> , replace ~with $HOME.

export PYTHONPATH=$PYTHONPATH:<path_to_tensorflow>/models/research/:<path_to_tensorflow>/models/research/slim

10. Install Jupyter Notebook

This is a powerful tool! With the notebook, you can write your code interactively in your web browser. You will be able to run code cells step by step or all at once. Give it a try!

Here you will find installation instructions: http://jupyter.org/install

If you are new to jupyter notebook:

• Yes, you can download your code as .py file
• The notebook is good to create a rapid demo
• You can also run system commands using exclamation mark e.g.!ls -la
• Give it a try!

11. Run the demo now!

Now you are ready to run following demo in jupyter notebook object_detection_tutorial.ipynb.

cd tensorflow/models/research/object_detection
jupyter-notebook object_detection_tutorial.ipynb

Now you can run each cell or all at once. The result will show up after the very last code cell. You will see detections on all of two images from test_images directory.

Other models

In this notebook, you can check different models by changing the MODEL_NAME.

MODEL_NAME = 'mask_rcnn_inception_v2_coco_2018_01_28'

Here you will find a list of available models: Model ZOO

If you want to use models trained on datasets other than MS COCO you will need to chage PATH_TO_LABELS respectively.

Using other models you can detect object masks!

What’s next?

Now you know how to run object detection for single images. You can try one of these ideas:

• Test nets on your images by copying them toobject_detection/test_images
• Check other models with bounding boxes and masks!
• Run object detection for the video — like here
• Try to track objects in following frames
• Run it on your android phone — like here

Do you have any other ideas?
Any questions or problems?
Write your comment below!

I will try to write about all these ideas in future posts!

Conclusion

Now you know how to install Tensorflow in simple steps. You prepared your environment to run object detection, and you ran object detection using one or more models.

Did you like this post?
What can be improved?
Please provide your feedback!

Links

• Repozytorium Tensorflow
• Repozytorium Tensorflow models
• ZOO
• TF Object Detection API Tutorial by Sentdex

Instance segmentation, object detection, drivable areas and lane markings — all you can find in Berkley DeepDrive 100K Dataset. It consists of more than 100 000 HD videos recorded at various times, seasons and weather. The dataset includes localization, timestamp and IMU data.

Data were collected in 4 locations which 3 are close to each other (SF, Berkeley and Bay Area), and the last one is New York.

Distribution across various weather conditions is not uniform — plots are in log scale!

In this dataset, there are only 10 object categories:

1. bike
2. bus
3. car
4. motor
5. person
6. rider
7. traffic light
8. traffic sign
9. train
10. truck

Instance Segmentation Instances [4]Instance segmentation categories:

1. banner
2. billboard
3. lane divider
4. parking sign
5. pole
6. polegroup
7. street light
8. traffic cone
9. traffic device
10. traffic light
11. traffic sign
12. sign frame
13. person
14. rider
15. bicycle
16. bus
17. car
18. caravan
19. motorcycle
20. trailer
21. train
22. truck

Drivable area categories:

• area/alternative
• area/drivable

Lane marking categories:

• lane/crosswalk
• lane/double other
• lane/double white
• lane/double yellow
• lane/road curb
• lane/single other
• lane/single white
• lane/single yellow

To sum up, we get a new dataset with the relatively low number of categories. These categories focused on road object detection.

I highly recommend training your own neural network using this dataset!
If you need any help — please contact me!

Links

1. Source Code on Github
2. BDD100K Page
3. BAIR Post
4. arXiv Paper

If you like this work show me your support!
Follow me on:

• YouTube: https://www.youtube.com/karolmajek
• Twitter: https://twitter.com/karol_majek
• Medium: https://medium.com/@karol_majek
• My personal page: http://karolmajek.pl/
• Deep Drive PL blog: https://deepdrive.pl/

Originally published on: https://deepdrive.pl/bdd100k-dataset/

30th April 2018 new version of Open Images Dataset V4 is released. There is also announced a challenge for best object detection results using this dataset.

Here you can see data examples: Open Images Dataset V4

ECCV 2018 Open Images Challenge

During ECCV 2018 conference there will be a workshop dedicated Open Images Challenge (presented by Vittorio Ferrari, Alina Kuznetsova, Jasper Uijlings, Rodrigo Benenson, Victor Gomes, Matteo Malloci). They will announce challenge results.

The Challenge has two tracks:

1. Object Class Detection: predicting a tight bounding box around all instances of the 500 classes.
2. Visual Relationship Detection: detecting pairs of objects in particular relations, e.g. “woman playing guitar”.

Dataset

There are image-level labels (table 1) and bounding boxes for object detection (table 2). Labels where generated automatically by machine and then checked by humans thanks to Crowdsource labeler.

Examples

Here are some examples, more you can find here: Open Images Dataset V4

Dates

• April 30th 2018: training set for object detection track released (with bounding box annotations).
• May 10 2018: visual relationship detection annotations on the training set will be released.
• May 31 2018: evaluation metric protocols and implementation will be released (as a part of the TF Object Detection API).
• July 1st 2018: a test set of 100k images will be released by Kaggle.
• September 1st 2018: deadline for submission of results.
• September 8th 2018: Open Images Challenge Workshop

Links

• Open Images Dataset V4
• Crowdsource labeler

If you like this work show me your support!
Follow me on:

• YouTube: https://www.youtube.com/karolmajek
• Twitter: https://twitter.com/karol_majek
• Medium: https://medium.com/@karol_majek
• My personal page: http://karolmajek.pl/
• Deep Drive PL blog: https://deepdrive.pl/

Originally published on: https://deepdrive.pl/open-images-dataset-v4-en/

I want to share my datasets I use for testing deep neural networks. I have already tested on 4k videos:

• Darknet YOLO
• Tensorflow Object Detection API: SSD, Faster RCNN
• Segmentation
• Mask RCNN

YOLO

I am using original implementation (Darknet by Joseph Redmon) with 4 different trained weights files. These weights are:

• Tiny YOLO VOC [https://pjreddie.com/media/files/tiny-yolo-voc.weights]
• YOLO VOC [https://pjreddie.com/media/files/yolo-voc.weights]
• YOLO COCO [https://pjreddie.com/media/files/yolo.weights]
• YOLO 9000 [https://pjreddie.com/media/files/yolo9000.weights]

Weights are downloaded from: pjreddie.com/darknet/yolo
Source code: https://github.com/karolmajek/darknet

SSD

I was using SSD Keras by Andrey Rykov:

rykov8/ssd_keras

This was easiest/fastest [for me] to run and get results. But now I switched to Tensorflow Object Detection API. Special thanks to Harrison Kinsley for great tutorial on using TF object detection API. I recommend this for everyone who wants to start training SSD:

Python Programming Tutorials

You can find results for models from TF model zoo:

• ssd_mobilenet_v1_coco
• ssd_inception_v2_coco
• faster_rcnn_nas
• [more coming soon]

tensorflow/models

Datasets — DOWNLOAD 4K

Need 4k dashcam vids? Download and make awesome things!

Datasets were recorded with Samsung S7 in 4K resolution.

Here is one of the videos:

Deep Drive PL Warsaw 0002 sequence - 4K dataset : Karol Majek : Free Download, Borrow, and Streaming : Internet Archive

Results

In this post I would like to show how to easily run image recognition in the cloud with a little help of powerful deep learning models. Several models are accessible using one REST API interface. You can upload your image and get specified insights. You can choose from following:

• Face detection [33 landmarks, emotional state, wearing headwear, Facial Recognition is NOT supported]
• Landmark detection [detect popular structures, predicts location]
• logo detection [detect popular product logos]
• label detection [image category]
• OCR [Optical Character Recognition, detect text, automatic language identification]
• document text [text and structure of document]
• safe search [detect explicit content such as adult or violent content]
• image properties [dominant colors, crop hints]
• web detection [relevant information from the Internet]

API have examples to run in following languages: C#, Go, Java, Node.js, PHP, Python and Ruby. There is also example how to use Curl to get results from the cloud.

You will need to register to Google Cloud Platform (you can try for free), install GCP client for your OS, login in using client.

Code for this post can be found here:

karolmajek/GCP-Vision-API

Face detection

Vision API can detect faces and localize features. The online demo is limited to detect up to 50 faces per image, but using API you can limit the maximum number of detections by yourself.

Face landmarks

Landmark detection can find 33 face landmarks: eyes (19), nose(4), mouth(5), chin(3), … You can find the list here: https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#Type_1

Curl

Here you can access Google tutorial for Curl:

Using curl to send requests | Google Cloud Vision API Documentation | Google Cloud Platform

Yeah! With curl you can get results in few seconds, let’s go! You will need:

• API Key — You need access to GCP Console; To create key go to https://console.cloud.google.com/apis/credentials
• input image — URI to input image; Google Cloud Storage can be used; link will be stored in request
• request.json — file which defines input image and the results we want to get
• Curl — library can help with this task

This is example request.json file which I used in this tutorial [imageUri — stores the link to input image]:

{
  "requests": [
    {
      "image": {
        "source": {
          "imageUri": "https://raw.githubusercontent.com/karolmajek/GCP-Vision-API/master/images/kinship.jpg"
        }
      },
      "features": [
        {
          "type": "FACE_DETECTION",
          "maxResults": 1000
        }
      ]
    }
  ]
}

Now we can use API with Curl [replace API_KEY with key you defined in GCP Console:

curl -v -s -H "Content-Type: application/json" \
    https://vision.googleapis.com/v1/images:annotate?key=API_KEY \
    --data-binary @request.json

Et voilà

The result is send as json string. You should receive response which can look like this one below. You can see the values, it can work with your production software, but you need to see what is really detected!

Ruby — I need to see detections!

Getting json string is really good too have, but we need to see the result and check if it looks good or not. So here is the Ruby example which will display the results:

GoogleCloudPlatform/ruby-docs-samples

To run this example you will need to:

1. Install Ruby on your machine — here you can find tutorial how to do it on different platforms: https://gorails.com/setup
2. You will need to install ImageMagick and rmagick. Type gem install rmagick to install rmagick for ruby.

Python — easy to integrate!

You will need to install Google Cloud for python:

pip install — upgrade google-cloud

I am using python sample code which I added to my repo. Original code can be found here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/vision/cloud-client/detect

python detect.py faces-uri https://raw.githubusercontent.com/karolmajek/GCP-Vision-API/master/images/kinship.jpg

This code limits the maximum number of detection to default value of 10. The output should look similarly to this:

Faces:
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (939,209),(996,209),(996,276),(939,276)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (401,844),(458,844),(458,910),(401,910)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (666,60),(823,60),(823,243),(666,243)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (501,582),(555,582),(555,644),(501,644)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (1349,281),(1406,281),(1406,347),(1349,347)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (1450,581),(1499,581),(1499,644),(1450,644)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (1350,843),(1407,843),(1407,910),(1350,910)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (1033,338),(1096,338),(1096,411),(1033,411)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (434,839),(498,839),(498,912),(434,912)
anger: VERY_UNLIKELY
joy: VERY_LIKELY
surprise: VERY_UNLIKELY
face bounds: (1033,901),(1096,901),(1096,974),(1033,974)

Visualization with Python

In this example the parameter max-results is available but it is not used, so there is hard limit for 10 results in this example. I reported this issue to GCP.

python faces.py --out out.jpg --max-results 1000 images/kinship.jpg

OCR, documents

Cloud Vision is able to detect text and the structure of documents as shown below.

Links

• CLOUD VISION API https://cloud.google.com/vision/
• How to Use the Google Cloud Vision API in Android Apps https://code.tutsplus.com/tutorials/how-to-use-the-google-cloud-vision-api-in-android-apps--cms-29009

If you like this work show me your support!
Follow me on:

• YouTube: https://www.youtube.com/karolmajek
• Twitter: https://twitter.com/karol_majek
• Medium: https://medium.com/@karol_majek

How to use Google Cloud Vision API was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is MooseFS?

MooseFS is a distributed file system. It spreads data over several physical commodity servers, which are visible to the user as one resource. For standard file operations MooseFS acts like any other Unix-like file system:

• A hierarchical structure (directory tree)
• Stores POSIX file attributes (permissions, last access and modification times)
• Supports special files (block and character devices, pipes and sockets)
• Symbolic links (file names pointing to target files, not necessarily on MooseFS) and hard links (different names of files which refer to the same data on MooseFS)

Distinctive features of MooseFS

• High availability (i.e. redundant metadata servers)
• High reliability (several copies of the data can be stored on separate computers)
• Capacity is dynamically expandable by simply adding new computers/disks
• Deleted files are retained for a configurable period of time (a file system level “trash bin”)
• Coherent snapshots of files, even while the file is being written/accessed
• Erasure coding support, for corrupted data recovery

Why use distributed file system to store data?

Exabytes of data are created every day. We have more and more tools to process those data, we are using deep neural networks for data analytics which requires reliable, large, highly available storage. Deep learning or machine learning in general is only a small part of the Big Data Landscape. The main reason to use distributed storage is requirement of high reliability and this goal can be achieved by using multiple machines. Another reason is high availability which is achieved by storing several copies of files. It is also easier to achieve higher capacities using many machines.

MooseFS Docker Cluster

In this tutorial we will use images from MooseFS Docker Hub to run cluster with 5 chunkservers, master and client machine. There is also another configuration available on GitHub repository — one master and 5 chunkserver/client machines. After a successful installation you will have a fully working MooseFS cluster!

Requirements

• Host machine with 64-bit operating system (Windows / Linux / Mac) compatible with Docker
• Git

Docker and Docker Compose installation

To install Docker with Docker compose on your Ubuntu machine please use:

• DigitalOcean tutorial (Ubuntu 16.04)
• DigitalOcean tutorial (Ubuntu 14.04)
• Windows
• Mac

Setup

If you already installed Docker and Docker Compose on your host machine you can download repository with code for running the cluster:

git clone https://github.com/moosefs/moosefs-docker-cluster

cd moosefs-docker-cluster

Start cluster

Build default cluster and run in the background:

docker-compose build
docker-compose up -d

Cluster is running!

Let us check if instances are running with:

docker ps

You should have 1 master, 5 chunkservers and 1 client running. Expected output should be similar to:

CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS                     NAMES
2fe620447b37        dockermoosefscluster_client         "/home/start-clien..."   5 minutes ago       Up 5 minutes                                  client
1951d867c078        dockermoosefscluster_chunkserver5   "/home/start-chunk..."   5 minutes ago       Up 5 minutes        9419-9420/tcp, 9422/tcp   chunkserver5
204c115cd8ad        dockermoosefscluster_chunkserver2   "/home/start-chunk..."   5 minutes ago       Up 5 minutes        9419-9420/tcp, 9422/tcp   chunkserver2
48343721de4f        dockermoosefscluster_chunkserver4   "/home/start-chunk..."   5 minutes ago       Up 5 minutes        9419-9420/tcp, 9422/tcp   chunkserver4
30ca217fa862        dockermoosefscluster_master         "/home/start.sh -d"      5 minutes ago       Up 5 minutes        9420-9425/tcp             master
28e2a64d0fb9        dockermoosefscluster_chunkserver1   "/home/start-chunk..."   5 minutes ago       Up 5 minutes        9419-9420/tcp, 9422/tcp   chunkserver1
c83c70580795        dockermoosefscluster_chunkserver3   "/home/start-chunk..."   5 minutes ago       Up 5 minutes        9419-9420/tcp, 9422/tcp   chunkserver3

You can attach to the client node (press “Enter” twice):

docker container attach mfsclient  #press the "Enter" key twice

To detach from container use the escape sequence Ctrl + p, Ctrl + q.
Don’t use Ctrl + d or exit command because it will kill container.

Now MooseFS filesystem is mounted as /mnt/mfs. If everything is ok you should see our welcome message with:

cd /mnt/mfs

cat welcome_to_moosefs.txt

CGI Monitor

The CGI Monitor is available here:

http://172.20.0.2:9425

Just open the link in your favorite browser and you should see status page similar to this one:

To stop all (all means ALL, not just MooseFS’s) your Docker containers:

docker stop $(docker ps -aq)

Your MooseFS Docker cluster is persistent. It means all files you created in the /mnt/mfs folder will remain unchanged even after restarting containers. However if you remove containers you will lose all your data.

Cluster configurations

Default configuration (file docker-compose.yml):

• master with management GUI http://172.20.0.2:9425
• chunkserver1 172.20.0.11 with 10 GiB of storage
• chunkserver2 172.20.0.12 with 10 GiB of storage
• chunkserver3 172.20.0.13 with 10 GiB of storage
• chunkserver4 172.20.0.14 with 10 GiB of storage
• chunkserver5 172.20.0.15 with 10 GiB of storage
• client 172.168.20.0.5

File docker-compose-chunkserver-client.yml

• master with CGI http://172.20.0.2:9425
• chunkserver1 172.20.0.11 with 10 GiB of storage and client (mount point: /mnt/mfs)
• chunkserver2 172.20.0.12 with 10 GiB of storage and client (mount point: /mnt/mfs)
• chunkserver3 172.20.0.13 with 10 GiB of storage and client (mount point: /mnt/mfs)
• chunkserver4 172.20.0.14 with 10 GiB of storage and client (mount point: /mnt/mfs)
• chunkserver5 172.20.0.15 with 10 GiB of storage and client (mount point: /mnt/mfs)

If you want to change storage size modify the chunkserver start script moosefs-chunkserver/start-chunkserver.sh

Default configuration is stored in docker-compose.yml

Other configurations

If you want to use other than default compose yml file (docker-compose.yml) use following commands:

docker-compose -f docker-compose-chunkserver-client.yml build
docker-compose -f docker-compose-chunkserver-client.yml up -d

When using docker-compose-chunkserver-client.yml you will have 5 chunkserver/client machines so you can attach to all of them mfschunkserverclient1, ..., mfschunkserverclient5.

Conclusion

We have successfully started MooseFS cluster with master, 5 chunkservers and client. We are able to really quick make simple tests using one physical machine. In next posts I will try to show you how to deploy MooseFS on multiple physical machines using docker.

Helpful links

• MooseFS https://moosefs.com
• MooseFS on GitHub https://github.com/moosefs/moosefs
• Docker https://docker.com/
• Docker Hub https://hub.docker.com/

If you like this work show me your support!
Follow me on:

• YouTube: https://www.youtube.com/karolmajek
• Twitter: https://twitter.com/karol_majek
• Medium: https://medium.com/@karol_majek

Team IMM approach to European Robotics League Emergency 2017

European Robotics League is funded by the European Union’s Horizon 2020 Program. It is continuation of three earlier projects:

• RoCKIn@Home (now: ERL Service Robots) tournament focuses on the domain of service robotics for home application.
• RoCKIn@Work (now: ERL Industrial Robots) tournament focuses on the domain of industrial robotics in the Factory of the Future and also deals with modern automation issues.
• euRathlon (now: ERL Emergency Robots) is a civilian, outdoor robotics competition, with a focus on realistic, multi-domain emergency response scenarios.

ERL Emergency 2017 is a continuation of Eurathlon 2015 robotic competition. This year 8 teams with robot from three domains cooperated to fulfill the objectives in scenarios.

Starting in this competition we assumed we will present automatic 3D mapping system, autonomous navigation and artificial intelligence for object detection.

About

Team IMM — Robdos — IIS Piombino CVP

There were 8 three domain teams created for the competition. Our team was

Team 4

• IMM, Poland (Land)
• Robdos, Spain (Sea)
• IIS Piombino CVP, Italy (Air)

Institute of Mathematical Machines (IMM) is a research and development institute. IMM participates in robotic competitions since 2013. IMM was established in 1957 (the word „computer” did not exist in the Polish language at that time, therefore the old expression „mathematical machines” is still used in its name).

Robdos Team, an underwater robotics association, came up in 2014 from our concern as a group of students specialized in different fields about putting into practice all the skills and knowledge gathered during university period. Marine, Industrial and Computer Engineering students from the Technical University of Madrid (Universidad Politécnica de Madrid), work hard on the development of an autonomous underwater platform.

IIS Piombino CVP is team from Technical Institute — local high school in Piombino

Scenarios

In all scenarios robots start in the same time and have the same amount of time finish all the tasks. During the mission robot operators should communicate with each other and exchange information. Ideally there should be also robot-robot communication.

Land + Sea

The goal of the scenario was to find the leaking pipe on the land by the ground robot, recognize the number on that pipe and go to the machine room. There was only one unobstructed entrance marked by green marker. The entrance and the machine room should be both identified automatically. In the machine room robot should identify and close the correct valve autonomously.

In the same time the underwater robot should start its mission and find the gate — a pair of acoustic and optical buoys. Robot should provide images of the gate and pass through it without touching it. The next task is to detect the plume buoys in real time, recognize numbers and provide images. Underwater robot should also identify the leak and exchange this information with the ground robot.

Both, ground and underwater, robots should provide maps of land pipes area, indoor map and 3D reconstruction of manipulation console respectively. In the end both (correct!) valves, underwater and in the building, should be closed simultaneously.

Sea + Air

In this scenario underwater robot should identify and pass through the gate like in the first scenario. Additionally robot should provide 2D acoustic or optical map of the debris. Robot localizes the missing worker underwater within a radius of 5 meters, then gives the dimensions and geometrical shape of the closest object to the worker. It should also provide 3D reconstruction of the worker and surface within a radius of 2 meters from the worker position.

The aerial robot starts its mission in the land area going through waypoints looking for a missing worker. Robot should detect leaking pipe, identify damages, build a map and find the missing worker. After identifying exact position of the worker drone deploys the first-aid kit within radius 2 m from the worker outside the building.

Robots start in the same time and are reporting underwater missing worker and land leaking pipe. Aerial robot should fly and take pictures of the place where worker is and underwater robot receives and decodes the message with the correct land leaking pipe sent by the aerial robot.

Land + Air

In the third scenario land robot is looking for OPIs and two missing workers one inside and the other outside the building. The ground robot should deliver the first-aid kit, dropped by the drone, to missing worker inside of the building.

The drone builds a map, finds the worker, and drop the first-aid kit. After that it drops another kit near the ground robot.

Grand Challenge: Land + Sea + Air

In the Grand Challenge all domains are doing the task simultaneously. It combines all previous tasks in a one large search and rescue mission. Operators of robots from all three domains are cooperating to achieve the goal.

3D Mapping

Our ground robot participated Enrich 2017, the robotic hackathon in Nuclear Power Plant in Austria in June 2017. Robot provided the 3D map of the reactor room online, without any data copy outside the internal robot computer. The resulting 3d is shown below:

And here are presented 3D maps from ERL Emergency 2017:

Object detection

Objects of Potential Interest (OPIs) are described in Rule Book for ERL Emergency 2017. Here are presented OPIs that our deep net was trained to detect. Below you can see sample images:

We used videos from Eurathlon 2015 to prepare annotated dataset. We generated 256 images with 7 classes:

• unblocked (unblocked entrance; green marker)
• blocked (blocked entrance; blue marker)
• pipes
• worker (missing worker)
• ericard
• damage (red marker)
• valve

250 images is very small dataset to train even a pretrained network, so we decided to augment training samples using random translate, brightness and mirroring. We increased image count to 12750 images. Original images were captured by few different cameras (normal and spherical).

To achieve online detection, which was crucial during the trials, we used Single Shot Multibox Detector (SSD) with MobileNet. We started training process from model trained on the COCO dataset. To train the deep net we used Tensorflow Object Detection API available on GitHub:

models/research/object_detection at master · tensorflow/models

And finally let’s see the AI in action! But wait, no one said we will want to use Pointgrey’s Ladybug 5 spherical camera for detection! To address the issue we are using 5 distorted images for detection separately. Yes, we do not expect anything above the robot!. Yes it is 5 times slower than detection from one camera, but since we have GPU, we can still process everything online!

And the video from the Grand Challenge:

Awards

We received three awards:

• 2nd Prize in Grand Challenge: IIS Piombino CVP, Italy (air) + Robdos, Spain (sea) + IMM, Poland (land)
• 1st Prize in Land and Air: IMM, Poland (land) + IIS Piombino CVP, Italy (air)
• Best Autonomy Award: IMM, Poland (land) for the best autonomy of land robots (autonomous navigation and automatic object detection).

Watch our video from the competition!

Links

• ERL Emergency 2017 https://eu-robotics.net/robotics_league/erl-emergency/about/index.html
• Referees’ special awards https://www.shadowrobot.com/blog/award-winners-of-erl-emergency-robots-2017-announced/
• Award winners of ERL Emergency Robots 2017 announcement https://www.eu-robotics.net/robotics_league/news/press/award-winners-of-erl-emergency-robots-2017.html
• Rule Book for ERL Emergency 2017 https://www.eu-robotics.net/robotics_league/upload/documents-2017/ERL_ER_Rule_book_v1_4.pdf
• Eurathlon Competition http://www.eurathlon.eu/

Heise.de:

• ERL Emergency: Kooperierende Roboter üben Katastropheneinsatz im AKW
  https://www.heise.de/newsticker/meldung/ERL-Emergency-Kooperierende-Roboter-ueben-Katastropheneinsatz-im-AKW-3834473.html
• Notfall-Robotik: Flug- und Tauchroboter suchen nach Vermissten — und finden sie auch manchmal https://www.heise.de/newsticker/meldung/Notfall-Robotik-Flug-und-Tauchroboter-suchen-nach-Vermissten-und-finden-sie-auch-manchmal-3837978.html
• Notfall-Robotik: Polnische Roboterteams beeindrucken mit Improvisationstalent https://www.heise.de/newsticker/meldung/Notfall-Robotik-Polnische-Roboterteams-beeindrucken-mit-Improvisationstalent-3839434.html
• Notfall-Robotik: Punktgenauer Abwurf eines medizinischen Hilfspakets und andere Erfolge
  https://www.heise.de/newsticker/meldung/Notfall-Robotik-Punktgenauer-Abwurf-eines-medizinischen-Hilfspakets-und-andere-Erfolge-3836770.html
• Roboter im Notfalleinsatz: Roboter stolpern durch den Sand
  https://www.heise.de/newsticker/meldung/Roboter-im-Notfalleinsatz-Roboter-stolpern-durch-den-Sand-3835059.html
• Notfall-Robotik: Bodenroboter bewähren sich an den Ventilen https://www.heise.de/newsticker/meldung/Notfall-Robotik-Bodenroboter-bewaehren-sich-an-den-Ventilen-3835940.html
• Roboter im Katastropheneinsatz: Rettung zu Wasser, zu Lande und in der Luft
  https://www.heise.de/newsticker/meldung/Roboter-im-Katastropheneinsatz-Rettung-zu-Wasser-zu-Lande-und-in-der-Luft-3839278.html

Team member links:

• IMM http://imm.org.pl/en/research-and-development/modelling-and-simulation/mobilemappingsystem/
• Robdos Team http://www.robdosteam.com/en/competicion/
• IIS Piombino CVP https://www.poloisiscvp.gov.it/

If you like this work show me your support!
Follow me on:

• YouTube: https://www.youtube.com/karolmajek
• Twitter: https://twitter.com/karol_majek
• Medium: https://medium.com/@karol_majek

Semantic segmentation is one of projects in 3rd term of Udacity’s Self-Driving Car Nanodegree program. The goal is to train deep neural network to identify road pixels using part of the KITTI dataset.

This solution uses VGG16 with 3 skip layers. The size of the input image is 576 x 160. Results for 4k video are generated by resizing prediction to 4k. The results are not perfect because of two factors:

• really small training dataset <400 images
• training shorter than <120k samples (with 16 samples per batch)

Data augmentation

For this result I was using only these 3 technics:

• flipping every image
• translating +/- 100 in x axis and +/- 30 in y axis
• brightness +/- 150 (in 0–255 range)

As future work I consider:

• adding noise
• adding random shadows
• changing contrast
• changing white balance
• adding objects (cars, …)

Example batch:

Source code on Github

Here is the link to source code of my solution.

karolmajek/CarND-Semantic-Segmentation

Result on 4k video from my 4k dashcam dataset:

More datasets

There are many datasets available, here is a good list:

Yet Another Computer Vision Index To Datasets (YACVID)

KITTI dataset

Most popular one; used in this work.

The KITTI Vision Benchmark Suite

Cityscapes dataset

Cityscapes Dataset - Semantic Understanding of Urban Street Scenes

Mapillary Vistas

• 25,000 Images
• 100 Categories
• 60 Instance-wise Categories
• 6 Continents
• Variety of Weather, Season, Time of Day, Camera, and Viewpoint

Mapillary - Street-level imagery, powered by collaboration and computer vision

ADE20K Dataset

ADE20K dataset

PASCAL Context Dataset

PASCAL-Context Dataset

COCO-Stuff 10K dataset v1.1

nightrome/cocostuff

2D-3D-S dataset

Large Scale Parsing

I am currently working with Mapillary Vistas Dataset so if you want to see some results follow me on medium or subscribe to my youtube channel.

Karol Majek

Inspired by this work Dat Tran, I prepared my own dataset and trained improved Pix2Pix net to generate Polish youtuber Krzysztof Gonciarz creating show “Zapytaj Beczkę”.

Face2face — A Pix2Pix demo that mimics the facial expression of the German chancellor

But first let’s give it a try:

In my first approach I just trained the original net from face2face-demo. It worked! But it was only 256 x 256 resolution which was not enough for me, so I decided to increase resolution to 1024 x 1024.

High resolution mod

To increase resolution one needs to add layers to encoder and decoder, there is no simpler way to do it. So I added few layers and tuned parameters to fit my 8GB memory on GTX980M in my laptop.

If you need to modify input/output size just add or remove some layers in encoder and decoder:

Encoder before (ngf = 64)

# encoder_2: [batch, 128, 128, ngf] => [batch, 64, 64, ngf * 2]
# encoder_3: [batch, 64, 64, ngf * 2] => [batch, 32, 32, ngf * 4]
# encoder_4: [batch, 32, 32, ngf * 4] => [batch, 16, 16, ngf * 8]
# encoder_5: [batch, 16, 16, ngf * 8] => [batch, 8, 8, ngf * 8]
# encoder_6: [batch, 8, 8, ngf * 8] => [batch, 4, 4, ngf * 8]
# encoder_7: [batch, 4, 4, ngf * 8] => [batch, 2, 2, ngf * 8]
# encoder_8: [batch, 2, 2, ngf * 8] => [batch, 1, 1, ngf * 8]

Encoder after (ngf = 64)

# encoder_2: [batch, 512, 512, ngf] => [batch, 256, 256, ngf * 2]
# encoder_3: [batch, 256, 256, ngf * 2] => [batch, 128, 128, ngf * 4]
# encoder_4: [batch, 128, 128, ngf * 4] => [batch, 64, 64, ngf * 8]
# encoder_5: [batch, 64, 64, ngf * 8] => [batch, 32, 32, ngf * 8]
# encoder_6: [batch, 32, 32, ngf * 8] => [batch, 16, 16, ngf * 16]
# encoder_7: [batch, 16, 16, ngf * 16] => [batch, 8, 8, ngf * 16]
# encoder_8: [batch, 8, 8, ngf * 16] => [batch, 4, 4, ngf * 16]
# encoder_9: [batch, 4, 4, ngf * 16] => [batch, 2, 2, ngf * 16]
# encoder_10: [batch, 2, 2, ngf * 16] => [batch, 1, 1, ngf * 32]

Training data

I used 4 videos as input for training:

• Zapytaj Beczkę #145 https://youtu.be/_aEnuh9tnKg
• Zapytaj Beczkę #143 https://youtu.be/yI8fJtcNkRA
• Zapytaj Beczkę #140 https://youtu.be/gii3w0t1KIs
• Zapytaj Beczkę #139 https://youtu.be/X3asNGDdUpw

I used frames with one face detected from every 10th frame of these videos and manually removed some bad examples (e.g. other people).

Results

Finally after 3 days of training (GTX980M) I get these results:

Source code

You can find source code on my GitHub:

karolmajek/face2face-demo

Conclusion

The original model (with Angela Merkel) was working good only in one position and distance from camera. Here I have similar problem — face was very close to the camera in all training videos. If you are going to train net to generate faces — remember to prepare very good dataset. The increase of input/output resolution was really good idea!

From my experience it was super easy to train this net and get first outputs. Totally different than with YOLO/SSD networks, where you need images and annotations in specified format. I will write when I successfully train YOLO or SSD!

Any ideas what to do next? Stay tuned for more deep learning posts!

Please check Dat Tran stories:

Dat Tran - Medium

If you like this work show me your support!
Follow me on:

• Youtube: https://www.youtube.com/karolmajek
• Twitter: https://twitter.com/karol_majek
• Medium: https://medium.com/@karol_majek