<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex P on Medium]]></title>
        <description><![CDATA[Stories by Alex P on Medium]]></description>
        <link>https://medium.com/@alexppppp?source=rss-4f1e99fc4a77------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Rm-moBIGAUW6yiSExbgFyQ.jpeg</url>
            <title>Stories by Alex P on Medium</title>
            <link>https://medium.com/@alexppppp?source=rss-4f1e99fc4a77------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:16:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexppppp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Train an Ensemble of Convolutional Neural Networks for Image Classification]]></title>
            <link>https://medium.com/@alexppppp/how-to-train-an-ensemble-of-convolutional-neural-networks-for-image-classification-8fc69b087d3?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fc69b087d3</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[convolutional-neural-net]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Wed, 09 Feb 2022 12:33:50 GMT</pubDate>
            <atom:updated>2022-02-12T12:59:36.641Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tutorial on how to create an ensemble of DenseNet161, ResNet152 and VGG19 for classification of TinyImageNet</h4><p>There are many different convolutional neural network (CNN) models for image classification (VGG, ResNet, DenseNet, MobileNet, etc.). All of them provide different accuracy.</p><p>An ensemble of several CNN models can significantly improve accuracy of our predictions, comparing to accuracy of any single model included into the ensemble.</p><p>If we have three different models for image classification, and they provide, for example, 81%, 83% and 85% accuracies respectively, then an ensemble of these three models can provide, for example, 87% accuracy, which is a pretty good improvement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEyEGnC7N3nvbtpYwhwitg.png" /></figure><p>In this tutorial we will use <strong>PyTorch</strong> to train three image classification models (<strong>DenseNet161</strong>, <strong>ResNet152</strong> and <strong>VGG19</strong>) on the <strong>TinyImageNet dataset</strong>. Then we will unite them in an ensemble.</p><p>TinyImageNet consists of 200 classes, train part contains 100,000 images, validation part contains 10,000 images, and test part contains 10,000 images. All images are of size 64×64.</p><h3>1. Imports</h3><p>Create a new notebook in Jupyter Notebook. First, we need to import the necessary modules and check GPU availability:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ed5eef336ee0736d309880b3e1d869c/href">https://medium.com/media/7ed5eef336ee0736d309880b3e1d869c/href</a></iframe><p>Output:</p><pre>CUDA is available. Working on GPU</pre><h3>2. Downloading TinyImageNet dataset</h3><p>Download and unzip the dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/802766b0ec911a48157ce57f5180089e/href">https://medium.com/media/802766b0ec911a48157ce57f5180089e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a5739f05a29f8daa180086113795295/href">https://medium.com/media/2a5739f05a29f8daa180086113795295/href</a></iframe><h3>3. Images and labels</h3><p>The dataset contains folders <strong>val/</strong>, <strong>train/</strong> and <strong>test/</strong>. For each folder we will create lists with paths to files (images) and lists with labels:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b49a71988add66730c011a5e20fcd265/href">https://medium.com/media/b49a71988add66730c011a5e20fcd265/href</a></iframe><p>Take a look at the output to better understand the structure of created lists:</p><pre>The first five files from the list of train images: [&#39;tiny-imagenet-200/train/n03970156/images/n03970156_391.JPEG&#39;, &#39;tiny-imagenet-200/train/n03970156/images/n03970156_35.JPEG&#39;, &#39;tiny-imagenet-200/train/n03970156/images/n03970156_405.JPEG&#39;, &#39;tiny-imagenet-200/train/n03970156/images/n03970156_92.JPEG&#39;, &#39;tiny-imagenet-200/train/n03970156/images/n03970156_162.JPEG&#39;]<br><br>The first five labels from the list of train labels: [&#39;n03970156&#39;, &#39;n03970156&#39;, &#39;n03970156&#39;, &#39;n03970156&#39;, &#39;n03970156&#39;]<br><br>The first five files from the list of validation images: [&#39;tiny-imagenet-200/val/images/val_9055.JPEG&#39;, &#39;tiny-imagenet-200/val/images/val_682.JPEG&#39;, &#39;tiny-imagenet-200/val/images/val_2456.JPEG&#39;, &#39;tiny-imagenet-200/val/images/val_7116.JPEG&#39;, &#39;tiny-imagenet-200/val/images/val_2825.JPEG&#39;]<br><br>The first five labels from the list of validation labels: [&#39;n03584254&#39;, &#39;n04008634&#39;, &#39;n02206856&#39;, &#39;n04532670&#39;, &#39;n01770393&#39;]<br><br>The first five files from the list of test images: [&#39;tiny-imagenet-200/test/images/test_0.JPEG&#39;, &#39;tiny-imagenet-200/test/images/test_1.JPEG&#39;, &#39;tiny-imagenet-200/test/images/test_10.JPEG&#39;, &#39;tiny-imagenet-200/test/images/test_100.JPEG&#39;, &#39;tiny-imagenet-200/test/images/test_1000.JPEG&#39;]</pre><p>You can see that the names of classes, or labels, are not the commonly used words like <em>‘table’</em>, <em>‘</em>bicycle<em>’</em>, <em>‘ariplane’</em>, etc. Instead, classes are named with codes, like <em>‘</em>n04379243<em>’</em>, <em>‘</em>n02834778<em>’</em>, <em>‘</em>n02691156<em>’</em>, etc. In total, there are 200 unique labels in the TinyImageNet dataset.</p><p>Besides creating the above mentioned lists, we have initialized an encoder for labels . Labels need to be encoded from string to integer because neural network doesn’t understand strings, it understands numbers only. So, instead of label <em>‘n01443537’</em> we will use number 0, instead of label <em>‘n01629819’</em> - number 1, instead of label <em>‘n01641577’</em> - number 2, and so on…</p><p>For example, this line of code encoder_labels.transform([‘n02206856’, ‘n04532670’, ‘n01770393’]) will return array([ 38, 170, 7]). And this line of code encoder_labels.inverse_transform([38, 170, 7]) will return array([‘n02206856’, ‘n04532670’, ‘n01770393’]). This is how encoder works.</p><h3>4. Dataset class</h3><p>The dataset should inherit from the standard torch.utils.data.Dataset class, and __getitem__ should return image (tensor) and target (integer number).</p><p>Let’s define dataset class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebc22ce60095c41172fcab3c052c9879/href">https://medium.com/media/ebc22ce60095c41172fcab3c052c9879/href</a></iframe><p>We have also defined two complex transforms transforms_train and transforms_val, which will be used for models training and validation.</p><p>Complex transform transforms_val for validation phase includes three simple transforms: resizing image to 224x224, converting it to tensor and normalizing it. Essentially, transforms_val implements the official guidance from PyTorch <a href="https://pytorch.org/vision/stable/models.html">website</a>:</p><blockquote>All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]</blockquote><p>Complex transform transforms_train includes the same simple transforms which are used in transforms_val, but it also includes random horizontal flip and random erasing, which will be applied during the training phase. Thus, we will essentially “create new images”, which will slightly differ from original ones, but still perfectly suitable for training our model. In other words, we will artificially expand our dataset.</p><h3>5. Visualizing random items from dataset</h3><p>Here we will initialize three datasets (for training, validation and testing), then we will look at an example of several items from the train dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/038ef162568ce57703696665b3cbfc83/href">https://medium.com/media/038ef162568ce57703696665b3cbfc83/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3_EtT_Jw1FzygRZpcFH1w.png" /></figure><p>You can see that some images have cropped parts of different size in different places, this is how random erasing transform works.</p><h3>6. Functions for training models</h3><p>Here we will define two functions: training() will do all the essential steps to train a model, and visualize_training_results() will show us charts with metrics for each epoch after the training is done.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d22185927c137ac253c3beb0782b2913/href">https://medium.com/media/d22185927c137ac253c3beb0782b2913/href</a></iframe><p>In function training() we check loss after each epoch. If validation loss during some epoch reaches a new minimum, then we update the best model weights and save them with torch.save() function.</p><h3>7. Training single models</h3><p>Let’s create dataloaders and train each model for 10 epochs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51149bcdfc55a8314b4b714d26f7173d/href">https://medium.com/media/51149bcdfc55a8314b4b714d26f7173d/href</a></iframe><p>Next, we will train the models and show the results:</p><h4>7.1. Training DenseNet161</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8dee3019832ef87e49ed5d04db97b59f/href">https://medium.com/media/8dee3019832ef87e49ed5d04db97b59f/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*kQwjVOE0P69YFMt8Lipm-w.png" /></figure><pre>Training results:<br>	Min val loss 0.0195 was achieved during iteration #13<br>	Val accuracy during min val loss is 0.6895</pre><h4>7.2. Training ResNet152</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b3d13dc60a6fdbf02b1d813a95a6fb2/href">https://medium.com/media/4b3d13dc60a6fdbf02b1d813a95a6fb2/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*z-Sl2yNbP5aSbVabkGkh0A.png" /></figure><pre>Training results:<br>	Min val loss 0.0198 was achieved during iteration #15<br>	Val accuracy during min val loss is 0.6810</pre><h4>7.3. Training VGG19</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1afd1c7a81f8b625ad8d818a3d5e552b/href">https://medium.com/media/1afd1c7a81f8b625ad8d818a3d5e552b/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*kNxU0Nrld2W_GG9kicx8-g.png" /></figure><pre>Training results:<br>	Min val loss 0.0270 was achieved during epoch #15<br>	Val accuracy during min val loss is 0.5753</pre><h4>7.4. Training summary</h4><p>We see that the valuation accuracies are <strong>68.95%</strong>, <strong>68.10%</strong>, <strong>57.53%</strong> for <strong>DenseNet161</strong>, <strong>ResNet152</strong> and <strong>VGG19</strong> respectively. Let’s try to achieve a better accuracy by uniting these models into an ensemble.</p><h3>8. Training ensemble of models</h3><p>Here is the most important part of this tutorial, we will define and initialize an ensemble model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a8bcd6fad195ff6afaebd1f27a1a600/href">https://medium.com/media/2a8bcd6fad195ff6afaebd1f27a1a600/href</a></iframe><p>Let’s train our ensemble model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/654e46e535bc07bb8df096a3ef73cdde/href">https://medium.com/media/654e46e535bc07bb8df096a3ef73cdde/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*7mElvWDflQEhHDV1Plk9bw.png" /></figure><pre>Training results:<br>	Min val loss 0.0176 was achieved during iteration #18<br>	Val accuracy during min val loss is 0.7113</pre><p>Valuation accuracy of ensemble model is <strong>71.13%</strong>, which is higher than valuation accuracies of independent models (<strong>68.95%</strong>, <strong>68.10% </strong>and <strong>57.53%</strong>).</p><h3>9. Classifying test dataset</h3><p>This part is optional, as the purpose of this tutorial to show how to create an ensemble of neural networks and show that it provides a better results than a single model.</p><p>However, here is the block of code, which classifies the test dataset and saves results into csv-file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/590c65de55918867122680dbb2756b51/href">https://medium.com/media/590c65de55918867122680dbb2756b51/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*siEpnDzr3mgFlYJpTgbr-Q.png" /></figure><p><a href="https://github.com/alexppppp/tinyimagenet-classification-ensemble">Here</a> is a GitHub repository and notebook with all the steps described above.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fc69b087d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create Synthetic Dataset for Computer Vision (Keypoint Detection)]]></title>
            <link>https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/78ba481cdafd</guid>
            <category><![CDATA[keypoint-detection]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[synthetic-datasets]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Tue, 25 Jan 2022 07:28:00 GMT</pubDate>
            <atom:updated>2022-02-11T17:55:59.211Z</atom:updated>
            <content:encoded><![CDATA[<h4>A simple and quick way to generate a large dataset with help of Python, OpenCV, Numpy and Albumentations</h4><p>Training a keypoint detection model, like <a href="https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da">Keypoint RCNN</a>, requires a <strong>dataset</strong> which contains <strong>images with objects of interest</strong> and <strong>annotations </strong>(text files with coordinates of objects’ keypoints and bounding boxes).</p><p>For example, on the image below you can see keypoints and bounding boxes visualized. Each object (glue tube) has two keypoints (head and tail).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqBkKLX-jLh3JTzOjWVYdg.jpeg" /><figcaption>Example of image with keypoints and bounding boxes</figcaption></figure><p>The more images dataset contains, the better the model will train, because it will see more examples during training process. Dataset which contains 200+ images is ok. Dataset with 1000+ images is much better. The exellent dataset contains 5000+ images.</p><p>Note that dataset should not just contain a lot of images, but all the images should be as varied as possible. Objects of interest on those images should be mixed with other objects, presented in different environments, with different backgrounds, in different positions, etc.</p><p>One way to create dataset is to create it <strong>manually</strong>. It means that we take a lot of photos, like the photo above, and annotate them by hand. Such approach is the best, because all photos are real, but it takes a lot of time to create such dataset.</p><p>The other way is to create a synthetic dataset <strong>automatically</strong>. With this approach, cropped photos of objects of interest are randomly scaled, rotated and added to background with python script. Annotations are created with the same script. Under this approach we create images, which are not real photos, but objects on these images look 100% realistic.</p><p>Example of an image from synthetic dataset is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qLRzfZ8lC-GigRyWmQ5luw.png" /><figcaption>Example of an image from synthetic dataset with keypoints and bounding boxes</figcaption></figure><p><strong>Automated process</strong> allows us to spend <strong>much less time</strong> to create a dataset comparing to <strong>manual process</strong>. For example, generation of 1000 synthetic images and annotations can take less than an hour. It’s much faster than taking 1000 various photos and annotate them by hand.</p><p>Below, I’ll describe all the steps of creating synthetic dataset for keypoint detection.</p><p>I’ll show how to create synthetic dataset with <strong>glue tubes</strong> for <a href="https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da">training Keypoint RCNN</a>. To do this, we need the following data:</p><ul><li>cropped photos and masks of the object of interest (<em>glue tubes</em>) in various positions + coordinates of keypoints (the 1st keypoint is <em>head</em>, the 2nd keypoint is <em>tail</em>) on every photo;</li><li>background images (just different photos from the internet);</li><li>cropped photos and masks of different objects (cars, chairs, guitars, etc.) which will be used as background noise to make background more complex.</li></ul><p>I’ve took <strong>14 photos of glue tube</strong> and created masks for them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MBqSbVzEyij3JSfKnigeg.png" /><figcaption>Photos and masks of objects of interest</figcaption></figure><p>I’ve also created <strong>14 json-files</strong> with coordinates of keypoints (head and tail) for each photo. Besides coordinates, json-files contain visibility of keypoints. That is, each glue tube has 2 keypoints, head and tail, which are described in [x, y, visibility] format. All keypoints are visible (i.e. visibility=1) in this dataset.</p><p>I’ve collected <strong>60 images</strong> which will be used as <strong>background</strong>. Take a look at some of these images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L93yPQXRW0byrQZvCTjWzw.png" /><figcaption>Images which will be used as a background</figcaption></figure><p>I’ve also collected <strong>107 images of different objects</strong> which will be used as <strong>background noise</strong>. These can be literally any objects which are not glue tubes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ydw0q3yS7DTOgFzOrjIt1A.png" /><figcaption>Objects which will be used as background noise</figcaption></figure><p>Download zip-archive with the described above data from <a href="https://drive.google.com/file/d/1imJda1jsgiYnLQqFYwkp00YSnBaRlT99/view?usp=sharing">here</a>.</p><p>Update. You can also check <a href="https://www.youtube.com/watch?v=kUKd8hNCwuI">this</a> short video tutorial to see how to create mask of the object with help of Photoshop.</p><p>Here is how to use the downloaded data to create a synthetic scene:</p><ul><li>First, we will randomly choose a background image from folder <strong>bg/</strong> and resize it to, for example, 1920x1080.</li><li>Second, we will randomly pick a background noise object from folder <strong>bg_noise/</strong>. Then we’ll ramdomly resize, rotate and add it to the background image.</li><li>We will repeat the second step several times.</li><li>Third, we will randomly pick an object of interest from folder <strong>images/</strong>. Then we’ll ramdomly resize, rotate and add it to the background image on the top of background noise objects from the previous step.</li><li>We will repeat the third step several times.</li></ul><p>Obtained random composition of objects is a synthetic scene.</p><p>Synthetic dataset consists of many synthetic scenes.</p><p>Let’s create a script which creates synthetic dataset.</p><h3>1. Imports</h3><p>Create a new notebook in Jupyter Notebook.</p><p>First, we need to import the necessary modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afab9791110e6b8be407e5661fc18884/href">https://medium.com/media/afab9791110e6b8be407e5661fc18884/href</a></iframe><h3>2. Paths to files</h3><p>Unzip downloaded archive with data to folder <strong>data/</strong> and create lists with paths to images, masks and keypoints:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/33277f87f6c0f305010cc5c3f5dc0ea0/href">https://medium.com/media/33277f87f6c0f305010cc5c3f5dc0ea0/href</a></iframe><p>Take a look at the output to better understand the structure of the created lists:</p><pre>The first five files from the sorted list of object images: [&#39;data\images\1.jpg&#39;, &#39;data\images\10.jpg&#39;, &#39;data\images\11.jpg&#39;, &#39;data\images\12.jpg&#39;, &#39;data\images\13.jpg&#39;]<br><br>The first five files from the sorted list of object masks: [&#39;data\masks\1.png&#39;, &#39;data\masks\10.png&#39;, &#39;data\masks\11.png&#39;, &#39;data\masks\12.png&#39;, &#39;data\masks\13.png&#39;]<br><br>The first five files from the sorted list of object keypoints: [&#39;data\keypoints\1.json&#39;, &#39;data\keypoints\10.json&#39;, &#39;data\keypoints\11.json&#39;, &#39;data\keypoints\12.json&#39;, &#39;data\keypoints\13.json&#39;]<br><br>The first five files from the sorted list of background images: [&#39;data\bg\bg_1.jpg&#39;, &#39;data\bg\bg_10.jpg&#39;, &#39;data\bg\bg_11.jpg&#39;, &#39;data\bg\bg_12.jpg&#39;, &#39;data\bg\bg_13.jpg&#39;]<br><br>The first five files from the sorted list of background noise images: [&#39;data\bg_noise\images\1.png&#39;, &#39;data\bg_noise\images\10.jpg&#39;, &#39;data\bg_noise\images\100.jpg&#39;, &#39;data\bg_noise\images\101.png&#39;, &#39;data\bg_noise\images\102.png&#39;]<br><br>The first five files from the sorted list of background noise masks: [&#39;data\bg_noise\masks\1.png&#39;, &#39;data\bg_noise\masks\10.png&#39;, &#39;data\bg_noise\masks\100.png&#39;, &#39;data\bg_noise\masks\101.png&#39;, &#39;data\bg_noise\masks\102.png&#39;]</pre><p>Later, our script will have a block of code which will randomly pick an object image from these lists, resize it, add augmentations to it, and add it to background.</p><h3>3. Images, masks and keypoints</h3><p>There are several types of masks:</p><ul><li><strong>Original mask </strong>is the mask where object area is filled with black color (0,0,0), background area is filled with white color (255,255,255).</li><li><strong>Boolean mask </strong>is the mask where object area is filled with True, background area is filled with False.</li><li><strong>Binary mask </strong>is the mask where object area is filled with 1, background area is filled with 0.</li></ul><p>For the purpose of this script, we will convert original masks to binary masks.</p><p>Here we define a function get_img_and_mask() which returns image of the object in an OpenCV format, and mask of the object in a binary format:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b8915a52b96d44286659bc24df0ef32/href">https://medium.com/media/4b8915a52b96d44286659bc24df0ef32/href</a></iframe><p>Also we will define a function visualize_single_img_with_keypoints() which visualizes an image with object of interest and draws object’s keypoints on this image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a9815bb99576343ec9d718beefd882f/href">https://medium.com/media/5a9815bb99576343ec9d718beefd882f/href</a></iframe><h4>3.1. Object of interest (with keypoints)</h4><p>Let’s look how get_img_and_mask() function works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcb042c561451d94681f990a78ee966a/href">https://medium.com/media/dcb042c561451d94681f990a78ee966a/href</a></iframe><p>Output:</p><pre>Image file: data\images\5.jpg<br>Mask file: data\masks\5.png<br><br>Shape of the image of the object: (735, 1111, 3)<br>Shape of the binary mask: (735, 1111)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*Q2nMAOv4nJQWC5qWbQklvA.png" /></figure><p>Note that the width of the image is 1111, and height is 735. Also, the image has 3 channels. That’s why the shape of the image is (735, 1111, 3). Binary mask has the same width and height, but only one channel. That’s why shape of the binary mask is (735, 1111).</p><p>Let’s visualize keypoints on this image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5875c438dec1ec3746683fd3fe011017/href">https://medium.com/media/5875c438dec1ec3746683fd3fe011017/href</a></iframe><p>Output:</p><pre>Keypoints: [[979, 103, 1], [132, 594, 1]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*R8F7Pq2oZ5Y3ENZrIjJN9g.png" /></figure><p>On the image above the first keypoint, head, has x coordinate 979, y coordinate 103, and visibility=1. The second keypoint, tail, has x coordinate 132, y coordinate 594, and visibility=1.</p><h4>3.2. Background noise object (without keypoints)</h4><p>Let’s get image and mask of a random noise object with get_img_and_mask() function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d964d4ed309be6bcfb661484fa7cc63a/href">https://medium.com/media/d964d4ed309be6bcfb661484fa7cc63a/href</a></iframe><pre>Image file: data\bg_noise\images\18.jpg<br>Mask file: data\bg_noise\masks\18.png<br><br>Shape of the image of the object: (1280, 1073, 3)<br>Shape of the binary mask: (1280, 1073)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*ZFXVK7zYlQvY8H9fY8im0Q.png" /></figure><h3>4. Resizing background images</h3><p>Images, which will be used as background, have different sizes. For example: 2114x1398, 3456x5184, 1920x1440, 3264x4080, etc. Some of them are horizontal (width &gt; height), others are vertical (height &gt; width).</p><p>But we may want all images in the synthetic dataset to have fixed dimensions: 1920x1080 for horizontal ones, and 1080x1920 for vertical ones. To achieve this, we will resize background images with help of resize_img() function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/922b185d31eed6f4b42fd16f08316918/href">https://medium.com/media/922b185d31eed6f4b42fd16f08316918/href</a></iframe><p>Let’s look how this function works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d37306b9d868daa5a29dd5c55cd42e7/href">https://medium.com/media/1d37306b9d868daa5a29dd5c55cd42e7/href</a></iframe><p>Output:</p><pre>Shape of the original background image: (3068, 2454, 3)<br>Shape of the resized background image (desired_max=1920, desired_min=None): (1920, 1535, 3)<br>Shape of the resized background image (desired_max=1920, desired_min=1080): (1920, 1080, 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*xmQyrI7Fx0Yjr4lEKFQUqw.png" /></figure><p>You can see that the function finds out which side of the image (width or height) is the longest, and resizes image to desired_max size along the longest side. If desired_min is not set, then the shortest side of the image is resized proportionally, otherwise image is resized to desired_min size along the shortest side.</p><h3>5. Resizing and transforming objects of interest (with keypoints)</h3><p>Function resize_transform_obj() for resizing and transforming objects is similar to the function for resizing background images, but has some additions.</p><p>Function resize_transform_obj() resizes image of the object and binary mask of the object. Also, transforms from <a href="https://albumentations.ai/">albumentations</a> library can be passed to the function as argument. Coordinates of keypoints are affected during resizing and transforms as well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdf447f35c12f75bb36eb73159023dc8/href">https://medium.com/media/cdf447f35c12f75bb36eb73159023dc8/href</a></iframe><p>In the code above a complex transform transforms_obj is defined. This transform rotates image and changes contrast &amp; brightness in a narrow range. It will be used to transform objects of interest.</p><p>It’s possible to add more options to transform. Read <a href="https://albumentations.ai/docs/">albumentations documentation</a> to find out how to do it.</p><p>Let’s look how function resize_transform_obj() works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f937038e51afbddd65ccf46c6e51f66f/href">https://medium.com/media/f937038e51afbddd65ccf46c6e51f66f/href</a></iframe><p>Output:</p><pre>Shape of the image of the transformed object: (983, 650, 3)<br>Shape of the transformed binary mask: (983, 650, 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*eDRbx75srSRieGCasgi8Lg.png" /></figure><p>You’ve seen these image and mask earlier, but now the shape of the image is (983, 650, 3) instead of (735, 1111, 3). Also, it the image is rotated and the brightness here is higher than before. This is how transforms work.</p><p>Let’s visualize keypoints on the transformed image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b3991747bd7a6562ba92954ac2a8af7/href">https://medium.com/media/3b3991747bd7a6562ba92954ac2a8af7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*Zq34Awb0jlVNeSigYNWiew.png" /></figure><h3>6. Resizing and transforming background noise objects (without keypoints)</h3><p>Here we will define function resize_transform_bg_obj() which transforms noise objects. The difference between function resize_transform_obj() and a new function is that the new function doesn’t transform keypoints, because background noise objects don’t have keypoints.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50df3c45ba4fcbe5bf12cf3101554cb7/href">https://medium.com/media/50df3c45ba4fcbe5bf12cf3101554cb7/href</a></iframe><p>In the code above a complex transform transforms_bg_obj is defined. This transform rotates image, flips image, adds blur, changes colors, contrast &amp; brightness in a wide range. It will be used to transform background noise objects.</p><p>Let’s look how function resize_transform_bg_obj() works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96be836f9e0d0da397d020e2071d7571/href">https://medium.com/media/96be836f9e0d0da397d020e2071d7571/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*cIBw7_jbDMaWnObwZ2lFPw.png" /></figure><p>You’ve seen these image and mask earlier, but now the image is rotated and the brightness here is higher than before.</p><h3>7. Adding objects of interest (with keypoints) to background</h3><h4>7.1. Adding one object</h4><p>Here we’ll define function add_obj() which adds object of interest to background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cc99e4391f8730e811296ee8ae6e26e/href">https://medium.com/media/2cc99e4391f8730e811296ee8ae6e26e/href</a></iframe><p>Function add_obj() returns image composition (background + added objects), mask composition (composition of masks for added objects) and keypoints composition (list of keypoints of added objects).</p><p>Also we will define a function visualize_composition_with_keypoints() which visualizes a composition of objects of interest and draws objects’ keypoints:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9acf811e9042a3ec0830195f795b75e/href">https://medium.com/media/b9acf811e9042a3ec0830195f795b75e/href</a></iframe><p>Let’s add a glue tube to background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47afab5d5fe8981c11fa656ff385a68c/href">https://medium.com/media/47afab5d5fe8981c11fa656ff385a68c/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*vHFXAqp5ex-o7XkTUCxRgQ.png" /></figure><p>The initial composition here is background image img_bg.</p><p>Array mask_comp = np.zeros((h,w), dtype=np.uint8) is a mask of the initial composition. Since the initial composition is just a background image without any objects on it, its mask contains only zeros.</p><p>With adding glue tube to img_bg, it’s mask is added to mask_comp by overlapping initial values with 1 in those pixels, which correspond to the added glue tube on the image composition. We’ve defined the number 1 for the mask of added glue tube by passing parameter idx=1 to function add_obj().</p><p>The right picture above is related to compostion mask: numbers 0 are marked in dark purple, numbers 1 are marked in yellow.</p><p>Let’s look at the keypoints:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa1a1b85af400a02b21ad02182d0f2dc/href">https://medium.com/media/fa1a1b85af400a02b21ad02182d0f2dc/href</a></iframe><p>Output:</p><pre>Keypoints: [[[1079, 203, 1], [232, 694, 1]]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2WsHcaty6OHwjpefrL2EQ.png" /></figure><p>Let’s add transformed glue tube:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/463c5f25d1e42597f7b8984210b990d8/href">https://medium.com/media/463c5f25d1e42597f7b8984210b990d8/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*7GpSL6cmt9xY1iiVOrF8zQ.png" /></figure><p>This time the initial composition img_comp already contains one glue tube, so the mask of the initial composition mask_comp contains numbers 0 and 1.</p><p>With adding one more glue tube to the composition, this tube’s mask is added to mask_comp by overlapping initial values with 2 in those pixels, which correspond to the added tube on the image composition. This time we defined the number 2 for the mask of added glue tube by passing parameter idx=2 to function add_obj().</p><p>The right picture above is related to compostion mask: numbers 0 are marked in dark purple, numbers 1 are marked in the mix of blue and green, numbers 2 are marked in yellow.</p><p>Let’s look at the keypoints:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08576c0a044ed1a1af0cd52889dd962d/href">https://medium.com/media/08576c0a044ed1a1af0cd52889dd962d/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_vV0fcpqcmNEv56kN9yazg.png" /></figure><h3>8. Adding noise objects (without keypoints) to background</h3><h4>8.1. Adding one object</h4><p>Here we’ll define function add_bg_obj() which adds noise object to background. To understand how this function works in details, I recommend you to read the article <a href="https://medium.com/@alexppppp/adding-objects-to-image-in-python-133f165b9a01">“Adding Objects to Image in Python”</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6bc728765040b46159eb1911f2d8ee1/href">https://medium.com/media/d6bc728765040b46159eb1911f2d8ee1/href</a></iframe><p>Function add_bg_obj() returns image composition (background + added objects) and mask composition (composition of masks for added objects).</p><p>Let’s see how it works by adding chair to background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/062183190341be805683b5e0e0c5fa07/href">https://medium.com/media/062183190341be805683b5e0e0c5fa07/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*DwmhPPXe4QJCHaspTkzemw.png" /></figure><p>Let’s add transformed chair:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/513d45588095ba630692ce113cd4c3f0/href">https://medium.com/media/513d45588095ba630692ce113cd4c3f0/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*V45B-5EQM9fB0mYqTsNOSQ.png" /></figure><h4>8.2. Adding many objects</h4><p>We want to have dataset with backgrounds as varied as possible. Various backgrounds are good for the training process of keypoint detection neural network. But we have only 60 background images, which is not much if we are going to create dataset of 1000 or more images.</p><p>To make backgrounds more varied, we will randomly add noise objects.</p><p>Noise objects will be added with function create_bg_with_noise():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3151cda865e4c3d83b2e4b54432711c4/href">https://medium.com/media/3151cda865e4c3d83b2e4b54432711c4/href</a></iframe><p>Here is the description of parameters:</p><ul><li>files_bg_imgs is a list with paths to background images;</li><li>files_bg_noise_imgs is a list with paths to noise objects images;</li><li>files_bg_noise_masks is a list with paths to noise objects masks;</li><li>bg_max and bg_min are the target sizes of the longest and the shortest sides of background image;</li><li>max_objs_to_add is a maximum number of noise objects to be added to background;</li><li>longest_bg_noise_min and longest_bg_noise_max are the minimum and maximum sizes of the longest side of noise objects. longest_bg_noise_max should be less than bg_min, longest_bg_noise_min should be at least 30.</li><li>blank_bg should be True if we want background to be a white color instead of random image.</li></ul><p>Let’s look how this function works if we set white background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffb2c8f7a453694f940574914691e891/href">https://medium.com/media/ffb2c8f7a453694f940574914691e891/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*DU-uaPLhGNS3Obvjb1Sw8w.png" /><figcaption>Noise objects added to background randomly. Background here is a white color.</figcaption></figure><p>This time we will chose a random image for background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3941ef9bb8d120c153fd90417d75bc4c/href">https://medium.com/media/3941ef9bb8d120c153fd90417d75bc4c/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*YFF_--b4phI8bdkY3RlBcA.png" /><figcaption>Noise objects added to background randomly. Background here is a random image.</figcaption></figure><p>Note that after each time function create_bg_with_noise() is called, we get a new conposition of noise objects, because they are chosen and placed above the background randomly.</p><h3>9. Controlling degree of overlapping</h3><p>A newly added object of interest can partially overlap previously added objects of interest on the composition. Sometimes it can overlap a significant part of another object, like 60% or 70% of its area, or even completely overlap it. But we don’t want this to happen.</p><p>We might want to control the degree of overlapping, and make it less then 20% or 30%. Or we might want our objects of interest don’t overlap at all.</p><p>Let’s define function check_overlapping() which checks if any of previously added objects is overlapped more than overlap_degree threshold:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/404195eddece9d19561317d28933dedf/href">https://medium.com/media/404195eddece9d19561317d28933dedf/href</a></iframe><p>After adding a new object to the composition, this function compares areas of not overlapped parts of the previously added objects with original areas of previously added objects. If any of the previously added objects is overlapped by more than overlap_degree, then the function returns False. If all of the previously added objects are overlapped by not more than overlap_degree or not overlapped at all, then the function returns True.</p><p>Parameter mask_comp is a composition of masks after a new object is added.</p><p>Parameter obj_areas is a list of objects’ original areas in order of their addition as if they were not overlapped. This list shouldn’t include a newly added object while passing it to check_areas() function.</p><h3>10. Creating synthetic composition</h3><p>Here we will define function create_composition() which creates synthetic composition of objects:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1827fe24ef56ba633e577c6de36a1bb4/href">https://medium.com/media/1827fe24ef56ba633e577c6de36a1bb4/href</a></iframe><p>Here is the description of parameters:</p><ul><li>img_comp_bg is a background to which objects of interest will be added.</li><li>max_objs is the maximum number of objects to be added.</li><li>longest_min and longest_max are the minimum and maximum sizes of the longest side of objects of interest.</li><li>overlap_degree is the threshold which defines if randomly added object of interest overlaps any of the earlier added objects of interest more than threshold defined by overlap_degree. If at least one of the objects is overlapped too much, then the function goes back to previous composition and adds the object again.</li><li>max_attempts_per_obj is the number of attempts the function will try to add object without overlapping other objects by more than threshold defined by overlap_degree.</li></ul><p>This function returns:</p><ul><li>img_comp: image with added objects of interest. In our case objects of interst are glue tubes.</li><li>mask_comp: composition of masks of added objects. Background pixels have value 0, pixels of the first added object have value 1, pixels of the second added object have value 2, etc.</li><li>keypoints_comp: list of keypoints of added objects.</li></ul><p>It’s possible to obtain bounding box for each object from its mask. We will define function create_bboxes_from_mask_comp() which returns coordinates of bounding boxes for objects of interest in form of a list:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/477112ec9e34405c013e1ce82a4504ac/href">https://medium.com/media/477112ec9e34405c013e1ce82a4504ac/href</a></iframe><p>Now we are ready to generate a synthetic composition and visualize it (here we set overlap_degree=0, so glue tubes don’t overlap at all):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf053e163657ac0da2d3c8658124eec6/href">https://medium.com/media/cf053e163657ac0da2d3c8658124eec6/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*2fT1CEfTdi4QIbKawtzcCQ.png" /><figcaption>Synthetic composition of objects (glue tubes are added randomly above background)</figcaption></figure><p>Let’s visualize keypoints and bounding boxes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2b783584f080cf9c6f1bc3e36795244/href">https://medium.com/media/e2b783584f080cf9c6f1bc3e36795244/href</a></iframe><p>Output:</p><pre>Keypoints: [[[473, 652, 1], [266, 72, 1]], [[1564, 716, 1], [1571, 283, 1]], [[862, 423, 1], [1164, 745, 1]]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybr-RF_FPfhjJc9febei3Q.png" /><figcaption>Keypoints and bounding boxes</figcaption></figure><h3>11. Creating and saving synthetic dataset</h3><p>We wrote a python script which creates synthetic images and masks. Now we will write script which creates annotations for images.</p><p>First, create folders <strong>dataset/train/images/</strong>, <strong>dataset/train/annotations/</strong>, <strong>dataset/valid/images/</strong>, <strong>dataset/valid/annotations/</strong> where function generate_dataset() will save images and annotations.</p><p>Here is the function which creates a dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62a4cac7672abf3e14837955328095bf/href">https://medium.com/media/62a4cac7672abf3e14837955328095bf/href</a></iframe><p>Let’s create dataset of 1000 training images and 200 validating images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77e4876dec0db5d2be194da45896dd0b/href">https://medium.com/media/77e4876dec0db5d2be194da45896dd0b/href</a></iframe><p>Output:</p><pre>100%|██████████████████████████████████████████████████████████████████████████████| 1000/1000 [17:13&lt;00:00,  1.03s/it]</pre><pre>Generation of 1000 synthetic images is completed. It took 1033 seconds, or 1.0 seconds per image<br>Images are stored in &#39;dataset\train\images&#39;<br>Annotations are stored in &#39;dataset\train\annotations&#39;</pre><pre>100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [03:19&lt;00:00,  1.00it/s]</pre><pre>Generation of 200 synthetic images is completed. It took 199 seconds, or 1.0 seconds per image<br>Images are stored in &#39;dataset\valid\images&#39;<br>Annotations are stored in &#39;dataset\valid\annotations&#39;</pre><p>Now we have a synthetic dataset and ready to train an object detetion model!</p><p>In my case, it took about 20 minutes to generate dataset of 1200 images on a PC with processor Intel Core i7–10700K and 32GB of RAM. One synthetic image was generated in about 1 second.</p><p>I’ve also took 23 photos with different compositions of glue tubes, and annotated them by hand. We can use these real photos to test the quality of the object detection model after training.</p><p><a href="https://drive.google.com/uc?id=1zqX70OFt27FWCiaT9ZIT0E96nr8R1km3&amp;export=download">Here</a> you can download the whole dataset of 1000 synthetic training images, 200 synthetic validating images, and 23 real testing images.</p><p><a href="https://github.com/alexppppp/synthetic-dataset-keypoint-detection">Here</a> is a GitHub repository and notebook with all the steps described above.</p><h3>12. Example from synthetic dataset</h3><p>Let’s look at a random image from generated synthetic dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJb8ekmB2IYJ9pleTNZEEg.jpeg" /><figcaption>Example of an image from synthetic dataset</figcaption></figure><p>Here is how the related json-file with annotations looks like:</p><p>{“bboxes”: [[1257, 475, 1901, 603], [199, 154, 637, 463]], “keypoints”: [[[1318, 530, 1], [1874, 547, 1]], [[249, 413, 1], [597, 198, 1]]]}</p><p>We see that there are two glue tubes on the image. File with annotations for this image contains coordinates for two bounding boxes. Also, there are coordinates of two keypoints for each glue tube (thus, there are four keypoints in total on this image).</p><p>Let’s visualize annotations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANfw5jGHgknH0wA9QAuMlg.png" /><figcaption>Example of an image from synthetic dataset with visualized keypoints and bounding boxes</figcaption></figure><p>Keypoints and annotations are in right places. It means that we have a visual confirmation that our script works properly.</p><h3>13. Training and testing keypoint detection model</h3><p>I’ve used the generated synthetic dataset to train Keypoint RCNN model.</p><p>Next, I used the trained model to detect keypoints of glue tubes on the video streaming from camera in real time. Here is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*tHL_PIHhirfXylP6KGRB9A.gif" /></figure><h3>Another in-depth articles about computer vision:</h3><p><a href="https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da"><strong>How to Train a Custom Keypoint Detection Model with PyTorch</strong></a><strong>:</strong> tutorial on how to fine-tune Keypoint RCNN.</p><p><a href="https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-object-detection-fd8ab2fa5249"><strong>How to Create Synthetic Dataset for Object Detection</strong></a><strong>:</strong> a simple and quick way to generate a large dataset with help of Python, OpenCV, Numpy and Albumentations.</p><p><a href="https://medium.com/@alexppppp/how-to-train-an-ensemble-of-convolutional-neural-networks-for-image-classification-8fc69b087d3"><strong>How to Train an Ensemble of Convolutional Neural Networks for Image Classification</strong></a>: tutorial on how to create an ensemble of DenseNet161, ResNet152 and VGG19 for classification of TinyImageNet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78ba481cdafd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create Synthetic Dataset for Computer Vision (Object Detection)]]></title>
            <link>https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-object-detection-fd8ab2fa5249?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd8ab2fa5249</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[yolo]]></category>
            <category><![CDATA[synthetic-datasets]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[yolov5]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Mon, 10 Jan 2022 18:50:54 GMT</pubDate>
            <atom:updated>2022-02-11T17:56:35.740Z</atom:updated>
            <content:encoded><![CDATA[<h4>A simple and quick way to generate a large dataset with help of Python, OpenCV, Numpy and Albumentations</h4><p>Training an object detection model, like <a href="https://github.com/ultralytics/yolov5">YOLOv5</a>, requires a <strong>dataset</strong> which contains <strong>images with objects of interest</strong> and <strong>annotations </strong>(text files with coordinates of objects’ bounding boxes).</p><p>For example, on the image below you can see bounding boxes visualized. Each bounding box indicates an object of interest related to a specific class: battery (red), lightbulb (green), padlock (blue).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inBsy24qIr8LBXeEvgRZAw.png" /><figcaption>Example of image with bounding boxes</figcaption></figure><p>The more images dataset contains, the better the model will train, because it will see more examples during training process. Dataset which contains 200+ images is ok. Dataset with 1000+ images is much better. The exellent dataset contains 5000+ images.</p><p>Note that dataset should not just contain a lot of images, but all the images should be as varied as possible. Objects of interest on those images should be mixed with other objects, presented in different environments, with different backgrounds, in different positions, etc.</p><p>One way to create dataset is to create it <strong>manually</strong>. It means that we take a lot of photos, like the photo above, and annotate them by hand. Such approach is the best, because all photos are real, but it takes a lot of time to create such dataset.</p><p>The other way is to create a synthetic dataset <strong>automatically</strong>. With this approach, cropped photos of objects of interest are randomly scaled, rotated and added to background with python script. Annotations are created with the same script. Under this approach we create images which are not exactly real photos, but objects and background on these images are 100% realistic .</p><p>Example of an image from synthetic dataset is below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDqRUXzUJbCxzGPRDXEmGA.png" /><figcaption>Example of an image from synthetic dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cnB_6iFDLj8iaRB87fJ6aQ.png" /><figcaption>Example of an image from synthetic dataset: initial background photo (top left), background photo with automatically added objects (botom left), bounding boxes of automatically added objects (bottom right), masks of automatically added objects (top right).</figcaption></figure><p><strong>Automated process</strong> allows us to spend <strong>much less time</strong> to create a dataset comparing to <strong>manual process</strong>. For example, generation of 1000 synthetic images and annotations can take less than an hour. It’s much faster than taking 1000 various photos and annotate them by hand.</p><p>Below, I’ll describe all steps of creating synthetic dataset for object detection.</p><p>I’ll show how to create synthetic dataset with <strong>batteries</strong>, <strong>lightbulbs</strong> and <strong>padlocks</strong> for training YOLOv5. To do this, we need the following data:</p><ul><li>cropped photos and masks of the objects of interest (batteries, lightbulbs, padlocks) in various positions;</li><li>background images (just different photos from the internet);</li><li>cropped photos and masks of different objects (cars, chairs, guitars, etc.) which will be used as background noise to make background more complex.</li></ul><p>I’ve took <strong>26 photos of battery</strong>, <strong>23 photos of lightbulb</strong>,<strong> 21 photos of padlock</strong>, and created masks for these objects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MTJMXzrTIisfb2AnHT-WfA.png" /><figcaption>Photos and masks of objects of interest</figcaption></figure><p>I’ve collected <strong>30 images</strong> which will be used as <strong>background</strong>. Take a look at some of these images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L93yPQXRW0byrQZvCTjWzw.png" /><figcaption>Images which will be used as a background</figcaption></figure><p>I’ve also collected <strong>107 images of different objects</strong> which will be used as <strong>background noise</strong>. These can be literally any objects which are not battery, lightbulb or padlock:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ydw0q3yS7DTOgFzOrjIt1A.png" /><figcaption>Objects which will be used as background noise</figcaption></figure><p>Download zip-archive with the described above data from <a href="https://drive.google.com/uc?id=1Bzf_UeMyN9QWrfc_gSPv3JFnyP8gUsX1&amp;export=download">here</a>.</p><p><strong>Update.</strong> You can also check <a href="https://www.youtube.com/watch?v=kUKd8hNCwuI">this</a> short video tutorial to see how to create mask of the object with help of Photoshop.</p><p>Here is how to use the downloaded data to create a synthetic scene:</p><ul><li>First, we will randomly choose a background image from folder <strong>bg/</strong> and resize it to, for example, 1920x1080.</li><li>Second, we will randomly pick a background noise object from folder <strong>bg_noise/</strong>. Then we’ll ramdomly resize, rotate and add it to the background image.</li><li>We will repeat the second step several times.</li><li>Third, we will randomly pick an object of interest from folders <strong>battery/</strong>, <strong>lightbulb/</strong>, <strong>padlock/</strong>. Then we’ll ramdomly resize, rotate and add it to the background image on the top of background noise objects from the previous step.</li><li>We will repeat the third step several times.</li></ul><p>Obtained random composition of objects is a synthetic scene.</p><p>Synthetic dataset consists of many synthetic scenes.</p><p>Let’s create a script which creates synthetic dataset.</p><h3>1. Imports</h3><p>Create a new notebook in Jupyter Notebook.</p><p>First, we need to import the necessary modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b42de812cdb6824dbfe4d573e49d2fd/href">https://medium.com/media/3b42de812cdb6824dbfe4d573e49d2fd/href</a></iframe><h3>2. Paths to files</h3><p>Unzip downloaded archive with data to folder <strong>data/</strong> and create lists with paths to images and masks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b02ad521ee054d475df5d6a6499a2643/href">https://medium.com/media/b02ad521ee054d475df5d6a6499a2643/href</a></iframe><p>Take a look at the output to better understand the structure of created lists:</p><pre>The first five files from the sorted list of battery images: [&#39;data\battery\images\1.png&#39;, &#39;data\battery\images\10.png&#39;, &#39;data\battery\images\11.png&#39;, &#39;data\battery\images\12.png&#39;, &#39;data\battery\images\13.png&#39;]<br><br>The first five files from the sorted list of battery masks: [&#39;data\battery\masks\1.png&#39;, &#39;data\battery\masks\10.png&#39;, &#39;data\battery\masks\11.png&#39;, &#39;data\battery\masks\12.png&#39;, &#39;data\battery\masks\13.png&#39;]<br><br>The first five files from the sorted list of background images: [&#39;data\bg\bg_1.jpg&#39;, &#39;data\bg\bg_10.jpg&#39;, &#39;data\bg\bg_11.jpg&#39;, &#39;data\bg\bg_12.jpg&#39;, &#39;data\bg\bg_13.jpg&#39;]<br><br>The first five files from the sorted list of background noise images: [&#39;data\bg_noise\images\1.png&#39;, &#39;data\bg_noise\images\10.jpg&#39;, &#39;data\bg_noise\images\100.png&#39;, &#39;data\bg_noise\images\101.jpg&#39;, &#39;data\bg_noise\images\102.png&#39;]<br><br>The first five files from the sorted list of background noise masks: [&#39;data\bg_noise\masks\1.png&#39;, &#39;data\bg_noise\masks\10.png&#39;, &#39;data\bg_noise\masks\100.png&#39;, &#39;data\bg_noise\masks\101.png&#39;, &#39;data\bg_noise\masks\102.png&#39;]</pre><p>Later, our script will have a block of code which will randomly pick an object image from these lists, resize it, add augmentations to it, and add it to background.</p><p>Also, to set the lower and upper bounds for resizing object images, we set ‘longest_min’: 150, ‘longest_max’: 800 for each object of interest in dictionary obj_dict. It means that the longest side of the image will be not less than 150px, but not more than 800px. You can set other numbers, but I would suggest the lower bound to be at least 30, and the upper bound should be less then both height and width of the background.</p><h3>3. Images and masks</h3><p>There are several types of masks:</p><ul><li><strong>Original mask </strong>is the mask where object area is filled with black color (0,0,0), background area is filled with white color (255,255,255).</li><li><strong>Boolean mask </strong>is the mask where object area is filled with True, background area is filled with False.</li><li><strong>Binary mask </strong>is the mask where object area is filled with 1, background area is filled with 0.</li></ul><p>For the purpose of this script, we will convert original masks to binary masks.</p><p>Here we define a function which returns image of the object in an OpenCV format, and mask of the object in a binary format:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7f835f5e973f06cbaf3f176397f191b/href">https://medium.com/media/e7f835f5e973f06cbaf3f176397f191b/href</a></iframe><p>Let’s look how this function works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca3500a953c58bd43d13bb73adb73424/href">https://medium.com/media/ca3500a953c58bd43d13bb73adb73424/href</a></iframe><p>Output:</p><pre>Image file: Data\Padlock\images\1.png<br>Mask file: Data\Padlock\masks\1.png</pre><pre>Shape of the image of the object: (962, 847, 3)<br>Shape of the binary mask: (962, 847)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*_smfN01n6fQJrGNJBwzAFw.png" /></figure><p>Note that the width of the image is 847, and height is 962. Also, the image has 3 channels. That’s why the shape of the image is (962, 847, 3). Binary mask has the same width and height, but only one channel. That’s why shape of the binary mask is (962, 847).</p><h3>4. Resizing background images</h3><p>Images, which will be used as background, have different sizes. For example: 2114x1398, 3456x5184, 1920x1440, 3264x4080, etc. Some of them are horizontal (width &gt; height), others are vertical (height &gt; width).</p><p>But we may want all images in the synthetic dataset to have fixed dimensions: 1920x1080 for horizontal ones, and 1080x1920 for vertical ones. To achieve this, we will resize background images with help of resize_img() function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e0571398dfdc5cb7bf8588a72d6840f/href">https://medium.com/media/9e0571398dfdc5cb7bf8588a72d6840f/href</a></iframe><p>Let’s look how this function works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/59c6a080718f570de888d2dec78a7c21/href">https://medium.com/media/59c6a080718f570de888d2dec78a7c21/href</a></iframe><p>Output:</p><pre>Shape of the original background image: (3068, 2454, 3)<br>Shape of the resized background image (desired_max=1920, desired_min=None): (1920, 1535, 3)<br>Shape of the resized background image (desired_max=1920, desired_min=1080): (1920, 1080, 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*xmQyrI7Fx0Yjr4lEKFQUqw.png" /></figure><p>You can see that the function finds out which side of the image (width or height) is the longest, and resizes image to desired_max size along the longest side. If desired_min is not set, then the shortest side of the image is resized proportionally, otherwise image is resized to desired_min size along the shortest side.</p><h3>5. Resizing and transforming objects</h3><p>Function resize_transform_obj() for resizing and transforming objects is similar to the function for resizing background images, but has some additions.</p><p>Function resize_transform_obj() resizes image of the object and binary mask of the object. Also, transforms from <a href="https://albumentations.ai/">albumentations</a> library can be passed to the function as argument.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48f12de23b7114b64af1ca45952a65ef/href">https://medium.com/media/48f12de23b7114b64af1ca45952a65ef/href</a></iframe><p>In the code above two complex transforms are defined:</p><ul><li>transforms_bg_obj rotates image, adds blur, changes colors, contrast &amp; brightness in a wide range. This aggressive transform will be used to transform background noise objects.</li><li>transforms_obj rotates image and changes contrast &amp; brightness in a narrow range. This negligible will be used to transform objects of interest.</li></ul><p>It’s possible to add more options to transforms. Read <a href="https://albumentations.ai/docs/">albumentations documentation</a> to find out how to do that.</p><p>Let’s look how function resize_transform_obj() works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcb62aee4c4f27ade68e17932e866b5e/href">https://medium.com/media/dcb62aee4c4f27ade68e17932e866b5e/href</a></iframe><p>Output:</p><pre>Shape of the image of the transformed object: (335, 381, 3)<br>Shape of the transformed binary mask: (335, 381)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*rOBLRCpQsMkYYeSD_wkyRA.png" /></figure><p>You’ve seen these image and mask earlier, but now the shape of the image is (335, 381, 3) instead of (962, 847, 3). Also, it the image is rotated and the brightness here is higher than before. This is how transforms work.</p><h3>6. Adding object to background</h3><p>Here we’ll define function add_obj() which adds object to background. To understand how this function works in details, I recommend you to read the article <a href="https://medium.com/@alexppppp/adding-objects-to-image-in-python-133f165b9a01">“Adding Objects to Image in Python”</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdb3984f3856ddc18eab9edf2c828447/href">https://medium.com/media/fdb3984f3856ddc18eab9edf2c828447/href</a></iframe><p>Function add_obj() returns image composition (background + added objects), mask composition (composition of masks for added objects), and mask of the last added object.</p><p>Let’s see how it works by adding padlock to background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f06046b1ffe0eea06fca38973321cd21/href">https://medium.com/media/f06046b1ffe0eea06fca38973321cd21/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*ssDFUNo-heUdkiJSXYyxvg.png" /></figure><p>The initial composition here is background image img_bg.</p><p>Array mask_comp = np.zeros((h,w), dtype=np.uint8) is a mask of the initial composition. Since the initial composition is just a background image without any objects on it, its mask contains only zeros.</p><p>With adding padlock to img_bg, it’s mask is added to mask_comp by overlapping initial values with 1 in those pixels, which correspond to the added padlock on the image composition. We’ve defined the number 1 for the mask of added padlock by passing parameter idx=1 to function add_obj().</p><p>The right picture above is related to compostion mask: numbers 0 are marked in dark purple, numbers 1 are marked in yellow.</p><p>Let’s add padlock one more time:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac958109c9b076a4b67effc1d74cbcf2/href">https://medium.com/media/ac958109c9b076a4b67effc1d74cbcf2/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*fwtL-tuoU4XBpaxs-IsfKQ.png" /></figure><p>This time the initial composition img_comp already contains one padlock, so the mask of the initial composition mask_comp contains numbers 0 and 1.</p><p>With adding one more padlock to the composition, this padlock’s mask is added to mask_comp by overlapping initial values with 2 in those pixels, which correspond to the added padlock on the image composition. This time we defined the number 2 for the mask of added padlock by passing parameter idx=2 to function add_obj().</p><p>The right picture above is related to compostion mask: numbers 0 are marked in dark purple, numbers 1 are marked in the mix of blue and green, numbers 2 are marked in yellow.</p><h3>7. Adding noise objects to background</h3><p>We want to have dataset with backgrounds as varied as possible. Various backgrounds are good for training process of object detection neural network. But we have only 30 background images, which is not much if we are going to create dataset of 1000 or more images.</p><p>To make backgrounds more varied, we will randomly add noise objects.</p><p>Noise objects will be added with function create_bg_with_noise():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc5eab730acce79bf2883ea45cb8c8fd/href">https://medium.com/media/cc5eab730acce79bf2883ea45cb8c8fd/href</a></iframe><p>Here is the description of parameters:</p><ul><li>files_bg_imgs is a list with paths to background images;</li><li>files_bg_noise_imgs is a list with paths to noise objects images;</li><li>files_bg_noise_masks is a list with paths to noise objects masks;</li><li>bg_max and bg_min are the target sizes of the longest and the shortest sides of background image;</li><li>max_objs_to_add is a maximum number of noise objects to be added to background;</li><li>longest_bg_noise_min and longest_bg_noise_max are the minimum and maximum sizes of the longest side of noise objects. longest_bg_noise_max should be less than bg_min, longest_bg_noise_min should be at least 30.</li><li>blank_bg should be True if we want background to be a white color instead of random image.</li></ul><p>Let’s look how this function works if we set white background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4df866a8df3441460829e1484374b39f/href">https://medium.com/media/4df866a8df3441460829e1484374b39f/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*1jY75uwgscaKt7sOYehyhg.png" /><figcaption>Noise objects added to background randomly. Background here is a white color.</figcaption></figure><p>This time we will chose a random image for background:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27504500fd111cadadbbee0a610b78bc/href">https://medium.com/media/27504500fd111cadadbbee0a610b78bc/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*GJFoBBgfoKOTmpvRoN950g.png" /><figcaption>Noise objects added to background randomly. Background here is a random image.</figcaption></figure><p>Note that after each time function create_bg_with_noise() is called, we get a new conposition of noise objects, because they are chosen and placed above the background randomly.</p><h3>8. Controlling degree of overlapping</h3><p>A newly added object of interest can partially overlap previously added objects of interest on the composition. Sometimes it can overlap a significant part of another object, like 60% or 70% of its area, or even completely overlap it. But we don’t want this to happen.</p><p>We might want to control the degree of overlapping, and make it less then 20% or 30%. Or we might want our objects of interest don’t overlap at all.</p><p>Let’s define function check_areas() which checks if any of previously added objects is overlapped more than overlap_degree threshold:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b925f2f0f9b030e475eee124fce892d/href">https://medium.com/media/1b925f2f0f9b030e475eee124fce892d/href</a></iframe><p>After adding a new object to the composition, this function compares areas of not overlapped parts of the previously added objects with original areas of previously added objects. If any of the previously added objects is overlapped by more than overlap_degree, then the function returns False. If all of the previously added objects are overlapped by not more than overlap_degree or not overlapped at all, then the function returns True.</p><p>Parameter mask_comp is a composition of masks after a new object is added.</p><p>Parameter obj_areas is a list of objects’ original areas in order of their addition as if they were not overlapped. This list shouldn’t include a newly added object while passing it to check_areas() function.</p><h3>9. Creating synthetic composition</h3><p>Here we will define function create_composition() which creates synthetic composition of objects:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89330c1dd76a20ead0b0523edb5b07ca/href">https://medium.com/media/89330c1dd76a20ead0b0523edb5b07ca/href</a></iframe><p>Here is the description of parameters:</p><ul><li>img_comp_bg is a background to which objects of interest will be added.</li><li>max_objs is the maximum number of objects to be added.</li><li>overlap_degree is the threshold which defines if a randomly added object of interest overlaps any of the earlier added objects of interest more than threshold defined by overlap_degree. If at least one of the objectsof interest is overlapped too much, then the function goes back to previous composition and adds the object again.</li><li>max_attempts_per_obj is the number of attempts the function will try to add object without overlapping other objects by more than threshold defined by overlap_degree.</li></ul><p>This function returns:</p><ul><li>img_comp: image with added objects of interest. In our case objects of interst are battery, lightbulb and padlock.</li><li>mask_comp: composition of masks of added objects. Background pixels have value 0, pixels of the first added object have value 1, pixels of the second added object have value 2, etc.</li><li>labels_comp: numerical representation of classes of added objects. For example, if objects were added in the following order [lightbulb, battery, padlock, padlock, lightbulb, padlock, battery], then array of labels would be [2, 1, 3, 3, 2, 3, 1]. This relation of classes and numbers is defined in obj_dict in the very begining of the script.</li><li>obj_areas: list of objects’ areas in order of their addition as if they were not overlapped.</li></ul><p>Let’s generate a synthetic composition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05e5a8b6ae907a8e698b816c9e82308f/href">https://medium.com/media/05e5a8b6ae907a8e698b816c9e82308f/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYXTlH3JMCbil1hibCT_qg.png" /><figcaption>Synthetic composition of objects (batteries, lampbulbs and padlocks are added randomly above background)</figcaption></figure><p>Here you can see batteries, lightbulbs and padlocks, but it’s not always easy to find them all quickly.</p><p>Let’s look at the mask of this synthetic composition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0484ca8d4dcefb9d3a1edc86a38f5178/href">https://medium.com/media/0484ca8d4dcefb9d3a1edc86a38f5178/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tOar7uvYfPPkmj25gCcXIg.png" /><figcaption>Composition of masks of added objects</figcaption></figure><p>If you look at the mask composition, you can easily find all the objects. Here you can see 2 batteries, 3 lampbulbs and 4 padlocks on the composition.</p><p>Let’s look at the array of labels:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/958c954bcb5651d4be21ede1347ffc14/href">https://medium.com/media/958c954bcb5651d4be21ede1347ffc14/href</a></iframe><p>Output:</p><pre>Labels (classes of the objects) on the composition in order of object&#39;s addition: [3, 1, 2, 2, 3, 2, 3, 1, 3]</pre><p>Here you can see that the first added object was a padlock (class 3), then a battery (class 1) was added, and so on…</p><p>Let’s also compare original areas of the objects (without overlapping) with areas on the composition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de997ef34c7a75e685dff8b7841e621c/href">https://medium.com/media/de997ef34c7a75e685dff8b7841e621c/href</a></iframe><p>Output:</p><pre>Degree of how much area of each object is overlapped:<br>0.8688065237500786<br>0.8778115434707346<br>1.0<br>1.0<br>1.0<br>1.0<br>1.0<br>1.0<br>1.0</pre><p>Here we see that the first added object is overlapped by 1 - 0.869 = 13.1%, and the second added object is overlapped by 1 - 0.878 = 12.2%. Both 13.1% and 12.2% are less than overlapping threshold of 0.2, which was passed to function create_composition() as parameter overlap_degree.</p><p>Also, we can see that the first added object is a padlock (the first element in labels_comp array has label 3), the second added object is a batery (second element in labels_comp array has label 1). If we look at the mask composition again, we can see that one padlock and one battery are overlapped by lampbulbs. It means that we have a visual confirmation that our script works properly.</p><p>Let’s also draw bounding boxes for each added object:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/820d66e8af5707e387602b0ae03fed64/href">https://medium.com/media/820d66e8af5707e387602b0ae03fed64/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0L8XLOvEldyKfIKhxthvA.png" /><figcaption>Bounding boxes</figcaption></figure><p>You can see that it’s possible to obtain bounding box for each object from its mask. On the image above each class has it’s own color (red for batteries, green for lightbulbs, blue for padlocks).</p><h3>10. Annotations in YOLO format</h3><p>We wrote a python script which creates synthetic images and masks. Now we will write script which creates annotations for images.</p><p>YOLO format requires annotations to be stored as txt-files. There should be a txt-file for each image, both of them should have the same name. Each txt-file consists of several lines; one line correspondents to one bounding box and consists of five numbers object_class x_center y_center width height.</p><p>The first number object_class is the number of object’s class. YOLO format requires object classes should start with 0.</p><p>The other four numbers are the coordinates of a bounding box in x_center y_center width height format. Coordinates must be presented in a normalized format (from 0 to 1). To get normalized coordinates, divide x_center and width by background image width, and y_center and height by background image height.</p><p>Here is the function which creates annotations for synthetic scene:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5c3f95e735b5d538b625d2d1720ada8/href">https://medium.com/media/a5c3f95e735b5d538b625d2d1720ada8/href</a></iframe><p>Function returns a list of annotations for each object presented on the mask_comp. Let’s look how it works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3fd6f0a6854a4c4140e3e6c843df4da/href">https://medium.com/media/d3fd6f0a6854a4c4140e3e6c843df4da/href</a></iframe><p>Output:</p><pre>2 0.66042 0.78472 0.18021 0.4287<br>0 0.28802 0.30139 0.18333 0.58056<br>1 0.84297 0.82593 0.2224 0.3463<br>1 0.74557 0.18194 0.1151 0.225<br>2 0.73385 0.55556 0.04792 0.17778<br>1 0.24844 0.08472 0.15937 0.16944<br>2 0.60547 0.40463 0.07656 0.25556<br>0 0.38333 0.79028 0.21875 0.28241<br>2 0.21589 0.70602 0.08802 0.31389</pre><p>Once again, it’s important to note that numeric classes of objects here are starting from 0, instead of 1. In array labels_comp 1 is related to battery, 2 is related to lightbulb and 3 is related to padlock. But, in annotations, classes of the objects should start with 0, it’s YOLO format requirement, so we decrease each number by one, which means that in annotations 0 is related to battery, 1 is related to lightbulb and 2 is related to padlock.</p><h3>11. Creating and saving synthetic dataset</h3><p>YOLOv5 requires training images and annotations to be stored in folders <strong>train/images/ </strong>and <strong>train/labels/</strong>. Validation part of the dataset should be stored in folders <strong>valid/images/</strong> and <strong>valid/labels/</strong>.</p><p>Here is the function which creates a dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d04be0d16d702291de0900c3ed940058/href">https://medium.com/media/d04be0d16d702291de0900c3ed940058/href</a></iframe><p>Now, create folders <strong>dataset/train/images/</strong>, <strong>dataset/train/labels/</strong>, <strong>dataset/valid/images/</strong>, <strong>dataset/valid/labels/</strong> where function generate_dataset() will save images and annotations.</p><p>Let’s create dataset of 1000 training images and 200 validating images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b20ccc3feebf84a8c86e315628a64d62/href">https://medium.com/media/b20ccc3feebf84a8c86e315628a64d62/href</a></iframe><p>Output:</p><pre>100%|████████████████████████████████████████████████████████████████████████████| 1000/1000 [1:04:37&lt;00:00,  3.88s/it]</pre><pre>Generation of 1000 synthetic images is completed. It took 3878 seconds, or 3.9 seconds per image<br>Images are stored in &#39;dataset\train\images&#39;<br>Annotations are stored in &#39;dataset\train\labels&#39;</pre><pre>100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [12:15&lt;00:00,  3.68s/it]</pre><pre>Generation of 200 synthetic images is completed. It took 735 seconds, or 3.7 seconds per image<br>Images are stored in &#39;dataset\valid\images&#39;<br>Annotations are stored in &#39;dataset\valid\labels&#39;</pre><p>Great! Now we have a synthetic dataset and ready to train an object detetion model!</p><p>In my case, it took about 1 hour 20 minutes to generate dataset of 1200 images on a laptop with processor Intel Core i7–6700HQ and 8GB of RAM while some other tasks were running. One synthetic image was generated in less than 4 seconds.</p><p>I’ve also took 43 photos with different compositions of batteries, lightbulbs &amp; padlocks, and annotated them by hand. We can use these real photos to test the quality of the object detection model after training.</p><p><a href="https://drive.google.com/uc?id=1usPEGZAXy-24NRHni4NDV8YKbMnB0qm2&amp;export=download">Here</a> you can download the whole dataset of 1000 synthetic training images, 200 synthetic validating images, and 43 real testing images.</p><p><a href="https://github.com/alexppppp/synthetic-dataset-object-detection">Here</a> is a GitHub repository and notebook with all the steps described above.</p><h3>12. Training and testing YOLOv5 model</h3><p>I’ve used the generated dataset to train YOLOv5x6 model in Google Colab. I’ve set the following hyperparameters for training: image size of 1280, 4 images per batch, 10 epochs.</p><p>After training the model, I’ve tested it on real photos. Results were pretty good (P is precision, R is recall, mAP is mean average precision):</p><pre>    Class  Images  Labels       P       R     mAP@.5    mAP@.5:.95<br>      all      43     354   0.976   0.944      0.956         0.883<br>  Battery      43     133   0.944    0.88      0.895         0.774<br>Lightbulb      43     110   0.985   0.991      0.995         0.949<br>  Padlock      43     111       1    0.96      0.978         0.926</pre><p>Let’s look at several test photos with detected objects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GgS-_ZTAAxOC_vgDBWjXvw.png" /></figure><p>I’ve intentionally made some photos with partially overlapped objects and some photos with objects in complex environments to make it harder for the model to identify the objects of interest. But the trained on synthetic dataset model recognized objects on these photos quite well.</p><h3>13. Importance of noise objects in synthetic scene</h3><p>I want you to pay attention to this test photo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ms2FJ52vONjrcdF126wXLA.jpeg" /></figure><p>You can see here that two erasers are identified as batteries. Probably, the model found some common features between erasers and batteries (shape + presense of text), and took erasers for batteries.</p><p>It’s possible to avoid such situations by adding eraser as noise object while generating synthetic dataset. So, during training process, the model could adjust it’s weights to not mistake erasers for batteries.</p><p>Moreover, the more different kinds of noise objects we add, the better model will train not to pay attention to them, it means that there will be less false positive detections.</p><p>That’s why it’s important to add noise objects to the background while generating synthetic scene.</p><p>Ok, now you know how to generate a synthetic dataset for object detection.</p><p>You can substitute battery, lightbulb and padlock with your objects of interest and generate dataset for your needs.</p><p>You can also change format of annotations if you need to create dataset not for YOLOv5, but for some other object detection model.</p><h3>Another in-depth articles about computer vision:</h3><p><a href="https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd"><strong>How to Create Synthetic Dataset for Keypoint Detection</strong></a><strong>:</strong> a simple and quick way to generate a large dataset with help of Python, OpenCV, Numpy and Albumentations.</p><p><a href="https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da"><strong>How to Train a Custom Keypoint Detection Model with PyTorch</strong></a><strong>:</strong> tutorial on how to fine-tune Keypoint RCNN.</p><p><a href="https://medium.com/@alexppppp/how-to-train-an-ensemble-of-convolutional-neural-networks-for-image-classification-8fc69b087d3"><strong>How to Train an Ensemble of Convolutional Neural Networks for Image Classification</strong></a>: tutorial on how to create an ensemble of DenseNet161, ResNet152 and VGG19 for classification of TinyImageNet.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd8ab2fa5249" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding Objects to Image in Python]]></title>
            <link>https://medium.com/@alexppppp/adding-objects-to-image-in-python-133f165b9a01?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/133f165b9a01</guid>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[opencv-python]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[numpy]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Sun, 12 Dec 2021 19:27:07 GMT</pubDate>
            <atom:updated>2022-02-03T11:17:14.595Z</atom:updated>
            <content:encoded><![CDATA[<h4>Guide on how to overlay a small image over a big image using Python, OpenCV and Numpy</h4><p>In this article I’ll show how to add object from one image to another image. To do this, we will need:</p><ul><li>background image, into which the object will be inserted;</li><li>image of the object;</li><li>image with mask of the object (mask has a black color, other space is white).</li></ul><p>In our case background is a photo of the sea, object is a cup of coffee. Here they are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AiSzDzvJjlGPOPbIBrXanQ.jpeg" /></figure><p>Download them: <a href="https://drive.google.com/file/d/1KJNq8QGw3XKU-80TyaYl0CqV-TKy-MZn">background.jpg</a>, <a href="https://drive.google.com/file/d/1zzhrhsSfE4z_bSBUi_JLnmmAsdDTjgVW">cup.png</a>, <a href="https://drive.google.com/file/d/1mBynzcd4r2nKo7ZHV-cjY3XJ8VLJy6IY">cup_mask.png</a>.</p><p><strong>Update.</strong> You can also check <a href="https://www.youtube.com/watch?v=kUKd8hNCwuI">this</a> short video tutorial to see how to create mask of the object with help of Photoshop.</p><p>Soon you’ll find out how to obtain the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tysrUFZimjWEYSv3GAVquw.png" /></figure><p>Let’s start!</p><h3>Basic concepts of digital images</h3><p>First, I want to remind you that images are stored as 3D arrays of integer numbers from 0 to 255.</p><p>The shape of these arrays is <strong>height</strong> x <strong>width </strong>x <strong>channels</strong>.</p><p>There are three channels: <strong>Red</strong>, <strong>Green</strong> and <strong>Blue</strong>. This is called <strong>RGB</strong> format.</p><p>Thus, every image consists of <strong>height </strong>x<strong> width</strong> pixels. Each pixel has three components. The numerical values of these components range from 0 to 255 and represent the brightness values of red, green and blue colors. The higher the number, the brighter the color.</p><p>For example:</p><ul><li>pixel [255, 0, 0] has red color, because there is zero brightness in green and blue colors, but red color has a maximum brightness;</li><li>pixel [255, 255, 255] has white color, which is a mix of red, green and blue colors with maximum brightness;</li><li>pixel [0, 0, 0] has a black color, because there is no brightness at all due to zero values of all colors.</li></ul><h3>1. Imports</h3><p>Now, create a new notebook in Jupyter Notebook. First, we need to import the necessary modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db5899fe5d7e9707f5e365bd37599c70/href">https://medium.com/media/db5899fe5d7e9707f5e365bd37599c70/href</a></iframe><h3>2. Reading and diaplaying images with OpenCV</h3><p>Let’s open images with help of cv2.imread() function and display them.</p><p><strong>Note!</strong> For some reason, OpenCV reads images in BGR format (Blue and Red colors are swapped). We need to convert BGR to RGB format with help of cv2.cvtColor() function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b22a4d1f998a4c5607f1705d4970a74/href">https://medium.com/media/2b22a4d1f998a4c5607f1705d4970a74/href</a></iframe><p>Output:</p><pre>Background shape: (1280, 1920, 3)<br>Image shape: (860, 1151, 3)<br>Mask shape: (860, 1151, 3)</pre><p>We see that background image has height=1280 and width=1920, object image has height=860 and width=1151.</p><p>Let’s look at the images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a97cab1eaa7a301ab66b39812d9ff7e/href">https://medium.com/media/3a97cab1eaa7a301ab66b39812d9ff7e/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*GqNv4-HbOy1_zEuUJu8UPw.png" /></figure><p><strong>Important!</strong> To better understand the further manipulations with images &amp; numpy arrays, you can read the article <a href="https://medium.com/@alexppppp/replacing-part-of-2d-array-with-another-2d-array-in-numpy-c83144576ddc">“Replacing Part of 2D Array with Another 2D Array in Numpy”</a>.</p><h3>3. Removing background from the image of the object</h3><p>Now we will define a function, which converts mask of the object to boolean array, or boolean mask.</p><p>On the original mask, object area is filled with black color, background area is filled with white color.</p><p>Boolean mask has the same height and width as the original mask, but only one channel. If a pixel belongs to the object area, its value is True, else — False.</p><p>Boolean mask will help us to remove all background pixels.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1430e25412e910450417aeeaa69d88e6/href">https://medium.com/media/1430e25412e910450417aeeaa69d88e6/href</a></iframe><p>Output:</p><pre>Image with removed background shape: (860, 1151, 3)<br>Boolean mask shape: (860, 1151)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*Sb9ykEcuNRVrOMs94_ENUQ.png" /></figure><h3>4. Adding object to background image</h3><p>Before we define the function which adds object to the background image, I need to explain and visualize several cases of images overlapping.</p><p>Let’s say, the background image has height <strong>h_background</strong> and width <strong>w_background</strong>, the object image has height <strong>h</strong> and width <strong>w</strong>.</p><p><strong>h</strong> should be leass than <strong>h_background</strong>, and <strong>w</strong> should be less than <strong>w_background</strong>.</p><p><strong>Case 1).</strong> If we place the object into the middle of the background, then everything is simple: the part of the background area of size <strong>h</strong> x <strong>w</strong> should be replaced with the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LhsEbg77yjgEFs9YX-bDKA.png" /><figcaption>Object is placed in the middle of the background. Overlapping area here is <strong>h </strong>x<strong> w</strong></figcaption></figure><p><strong>Case 2).</strong> If we place the object into the top left corner of the background, then part of the object may be outside of the background area. In this case the part of the background area of size <strong>(h - y)</strong> x <strong>(w - x)</strong> should be replaced with the object.</p><p>Here <strong>-x</strong> and <strong>-y</strong> are the coordinates of the top left corner of the object image. Sign ‘-’ is here because the top left corner of the background image has coordinates x=0 and y=0. Everything which is on the left from the top left corner of the background image has a negative x coordiate, and everything which is higher then the top left corner of the background image has a negative y coordiate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Orhl2g_9DVqkPhG-oMuEsA.png" /><figcaption>Object placed into the top left corner of the background. Overlapping area here is <strong>(h — y) </strong>x<strong> (w — x)</strong>.</figcaption></figure><p><strong>Case 3).</strong> If we place the object into the bottom left corner of the background, then part of the object may be outside of the background area. In that case the part of the background area of size <strong>(h_background - y)</strong> x <strong>(w - x)</strong> should be replaced with the object.</p><p>Generally, the area can be calculated as <strong>(h - max(0, y + h - h_background))</strong> x <strong>(w - x)</strong>, because if the lowest border of the object image is above the lowest border of the background image, then <strong>h </strong>x <strong>(w - x)</strong> area should be replaced with the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AciGKRZ5LnW_3Fo2Ib_pYA.png" /><figcaption>Object placed into the bottom left corner of the background. Overlapping area here is <strong>(h_background — y) </strong>x<strong> (w — x)</strong>. General formula for overlapping area here is <strong>(h — max(0, y + h — h_background)) </strong>x<strong> (w — x)</strong>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rR3TLk7JKQevfvs3IJpKyg.png" /><figcaption>Overlapping area here is <strong>h </strong>x<strong> (w — x)</strong>. General formula for overlapping area here is <strong>(h — max(0, y + h — h_background)) </strong>x<strong> (w — x)</strong>.</figcaption></figure><p><strong>Case 4).</strong> If we place the object into the top right corner of the background, then part of the object may be outside of the background area. In that case the part of the background area of size <strong>(h - y)</strong> x <strong>(w_background - x)</strong> should be replaced with the object.</p><p>Generally, the area can be calculated as <strong>(h - y)</strong> x <strong>(w - max(0, x + w - w_background))</strong>, because if the right border of the object image is on the left from the right border of the background image, then <strong>(h - y) </strong>x <strong>w</strong> area should be replaced with the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_Jt70LKbe7FrBS6TpPpyw.png" /><figcaption>Object placed into the top right corner of the background. Overlapping area here is <strong>(h — y) </strong>x<strong> (w_background — x)</strong>. General formula for overlapping area here is <strong>(h — y) </strong>x<strong> (w — max(0, x + w — w_background))</strong>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oHoOmBhtll97SR4s93oZfw.png" /><figcaption>In this case overlapping area is <strong>(h — y) </strong>x<strong> w</strong>. General formula for overlapping area here is <strong>(h — y) </strong>x<strong> (w — max(0, x + w — w_background))</strong>.</figcaption></figure><p><strong>Case 5).</strong> If we place the object into the bottom right corner of the background, then part of the object may be outside of the background area. In that case the part of the background area of size <strong>(h_background - y)</strong> x <strong>(w_background - x)</strong> should be replaced with the object.</p><p>Generally, the area can be calculated as <strong>(h - max(0, y + h - h_background))</strong> x <strong>(w - max(0, x + w - w_background))</strong>, because if the right part of the object image is on the left from the right part of the background image, and if the lowest part of the object image is above the lowest part of the background image, then <strong>h </strong>x <strong>w</strong> area should be replaced with the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMHzSqvLAX-seJN_ESJ2Sg.png" /><figcaption>Object placed into the bottom right corner of the background. Overlapping area here is (h_background — y) x (w_background — x). General formula for overlapping area here is (h — max(0, y + h — h_background)) x (w — max(0, x + w — w_background)).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHgLBgZvs1mhdLnE6V6tsQ.png" /><figcaption>In this case overlapping area here is <strong>h </strong>x<strong> w</strong>. General formula for overlapping area here is <strong>(h — max(0, y + h — h_background)) </strong>x<strong> (w — max(0, x + w — w_background))</strong>.</figcaption></figure><p>Now, taking into account all described above cases, lets define the function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff3d7548cb437e58e4903f5ea9d3578d/href">https://medium.com/media/ff3d7548cb437e58e4903f5ea9d3578d/href</a></iframe><p>Besides passing background, object and mask images to the function, we pass coordinates x and y, which define where the center of the object will be placed.</p><p>Coordinate (0, 0) is the top left corner of the background.</p><p>w_bg and h_bg are the width and height of background.</p><p>x and y should meet the following conditions: 0 &lt; x &lt; w_bg and 0 &lt; y &lt; h_bg.</p><p>Let’s look how the function works.</p><p><strong>Example 1).</strong> Let’s place the cup in the center of background. The width of background is 1920, the height is 1280, so coordinates of the object’s center are x=1920/2=960 and y=1280/2=640.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b793e1162f1fc3cb640012b307dbe04/href">https://medium.com/media/1b793e1162f1fc3cb640012b307dbe04/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*Ze5G184hBb_byT2UdjZOog.png" /></figure><p><strong>Example 2).</strong> Let’s place the cup in the bottom left of background. This time the coordinates of the object’s center are x=200 and y=1100.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4cca4bac633994fece70c6c4344a1ad/href">https://medium.com/media/d4cca4bac633994fece70c6c4344a1ad/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*yGGlGq2rzv4weoEkU1y-GQ.png" /></figure><p><strong>Example 3). </strong>Let’s place the cup in the bottom right of background. This time the coordinates of the object’s center are x=1800 and y=1100.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8de847efe6c1a3ec828d36e043b2e85/href">https://medium.com/media/d8de847efe6c1a3ec828d36e043b2e85/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*e8V2C_28nU_FtymYwIuKbA.png" /></figure><p><strong>Example 4). </strong>Let’s place the cup in the top left of background. This time the coordinates of the object’s center are x=200 and y=200.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a16082001d15237ffb96d03261cdea10/href">https://medium.com/media/a16082001d15237ffb96d03261cdea10/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*Jhh0Gfkb-AftMxPCOsvatw.png" /></figure><p><strong>Example 5). </strong>Let’s place the cup in the top right of background. This time the coordinates of the object’s center are x=1800 and y=200.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a063a51143793cdd36f4021e2613cf18/href">https://medium.com/media/a063a51143793cdd36f4021e2613cf18/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*p5QkHmD3mckCC6qkutUcBw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=133f165b9a01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replacing Part of 2D Array with Another 2D Array in Numpy]]></title>
            <link>https://medium.com/@alexppppp/replacing-part-of-2d-array-with-another-2d-array-in-numpy-c83144576ddc?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/c83144576ddc</guid>
            <category><![CDATA[arrays]]></category>
            <category><![CDATA[numpy-array]]></category>
            <category><![CDATA[numpy-tutorial]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Thu, 09 Dec 2021 19:00:36 GMT</pubDate>
            <atom:updated>2021-12-29T19:24:06.891Z</atom:updated>
            <content:encoded><![CDATA[<h4>Explanation with examples</h4><p>In this article I’ll show a couple of simple and powerful techniques for working with arrays in Python.</p><p>For example, we have 7x7 array:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*Pwng87SY6fTtnMooRWUNjA.png" /></figure><h3>Simple replacement</h3><p>Suppose that we need to replace the part of 7x7 array, which is highlighted with green color, with another 2x4 array:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*TXwwktb6QmU12T1a2LdK5w.png" /></figure><p>Create a new notebook in Jupyter Notebook. Import Numpy library and create arrays a, a_copy and b:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f566948121eb05af9938b075aa53ab47/href">https://medium.com/media/f566948121eb05af9938b075aa53ab47/href</a></iframe><p>Output:</p><pre>Array a:<br>[[1 1 1 1 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 1 1 1 1 1 1]]<br><br>Copy of array a:<br>[[1 1 1 1 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 1 1 1 1 1 1]]<br><br>Array b:<br>[[8 5 5 8]<br> [8 5 5 8]]</pre><p>Let’s replace elements in array a in the 4th &amp; 5th rows and in the 3rd, 4th, 5th &amp; 6th columns with array b:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c18ade7603cd86be42b1dd3667c297ea/href">https://medium.com/media/c18ade7603cd86be42b1dd3667c297ea/href</a></iframe><p>Output:</p><pre>Array a after being partially replaced with array b:<br>[[1 1 1 1 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 8 5 5 8 1]<br> [1 4 8 5 5 8 1]<br> [1 4 4 4 1 1 1]<br> [1 1 1 1 1 1 1]]</pre><p>Well, that was easy! A short line of code a[3:5,2:6] = b did the job!</p><h3>Conditional replacement with help of boolean mask</h3><p>Now, suppose that we need to do the same replacement, but with one condition. Only those values in array a which are overlapped with number 5 from array b should be replaced:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*tF0hP3QUwAfAdgv71bXJ0g.png" /></figure><p>We will do that with help of boolean mask.</p><p>Boolean mask can be easily retrieved with this code b_mask_boolean = b == 5. Let’s see how it looks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4317349227ff9539ec9a0f1f9385a68b/href">https://medium.com/media/4317349227ff9539ec9a0f1f9385a68b/href</a></iframe><p>Output:</p><pre>Boolean mask for numbers equal to 5 in array b:<br>[[False  True  True False]<br> [False  True  True False]]</pre><p>We can do cool things with help of boolean mask. For example, assign zero values to the elements of array, in which we are not interested in, or vice versa:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c134254c97bfeace135ce16ae165ba1/href">https://medium.com/media/8c134254c97bfeace135ce16ae165ba1/href</a></iframe><p>Output:</p><pre>Array b multiplied by boolean mask:<br>[[0 5 5 0]<br> [0 5 5 0]]<br><br>Array b multiplied by inversed boolean mask:<br>[[8 0 0 8]<br> [8 0 0 8]]<br><br>Part of array a (4th and 5th rows; 3rd, 4th, 5th and 6th columns) multiplied by boolean mask:<br>[[0 4 1 0]<br> [0 4 1 0]]<br><br>Part of array a (4th and 5th rows; 3rd, 4th, 5th and 6th columns) multiplied by inversed boolean mask:<br>[[4 0 0 1]<br> [4 0 0 1]]</pre><p>Let’s do the replacement:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cb7b0ff87bf180256410834b97868e2/href">https://medium.com/media/2cb7b0ff87bf180256410834b97868e2/href</a></iframe><p>Output:</p><pre>Array a after being partly replaced with array b (with numbers 5 from array b only):<br>[[1 1 1 1 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 4 1 1 1]<br> [1 4 4 5 5 1 1]<br> [1 4 4 5 5 1 1]<br> [1 4 4 4 1 1 1]<br> [1 1 1 1 1 1 1]]</pre><p>Essentially we have added up two these arrays:</p><pre>[[4 0 0 1]<br> [4 0 0 1]]</pre><p>and</p><pre>[[0 5 5 0]<br> [0 5 5 0]]</pre><p>to get this array</p><pre>[[4 5 5 1]<br> [4 5 5 1]]</pre><p>and replaced with this array the elements in array a in the 4th &amp; 5th rows and in the 3rd, 4th, 5th &amp; 6th columns.</p><p>Again, this solution was easy too! A line of code a_copy[3:5,2:6] = a_copy[3:5,2:6] * ~b_mask_boolean + b * b_mask_boolean did the job.</p><h3>Real example</h3><p>Here you can see how the described technique can be applied for images.</p><p>We have two images (background and a cup of coffee):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U5fJWsrhKKp9EKznE3_FWQ.png" /></figure><p>We can add a cup of coffee to the background in a simple way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*fN4Hl8D6C1CkSmaIrwyIag.png" /></figure><p>Or using a boolean mask:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*vyo9c1p93eL0nT6WflIsbQ.png" /></figure><p>However, you should remember, when we work with <strong><em>images</em></strong>, we work with <strong><em>3D arrays</em></strong>, because, besides the dimensions related to <strong><em>height</em></strong> and <strong><em>width</em></strong>, there is an additional dimension related to <strong><em>RGB channels</em></strong>. Read more details in the article <a href="https://medium.com/@alexppppp/adding-objects-to-image-in-python-133f165b9a01">“Adding Objects to Image in Python”</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c83144576ddc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Annotate Keypoints Using Roboflow]]></title>
            <link>https://medium.com/@alexppppp/how-to-annotate-keypoints-using-roboflow-9bc2aa8915cd?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bc2aa8915cd</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[annotation-tools]]></category>
            <category><![CDATA[keypoint-detection]]></category>
            <category><![CDATA[roboflow]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Fri, 26 Nov 2021 13:11:20 GMT</pubDate>
            <atom:updated>2023-06-18T10:28:18.860Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step guide on how to use an online annotation tool</h4><p><a href="https://roboflow.com/">Roboflow</a> is an easy to use online software for image annotation. I always use it when I need to annotate a dataset for object detection.</p><p>Annotating image for object detection means drawing and labeling bounding boxes around objects on the image.</p><p>However, when it comes to annotation of keypoints, Roboflow is showing the following message:</p><blockquote>Currently, we only support <strong>Object Detection (Bounding Box)</strong> and <strong>Single-Class Classification</strong> projects. We&#39;re prioritizing based on user demand. You can log your support and request this feature by adding your vote.</blockquote><p>Recently, I needed to do keypoint annotations for a dataset of images with glue tubes for <a href="https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da">training a custom Keypoint RCNN</a>.</p><p>Here are several random images from the <a href="https://drive.google.com/file/d/1ay5mrBYvlOodNHVun1cCN6PL_D6M5F-0/view?usp=sharing">dataset</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*JPs5rO_Y4INImSHeNVOq0Q.png" /></figure><p>Annotations for each image should include:</p><ul><li>coordinates of bounding boxes (each glue tube should have a bounding box, which is described with top left and bottom right corners in [x1, y1, x2, y2] format);</li><li>coordinates and visibility of keypoints (each glue tube should have 2 keypoints — <em>head</em> and <em>tail</em>— which are described in [x, y, visibility] format).</li></ul><p>Here is a vizualization of how annotations should look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqBkKLX-jLh3JTzOjWVYdg.jpeg" /></figure><p>But I couldn’t find an online tool to annotate keypoints. So I came up with an idea how to do that by using available Roboflow functionality and a custom python script.</p><p>On each image I have:</p><ul><li>drawn small rectangles in the places where keypoints should be;</li><li>drawn big rectangles, which should be the bounding boxes;</li><li>created a custom python script to calculate coordinates of the centers of small rectangles and saved that info to a json-file.</li></ul><p>Below is step-by-step description of the process.</p><h3>Annotate keypoints and bounding boxes</h3><p><strong>1).</strong> Register a free account at <a href="https://roboflow.com/">https://roboflow.com</a>, then log in and click “Create New Project”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X4hsXeKMgjtcmxPn8YPxLA.png" /></figure><p><strong>2).</strong> Give a name to your project and click “Create Public Project”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*ICd5kccUHJjJxkR2tDkcaw.png" /></figure><p><strong>3.1).</strong> Upload images from this <a href="https://drive.google.com/file/d/1ay5mrBYvlOodNHVun1cCN6PL_D6M5F-0/view?usp=sharing">archive</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JLYhWUUFXp5rCfzHhPXkrQ.png" /></figure><p><strong>3.2).</strong> After uploading you’ll be asked in which proportions (train/valid/test) do you want to split these images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*ZDfi4sVNv211lT-tOrOXRw.png" /></figure><p><strong>3.3).</strong> Choose 100% / 0% / 0% (if needed, you’ll be able to split the images manually later) and click “Continue”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*QZorBf9qm35F9ElQ0O9qKw.png" /></figure><p><strong>4.1).</strong> Now you’ll start the annotation process. Click on the first image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3cqRvOvzS1zHff3WC__wIw.png" /></figure><p><strong>4.2).</strong> After clicking on the first image, you’ll see the following window:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rYzMK9mvpHW6Kx2otxSEPg.png" /></figure><p><strong>4.3).</strong> Draw a rectangle around the left glue tube and set class name <strong><em>Tube</em></strong> for it, press “Enter”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cLf5fbieQOQ-qWXs2gX1hw.png" /></figure><p>You’ll see that the new class <strong><em>Tube</em></strong> appeared in the annotations column on the left:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTESICYLIQf-88NndCcI8w.png" /></figure><p>Draw a rectangle around the right glue tube. Now you have two bounding boxes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yHp1tOr9cU-2y8OePcVAHw.png" /></figure><p><strong>4.4).</strong> Draw a small rectangle in the center of the left glue tube cap. The center of this rectangle will be the first keypoint of glue tube. Set class name <strong><em>Head</em></strong> for it, press “Enter”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hXMylBtP6RWAzRaEcgxM1Q.png" /></figure><p>You’ll see that a new class <strong><em>Head</em></strong> appeared in the annotations column on the left:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7KKj6zZnkTVk8ay5kwAHg.png" /></figure><p>Do the same for the right glue tube cap. Now you have two rectangles related to the head keypoints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7H6Phr-QwhIjjH_i8oDXw.png" /></figure><p><strong>4.4).</strong> Draw a small rectangle in the center of the left glue tube tail. The center of this rectangle will be the second keypoint of glue tube. Set class name <strong><em>Tail</em></strong> for it, press “Enter”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knUmkw_xUO7oPXEcIGfgNA.png" /></figure><p>You’ll see that a new class <strong><em>Tail</em></strong> appeared in the annotations column on the left:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P53ecgoqKbV8vlUcFiMrRA.png" /></figure><p>Do the same for the right glue tube. Now you have two rectangles related to the tail<em> </em>keypoints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fj_k-Fy8xlILZT5a1z_GnA.png" /></figure><p><strong>4.5). Important!</strong> To convert the keypoint-related rectangles into keypoints, you’ll need to use a custom python script which I’ll provide later in this article. This script works correctly only if the bounding box of an object (glue tube in our case) contains only those keypoints, which are related to that particular object.</p><p>For example, on the image below the bounding box of the left glue tube contains not only its own head and tail keypoints, but also tail keypoint of the right glue tube. You should avoid such overlappings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYDOZ7Y2b4bsVOe1triSKw.png" /><figcaption>Avoid such overlappings</figcaption></figure><p>On the image below bounding boxes partially overlap, but it’s possible to draw keypoints in such a way that each bounding box contains only head and tail keypoints related to its glue tube:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVMw-joK4JGtzXfSPcTQaw.png" /><figcaption>Such overlappings are ok</figcaption></figure><p><strong>5.1).</strong> Now, after you’ve marked all the images, you need to download the dataset. Click on “Dataset” link in the left menu:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7l7WAzN1dNbxbOpbbPPktQ.png" /></figure><p><strong>5.2).</strong> By default, such preprocessing steps as <strong><em>Auto-Orient</em></strong> and <strong><em>Resize</em></strong> will be added. Remove those. Also, don’t add any augmentation on the next step.</p><p>Click the “Generate” button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQTLr5DahPylbgMP9Gi1rg.png" /></figure><p><strong>5.3).</strong> After dataset is generated you can give it a name and export it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFDB92HKD_-0NitFmOmsjQ.png" /></figure><p><strong>5.4).</strong> Among the export options choose <strong><em>“YOLO v5 PyTorch”</em></strong> format and <strong><em>“download zip to computer” </em></strong>and click <strong>“Continue”</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*qHa6BGCckQRbkfZkSvgQzQ.png" /></figure><p><strong>5.5). Good news!</strong> This time you don’t have to annotate the whole datased with glue tubes by yourself in Roboflow, I’ve already done this for you!) Just download it from <a href="https://drive.google.com/uc?id=101hZpYQMe75uoM1c6THfqYAyDaZkmV0L&amp;export=download">here</a> and unzip the archive.</p><h3>Convert annotations using python script</h3><p>In the downloaded archive you’ll see the file <strong>data.yaml</strong> and the folders <strong>train/images</strong> and <strong>train/labels</strong>.</p><p>File <strong>data.yaml</strong> contains the following list of classes names: [&#39;Tube&#39;, &#39;Head&#39;, &#39;Tail&#39;].</p><p>Every image in <strong>train/images</strong> folder has a corresponding txt-file in <strong>train/labels</strong> folder with the same name.</p><p>Txt-files from <strong>train/labels</strong> folder have the following structure (here is an example):</p><pre>2 0.7460938 0.3745370 0.0015625 0.0027778<br>0 0.6315104 0.4097222 0.2598958 0.1712963<br>1 0.5307292 0.4509259 0.0020833 0.0037037<br>1 0.4484375 0.4944444 0.0020833 0.0037037<br>0 0.3372396 0.5666667 0.2859375 0.2268519<br>2 0.2044271 0.6171296 0.0026042 0.0046296</pre><p>Each line in txt-file correspondents to some rectangle and consists of five numbers. The first number is the index of a rectangle class from the list [&#39;Tube&#39;, &#39;Head&#39;, &#39;Tail&#39;]. The other four numbers are the normalized coordinates of a rectangle in x_center y_center width height format.</p><p><strong>Hint!</strong> If you need, for example, to transform x-coordinate from normalized to absolute format, you should multiply a normalized x-coordinate by the width of the image in pixels.</p><p>To get keypoints and their coordinates, you need to convert those lines. If the first number is 0, the rectangle coordinates should be converted to the absolute coordinates of the glue tube bounding box in [x_top_left, y_top_left, x_bottom_right, y_bottom_right] format. If the first number is 1 or 2, the rectangle should be converted to the absolute coordinates of keypoints (head or tail) in [x, y, visibility] format.</p><p><strong>6.1).</strong> Create a new notebook in Jupyter Notebook. First, you need to import the necessary modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8fe5328b4e80c9a90018c2ce13b81cf/href">https://medium.com/media/a8fe5328b4e80c9a90018c2ce13b81cf/href</a></iframe><p><strong>6.2).</strong> Let’s look at the first image in <strong>/train/images</strong> folder:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a17f157157d5efedeed8cf1d86cb06c3/href">https://medium.com/media/a17f157157d5efedeed8cf1d86cb06c3/href</a></iframe><p>You’ll see the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60ygsnWhtBw1Uo9KNex6ig.png" /></figure><p><strong>6.3).</strong> Now let’s look what’s inside of the first txt-file in <strong>/train/labels</strong> folder:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90a780b43673710ebbf62e6716d8b3ce/href">https://medium.com/media/90a780b43673710ebbf62e6716d8b3ce/href</a></iframe><p>You’ll see the following output:</p><pre>Rectangle 1: [2, 0.7460938, 0.374537, 0.0015625, 0.0027778]<br>Rectangle 2: [0, 0.6315104, 0.4097222, 0.2598958, 0.1712963]<br>Rectangle 3: [1, 0.5307292, 0.4509259, 0.0020833, 0.0037037]<br>Rectangle 4: [1, 0.4484375, 0.4944444, 0.0020833, 0.0037037]<br>Rectangle 5: [0, 0.3372396, 0.5666667, 0.2859375, 0.2268519]<br>Rectangle 6: [2, 0.2044271, 0.6171296, 0.0026042, 0.0046296]</pre><p>Here the 2nd and the 5th rectangles are related to the bounding boxes, the 3rd and the 4th rectangles are related to head keypoints, the 1st and the 6th rectangles are related to tail keypoints. Now you need to find a match between keypoint-related rectangles and bounding box-related rectangles.</p><p><strong>6.4).</strong> Here is a function which matches keypoints and bounding boxes, and converts annotations from Roboflow format to the format required by KeypointRCNN model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d9527c5068c98fb02d2687f8d98635a/href">https://medium.com/media/0d9527c5068c98fb02d2687f8d98635a/href</a></iframe><p><strong>6.5).</strong> Let’s look how this function works:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c7eb34beb6d55bd3a85f8b3001b6522/href">https://medium.com/media/4c7eb34beb6d55bd3a85f8b3001b6522/href</a></iframe><p>You’ll see the following output:</p><pre>Bboxes: [[962, 350, 1462, 534], [374, 490, 922, 734]]<br>Keypoints: [[[1019, 487, 1], [1432, 405, 1]], [[861, 534, 1], [393, 667, 1]]]</pre><p>Here you can see that keypoints with coordinates [[1019, 487, 1], [1432, 405, 1]] are related to the bounding box with coordinates [962, 350, 1462, 534], and keypoints with coordinates [[861, 534, 1], [393, 667, 1]] are related to the bounding box with coordinates [374, 490, 922, 734].</p><p>Every keypoint here has visibility equal to 1.</p><p><strong>6.6).</strong> Let’s visualize bounding boxes and keypoints on the image:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd5f86c96db6b9b6d7130e7691025ddd/href">https://medium.com/media/bd5f86c96db6b9b6d7130e7691025ddd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*kGvS_5DgueggsDkGfm1PBA.png" /></figure><p><strong>6.7).</strong> Let’s define a function which saves annotations into json-file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5dd73fb11acae1f8c7c954405808f04/href">https://medium.com/media/b5dd73fb11acae1f8c7c954405808f04/href</a></iframe><p>The function dump2json() would save annotatins for the image from example above in the following way:</p><pre>{&quot;bboxes&quot;: [[962, 350, 1462, 534], [374, 490, 922, 734]], &quot;keypoints&quot;: [[[1019, 487, 1], [1432, 404, 1]], [[861, 534, 1], [392, 666, 1]]]}</pre><p><strong>6.8).</strong> Create <strong>/train/annotations</strong> folder.</p><p><strong>6.9). </strong>Run the final block of code which converts labels into annotations and saves them into <strong>/train/annotations</strong> folder:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e36f2b2df0d5e04302df7fec7f72d18/href">https://medium.com/media/0e36f2b2df0d5e04302df7fec7f72d18/href</a></iframe><p>After converting the annotations you need to split the dataset to train/test subsets manually.</p><p>Now, the dataset of glue tubes images with annotated keypoints is ready. You can also download it from <a href="https://drive.google.com/uc?id=1zO4nRvVOtiVChZdPpmh5Q08rmQMHwpXm&amp;export=download">here</a>.</p><p><a href="https://github.com/alexppppp/annotate-keypoints-roboflow">Here</a> is a GitHub repository and a notebook with all the steps described above.</p><p><strong>UPDATE:</strong> It may be interesting for you to read this article: <a href="https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da">“How to Train a Custom Keypoint Detection Model with PyTorch”</a></p><h3>Another in-depth articles about computer vision:</h3><p><strong>Tutorails</strong> on <strong>how to generate synthetic datasets for computer vision</strong> with help of Python, OpenCV, Numpy and Albumentations:</p><ul><li><strong>For object detection</strong>: <a href="https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-object-detection-fd8ab2fa5249">https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-object-detection-fd8ab2fa5249</a></li><li><strong>For keypoint detection</strong>: <a href="https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd">https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bc2aa8915cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Train a Custom Keypoint Detection Model with PyTorch]]></title>
            <link>https://medium.com/@alexppppp/how-to-train-a-custom-keypoint-detection-model-with-pytorch-d9af90e111da?source=rss-4f1e99fc4a77------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9af90e111da</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[keypoint-detection]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Alex P]]></dc:creator>
            <pubDate>Sun, 14 Nov 2021 10:49:39 GMT</pubDate>
            <atom:updated>2023-06-18T10:09:45.624Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tutorial on how to fine-tune Keypoint RCNN</h4><p>By default, PyTorch provides a Keypoint RCNN model which is pre-trained to detect 17 keypoints of the human body (nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles).</p><p>The keypoints on this picture are predicted by this model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bGWF79zIYOYIxvn_F77Erg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@jvdm?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Jean van der Meulen</strong></a> from <a href="https://www.pexels.com/photo/man-wakeboarding-1547711/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels"><strong>Pexels</strong></a></figcaption></figure><p>I will demonstrate how to fine-tune the above-mentioned model using a custom dataset. For this purpose I’ve created a dataset of images with glue tubes and assigned two keypoints to each glue tube (<em>head</em> and <em>tail</em>).</p><h3>Images and annotations (custom dataset)</h3><p>The dataset includes 111 training and 23 test images. Each image has one or two objects (glue tubes).</p><p>Annotations for each image include:</p><ul><li>coordinates of bounding boxes (each object has a bounding box, which is described with top left and bottom right corners in [x1, y1, x2, y2] format);</li><li>coordinates and visibility of keypoints (each object has 2 keypoints, which are described in [x, y, visibility] format).</li></ul><p>All keypoints are visible (i.e. visibility=1) in this dataset. The 1st keypoint is <em>head</em>, the 2nd one is <em>tail</em>.</p><p>You can download the dataset <a href="https://drive.google.com/file/d/1VgY4IfjtgdI78s380UvtpdRWAwWOL3R9/view?usp=sharing">here</a>.</p><p>Take a look at several random images from the dataset and a random image with visualized annotations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*zV1TzXT81AjehFdihyCNqw.png" /><figcaption>Random images from the <a href="https://drive.google.com/file/d/1VgY4IfjtgdI78s380UvtpdRWAwWOL3R9/view?usp=sharing">dataset</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqBkKLX-jLh3JTzOjWVYdg.jpeg" /><figcaption>Random image with visualized annotations</figcaption></figure><h3>Pycocotools library adjustment</h3><p>During the training process we will evaluate some metrics of our model. This is done with help of <a href="https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools">pycocotools</a> library. Go ahead and install it with pip install pycocotools command.</p><p>To evaluate how precisely predicted keypoints match with the ground truth keypoints, pycocotools uses class COCOeval which, by default, is tuned to evaluate 17 keypoints of human body. But if we want to evaluate a custom set of keypoints (in our case it’s only 2 keypoints), we need to change the predefined array of coefficients kpt_oks_sigmas in that script.</p><p>To do that we need to open <strong><em>pycocotools/cocoeval.py</em></strong> file and change the line self.kpt_oks_sigmas = np.array([.26, .25, .25, .35, .35, .79, .79, .72, .72, .62,.62, 1.07, 1.07, .87, .87, .89, .89])/10.0 to self.kpt_oks_sigmas = np.array([.5, .5])/10.0</p><p>For example, in Google Colab this file can be found via the following path: <strong><em>/usr/local/lib/python3.7/dist-packages/pycocotools/cocoeval.py</em></strong></p><p>You can read <a href="https://cocodataset.org/#keypoints-eval">here</a> the description of keypoint evaluation metrics, object keypoint similarity (OKS) and OKS coefficients.</p><p><strong>Update</strong>. It’s possible not to edit <strong><em>pycocotools/cocoeval.py</em> </strong>file in pycocotools library to change kpt_oks_sigmas, but to edit <strong><em>coco_eval.py</em></strong> file, as <a href="https://medium.com/u/9de8cdcafc94">Diogo Santiago</a> suggested:</p><pre># self.coco_eval[iou_type] = COCOeval(coco_gt, iouType=iou_type)</pre><pre>coco_eval = COCOeval(coco_gt, iouType=iou_type)</pre><pre>coco_eval.params.kpt_oks_sigmas = np.array([.5, .5]) / 10.0</pre><pre>self.coco_eval[iou_type] = coco_eval</pre><h3>1. Imports</h3><p>Create a new notebook in Jupyter Notebook. First, we need to import the necessary modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76181dfae0fa3a58425d60d92e3aeae2/href">https://medium.com/media/76181dfae0fa3a58425d60d92e3aeae2/href</a></iframe><p>Next, download <strong><em>coco_eval.py</em></strong> , <strong><em>coco_utils.py</em></strong>, <strong><em>engine.py</em></strong>, <strong><em>group_by_aspect_ratio.py</em></strong>, <strong><em>presets.py</em></strong>, <strong><em>train.py</em></strong>, <strong><em>transforms.py</em></strong>, <strong><em>utils.py</em></strong> files from this <a href="https://github.com/pytorch/vision/tree/main/references/detection">repository</a> and place them into the folder with your notebook.</p><p>Import these modules as well:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe757d6797dd07b6d7d6ea48bb554f20/href">https://medium.com/media/fe757d6797dd07b6d7d6ea48bb554f20/href</a></iframe><h3>2. Augmentations</h3><p>Here we will define a function with augmentations for the training process. This function will apply different transforms to the images before each training iteration. Among such transforms there could be a random change of brightness and contrast or an image rotation by 90 degrees random number of times.</p><p>Thus, we essentially “create new images”, which are in some ways differ from original ones, but still perfectly suitable for training our model.</p><p>We will use <a href="https://albumentations.ai/">albumentations</a> library for augmentations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/675610e9d6ea832323891ca61a84abe3/href">https://medium.com/media/675610e9d6ea832323891ca61a84abe3/href</a></iframe><h3>3. Dataset class</h3><p>The dataset should inherit from the standard torch.utils.data.Dataset class, and __getitem__ should return images and targets.</p><p>Below is the description of the parameters for the targets:</p><ul><li>boxes (FloatTensor[N, 4]): the coordinates of the N bounding boxes in [x0, y0, x1, y1] format.</li><li>labels (Int64Tensor[N]): the label for each bounding box. 0 always represents the background class.</li><li>image_id (Int64Tensor[1]): an image identifier.</li><li>area (Tensor[N]): the area of the bounding box.</li><li>iscrowd (UInt8Tensor[N]): instances with iscrowd=True will be ignored during evaluation.</li><li>keypoints (FloatTensor[N, K, 3]): for each one of the N objects, it contains the K keypoints in [x, y, visibility] format, defining the object. visibility=0 means that the keypoint is not visible.</li></ul><p>Let’s define dataset class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d4bd1d423ea24205762a324938e5f33/href">https://medium.com/media/9d4bd1d423ea24205762a324938e5f33/href</a></iframe><p><strong>Here are additional explanations for the part of the dataset class where augmentations are applied (right after </strong><strong>if self.transform: line):</strong></p><p>The description of Keypoint RCNN states that keypoints should be provided in [x, y, visibility] format.</p><p>In case we want to apply augmentations to the image and its annotations using albumentations library, we should use [x, y] format. Besides that, the list of all keypoints should not be nested.</p><p>So, we need to modify keypoints in the initial list from [x, y, visibility] format to [x, y] format and flatten the list, then apply augmentations, and afterwards unflatten the list and modify keypoints back from [x, y] format to [x, y, visibility] format.</p><p><strong>For example</strong>, if the image contains two objects, and keypoints are described with the list [[[392, 1247, 1], [152, 1055, 0]], [[530, 993, 1], [622, 660, 1]]]:</p><ul><li>First, we modify the list to [[392, 1247], [152, 1055], [530, 993], [622, 660]], which is ok for albumentations API.</li><li>Next, after we apply albumentations augmentations, we get a list of transformed keypoints [[672, 392], [864, 152], [926, 530], [1259, 622]].</li><li>Finally, we modify the list of transformed keypoints back to [[[672, 392, 1], [864, 152, 0]], [[926, 530, 1], [1259, 622, 1]]], which is ok for Keypoint RCNN API.</li></ul><h3>4. Visualizing a random item from dataset</h3><p>Here we will look at an example of original and transformed targets:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8329e555587ba2a8ceb381675b108bd/href">https://medium.com/media/b8329e555587ba2a8ceb381675b108bd/href</a></iframe><p>Output:</p><pre>Original targets:<br>({&#39;boxes&#39;: tensor([[296., 116., 436., 448.],<br>                   [577., 589., 925., 751.]]),<br>&#39;labels&#39;: tensor([1, 1]),<br>&#39;image_id&#39;: tensor([15]),<br>&#39;area&#39;: tensor([46480., 56376.]),<br>&#39;iscrowd&#39;: tensor([0, 0]),<br>&#39;keypoints&#39;: tensor([[[408., 407.,   1.],<br>                      [332., 138.,   1.]],<br>                     [[886., 616.,   1.],<br>                      [600., 708.,   1.]]])},<br>)<br></pre><pre>Transformed targets:<br>({&#39;boxes&#39;: tensor([[ 116., 1484.,  448., 1624.],<br>                   [ 589.,  995.,  751., 1343.]]),<br>  &#39;labels&#39;: tensor([1, 1]),<br>  &#39;image_id&#39;: tensor([15]),<br>  &#39;area&#39;: tensor([46480., 56376.]),<br>  &#39;iscrowd&#39;: tensor([0, 0]),<br>  &#39;keypoints&#39;: tensor([[[4.0700e+02, 1.5110e+03, 1.0000e+00],<br>                        [1.3800e+02, 1.5870e+03, 1.0000e+00]],<br>                       [[6.1600e+02, 1.0330e+03, 1.0000e+00],<br>                        [7.0800e+02, 1.3190e+03, 1.0000e+00]]])},<br>)</pre><p>Here we will look at an example of original and transformed images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae7fe15401d50475f8eef7d1d805c5e9/href">https://medium.com/media/ae7fe15401d50475f8eef7d1d805c5e9/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Td-rUpVZp-QG2omeKQTUTA.jpeg" /></figure><h3>5. Training</h3><p>Here we define a function which returns Keypoint RCNN model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91ce5fdfda3fb5e3ff60531c7c60989c/href">https://medium.com/media/91ce5fdfda3fb5e3ff60531c7c60989c/href</a></iframe><p>By default, the AnchorGenerator class in PyTorch has 3 different sizes sizes=(128, 256, 512) and 3 different aspect ratios aspect_ratios=(0.5, 1.0, 2.0 (look <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/anchor_utils.py">here</a>). I’ve extended those parameters to sizes=(32, 64, 128, 256, 512) and aspect_ratios=(0.25, 0.5, 0.75, 1.0, 2.0, 3.0, 4.0).</p><p>Training loop:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6f60b50abe0e697bb11e5c4dfd47aef/href">https://medium.com/media/f6f60b50abe0e697bb11e5c4dfd47aef/href</a></iframe><p>In the training loop I’ve used 3 images per batch. In this case about 10 GB of GPU VRAM are used, so the model can be trained using Google Colab.</p><p>I’ve got very good metrics already after the 5th epoch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*IQNNl27PvjamLMxi5rVwbQ.jpeg" /></figure><h3>6. Visualizing model predictions</h3><p>Now let’s look how a trained model predicts bounding boxes and keypoints for glue tubes on a random image from the test dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc535beaa6d4731118ebe179e4631c68/href">https://medium.com/media/fc535beaa6d4731118ebe179e4631c68/href</a></iframe><p>Output:</p><pre>Predictions:</pre><pre>[{&#39;boxes&#39;: tensor([[ 618.9335,  144.0377, 1111.2960,  529.3129],<br>                   [ 741.4827,  420.9630, 1244.8071,  930.4985],<br>                   [ 653.7405,  258.7889, 1018.7531,  509.9501],<br>                   [ 824.6623,  540.7152, 1170.4821,  886.6503],<br>                   [ 711.1497,    0.0000, 1134.0641, 1066.0247],<br>                   [ 708.5067,  177.0665, 1102.3306,  385.1994],<br>                   [ 657.0708,  398.0692,  987.9990,  498.4578],<br>                   [ 887.4133,  453.8322, 1184.2448,  727.9111],<br>                   [ 895.7014,   52.4423, 1106.8652, 1080.0000],<br>                   [ 545.8564,  318.9463, 1276.8043,  519.7277],<br>                   [ 732.6523,    0.0000,  891.0267,  918.9849],<br>                   [ 794.4460,  667.6695, 1091.6316,  861.5293],<br>                   [ 809.3927,  273.1192, 1037.3994,  915.0168],<br>                   [ 603.3748,  293.8343, 1473.1097,  860.4436],<br>                   [ 991.6447,  218.8240, 1144.5980,  924.2585],<br>                   [ 419.0262,  196.2676, 1204.9933,  679.9295],<br>                   [ 880.3656,  274.3975, 1166.3279,  863.6169],<br>                   [1006.1213,  478.2608, 1208.6801,  746.1869],<br>                   [ 390.1542,  234.1698, 1592.7747,  502.9070],<br>                   [ 433.5611,  472.5373, 1346.7277, 1010.1754],<br>                   [ 394.9036,   59.5816, 1268.1086,  491.0312]],                  device=&#39;cuda:0&#39;),</pre><pre>  &#39;labels&#39;: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], device=&#39;cuda:0&#39;),</pre><pre>  &#39;scores&#39;: tensor([0.9955, 0.9911, 0.7638, 0.7525, 0.7217, 0.3831, 0.3320, 0.3311, 0.2415, 0.1709, 0.1700, 0.1456, 0.1174, 0.1086, 0.1041, 0.1025, 0.0758, 0.0608, 0.0604, 0.0582, 0.0510], device=&#39;cuda:0&#39;),</pre><pre>  &#39;keypoints&#39;: tensor([[[6.6284e+02, 4.6822e+02, 1.0000e+00],<br>                        [1.0645e+03, 2.0082e+02, 1.0000e+00]],<br>  <br>                       [[1.1794e+03, 4.8645e+02, 1.0000e+00],<br>                        [8.3855e+02, 8.4773e+02, 1.0000e+00]],<br>  <br>                       [[6.6883e+02, 4.6905e+02, 1.0000e+00],<br>                        [6.5446e+02, 4.7048e+02, 1.0000e+00]],<br>  <br>                       [[8.2538e+02, 8.4989e+02, 1.0000e+00],<br>                        [8.4260e+02, 8.4557e+02, 1.0000e+00]],<br>  <br>                       [[1.1333e+03, 2.0672e+02, 1.0000e+00],<br>                        [8.3846e+02, 8.5642e+02, 1.0000e+00]],<br>  <br>                       [[1.0571e+03, 1.7778e+02, 1.0000e+00],<br>                        [1.0628e+03, 2.0219e+02, 1.0000e+00]],<br>  <br>                       [[6.7074e+02, 4.6476e+02, 1.0000e+00],<br>                        [6.5779e+02, 4.9774e+02, 1.0000e+00]],<br>  <br>                       [[1.1721e+03, 4.9329e+02, 1.0000e+00],<br>                        [1.1835e+03, 4.9329e+02, 1.0000e+00]],<br>  <br>                       [[1.1061e+03, 2.1457e+02, 1.0000e+00],<br>                        [1.0573e+03, 2.0160e+02, 1.0000e+00]],<br>  <br>                       [[6.6456e+02, 4.6882e+02, 1.0000e+00],<br>                        [6.6312e+02, 4.7025e+02, 1.0000e+00]],<br>  <br>                       [[8.9031e+02, 9.1682e+02, 1.0000e+00],<br>                        [8.4279e+02, 8.5057e+02, 1.0000e+00]],<br>  <br>                       [[7.9516e+02, 8.6081e+02, 1.0000e+00],<br>                        [8.3823e+02, 8.4358e+02, 1.0000e+00]],<br>  <br>                       [[8.1011e+02, 8.4521e+02, 1.0000e+00],<br>                        [8.4166e+02, 8.4809e+02, 1.0000e+00]],<br>  <br>                       [[6.6745e+02, 4.6612e+02, 1.0000e+00],<br>                        [8.3017e+02, 8.5828e+02, 1.0000e+00]],<br>  <br>                       [[1.1439e+03, 4.9884e+02, 1.0000e+00],<br>                        [1.0696e+03, 2.2098e+02, 1.0000e+00]],<br>  <br>                       [[6.6590e+02, 4.6905e+02, 1.0000e+00],<br>                        [1.0632e+03, 1.9699e+02, 1.0000e+00]],<br>  <br>                       [[1.1656e+03, 4.9553e+02, 1.0000e+00],<br>                        [8.8108e+02, 8.6146e+02, 1.0000e+00]],<br>  <br>                       [[1.1749e+03, 4.9195e+02, 1.0000e+00],<br>                        [1.1749e+03, 4.7898e+02, 1.0000e+00]],<br>  <br>                       [[6.6741e+02, 4.6914e+02, 1.0000e+00],<br>                        [1.1859e+03, 5.0219e+02, 1.0000e+00]],<br>  <br>                       [[1.1804e+03, 4.7470e+02, 1.0000e+00],<br>                        [8.3901e+02, 8.4514e+02, 1.0000e+00]],<br>  <br>                       [[6.6463e+02, 4.9031e+02, 1.0000e+00],<br>                       [1.0646e+03, 1.9980e+02, 1.0000e+00]]],  device=&#39;cuda:0&#39;),</pre><pre>  &#39;keypoints_scores&#39;: tensor([[36.9580, 26.7403],<br>                              [31.9451, 28.6134],<br>                              [22.5176, -0.4728],<br>                              [ 7.7444, 21.3082],<br>                              [ 1.3215,  7.6223],<br>                              [ 2.0522, 22.6735],<br>                              [26.5938, -2.3956],<br>                              [19.8818,  2.7854],<br>                              [ 0.5259, 16.2155],<br>                              [39.5929, -0.1582],<br>                              [ 0.4924, 21.0935],<br>                              [ 0.5597, 19.3637],<br>                              [ 3.4223, 25.5078],<br>                              [17.6618,  0.4896],<br>                              [ 5.9306, -1.5709],<br>                              [27.4080,  2.4160],<br>                              [11.7086, -1.3879],<br>                              [26.0192,  3.0886],<br>                              [15.6420, -1.7428],<br>                              [ 7.1422, 10.9291],<br>                              [14.1688, 15.1565]], device=&#39;cuda:0&#39;)}]</pre><p>Here we see a lot of predicted objects. We will choose only those with high confidence score (for example, &gt; 0.7). Then we will apply Non-Maximum Surpression (NMS) procedure to select the most appropriate bounding boxes among remaining ones.</p><p>Essentially, NMS leaves the boxes with the highest confidence score (the best candidates) and removes other boxes, which partially overlap the best candidates. To define the degree of this overlapping, we will set the threshold for Intersection over Union (IoU) equal 0.3.</p><p>Read more about NMS implementation in PyTorch <a href="https://pytorch.org/vision/stable/ops.html">here</a>.</p><p>Let’s visualize predictions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b53e07ad13a24256674e30e1b7cc55e/href">https://medium.com/media/9b53e07ad13a24256674e30e1b7cc55e/href</a></iframe><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jRsCNLrikdvITkqZc3IPg.jpeg" /></figure><p>Predictions look good: bounding boxes are almost precise, keypoints are in the right places. It means the model is trained quite well.</p><p>In the same manner you can train Keypoint RCNN using another dataset, choosing any number of keypoints.</p><p><a href="https://github.com/alexppppp/keypoint_rcnn_training_pytorch">Here</a> is a GitHub repository and notebook with all the steps described above.</p><p><strong>UPDATE:</strong> It may be interesting for you to read this article: <a href="https://medium.com/@alexppppp/how-to-annotate-keypoints-using-roboflow-9bc2aa8915cd">“How to Annotate Keypoints Using Roboflow”</a></p><h3>Another in-depth articles about computer vision:</h3><ol><li><a href="https://medium.com/@alexppppp/how-to-train-an-ensemble-of-convolutional-neural-networks-for-image-classification-8fc69b087d3"><strong>How to Train an Ensemble of Convolutional Neural Networks for Image Classification</strong></a>: tutorial on how to create an ensemble of DenseNet161, ResNet152 and VGG19 for classification of TinyImageNet.</li><li><strong>Tutorails</strong> on <strong>how to generate synthetic datasets for computer vision</strong> with help of Python, OpenCV, Numpy and Albumentations:</li></ol><ul><li><strong>For object detection</strong>: <a href="https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-object-detection-fd8ab2fa5249">https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-object-detection-fd8ab2fa5249</a></li><li><strong>For keypoint detection</strong>: <a href="https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd">https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9af90e111da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>