<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Edward Umaña on Medium]]></title>
        <description><![CDATA[Stories by Edward Umaña on Medium]]></description>
        <link>https://medium.com/@edwarduw?source=rss-73368d896cfc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7nfv_emjP42YryJx.jpg</url>
            <title>Stories by Edward Umaña on Medium</title>
            <link>https://medium.com/@edwarduw?source=rss-73368d896cfc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:37:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@edwarduw/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nvidia Jetson Nano: Custom Object Detection from scratch using Tensorflow and OpenCV]]></title>
            <link>https://medium.com/swlh/nvidia-jetson-nano-custom-object-detection-from-scratch-using-tensorflow-and-opencv-113fe4dba134?source=rss-73368d896cfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/113fe4dba134</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[jetson-nano]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Edward Umaña]]></dc:creator>
            <pubDate>Tue, 27 Aug 2019 03:37:09 GMT</pubDate>
            <atom:updated>2019-08-27T20:25:53.077Z</atom:updated>
            <content:encoded><![CDATA[<p>The purpose of this blog is to guide users on the creation of a custom object detection model with performance optimization to be used on an NVidia Jetson Nano.</p><p>This is a report for a final project but we decided to make it similar to a tutorial so it’s posible for other developers to reproduce this project. For simplicity we will asume everything is installed. So take a look at the requirements before starting the tutorial.</p><blockquote>By: Luis Arias, Guillermo López and Edward Umaña</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0EQ4wsGPi2V5DE5OYTv9w.jpeg" /></figure><h3>Table of Contents</h3><ol><li>Requirements</li><li>Creating a custom dataset</li><li>Pre-processing</li><li>Labeling</li><li>Creating TFRecords</li><li>CNN</li><li>Training</li><li>Models</li><li>RPI Camera With Jetson Nano</li><li>Conclusions</li><li>Demo</li></ol><h3>Requirements</h3><ul><li>TensorFlow</li><li>OpenCV</li><li>Jupyter Notebook</li><li>Jetson Nano Developer Kit</li><li>RPI Camera</li><li>Python</li></ul><h4>Selection criteria</h4><p>We chose Python as the development language as it supports a vast amount of libraries for handling data, and because it easily integrates with TensorFlow. It is also a powerful scripting tool which helped us automate tedious tasks. Additionally, all of the object detection architectures explored in this blog were provided with utility Python scripts.</p><p>TensorFlow, as a well known framework for Deep Learning applications, seemed like a good option for us to explore and learn. It provides good documentation and many examples are available online.</p><p>OpenCV proved to be a good library for image processing, as per our experience in previous projects. It was as useful in the development described in this blog, as it facilitated image retrieval and handling on the Jetson Nano with the RPI Camera.</p><p>With that being said, let’s roll out!</p><h3>Creating a custom dataset</h3><p>The first thing we need to perform for custom object detection was creating our own dataset. The neural network to train will need several images of the objects we want to detect, in different places, with different conditions, positions and backgrounds.</p><p>For instance, we decided to detect some of the most popular beers in Costa Rica, such as Imperial, Pilsen and Bohemia. After a quick visit to the supermarket we bought six diferent brands of beer and then took about 140 pictures for each of them. Here are some samples of our images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*dvJRrDFl8oxlX9K0opNxaQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*Kout9iuWziiR-eP7TJAc3A.jpeg" /></figure><h3>Pre-processing</h3><p>Now that we have a plethora of images (840 in our case), we need to prepare them for training our custom model. At this moment they have a resolution a little too big for efficient use, so we chose to downscale them to an smaller resolution.</p><p>Please don’t do that manually; you may use the code attached below to downscale all the images inside a folder. Feel free to change the height and width specified in the script, but remember: the bigger the image, the greater the amount of computation required for training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e10c579396d907349a080501f2f0e53/href">https://medium.com/media/5e10c579396d907349a080501f2f0e53/href</a></iframe><p>Now we need to group the images in folders according to the category they belong to. We can go a step further by creating a Train and Test folder with all categories inside them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*EUC4_HV6uFpOHm3gk0xWGg.png" /></figure><h3>Labeling</h3><p>With the images resized accordingly, now we need to label them. In this activity we go image by image and indicate the coordinates of the object we want to detect and the label to which it belongs.</p><p>We used the open source tool <strong>LabelImg </strong>to make this process easier. You can download it from <a href="https://tzutalin.github.io/labelImg/">https://tzutalin.github.io/labelImg/</a>.</p><p>At the end of the process, you should have each image with its corresponding label, and an xml file with all the metadata of the object’s location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*TRvOmA0frhnAJBqhmDP1Dw.png" /><figcaption>Image with the corresponding label</figcaption></figure><h3>Creating TFRecords</h3><p>After finishing with the image labeling for the different classes, the next step is to split them into a group for training and another for testing. In this case we separate the images along with their corresponding .xml files into different folders. The division made was 80/20, which means that 80% of the images would go to the Train folder and the remaining 20% to the Test folder.</p><p>Now we need to create our TFRecords! These files will contain all the required information to train the neural network, including the images. If you want more details about TFRecords, we found this medium blog very helpful: <a href="https://medium.com/mostly-ai/tensorflow-records-what-they-are-and-how-to-use-them-c46bc4bbb564">blog here</a>.</p><p>We created a script that creates the TFRecords for us. To use the python script you just need to make two changes: the variables <em>PATH_*</em> and the function <em>class_text_to_int</em>. The <em>PATH_*</em> variables shall point to the train and test folders of the dataset and output files (.record). The <em>class_text_to_int()</em> function’s purpose is to translate the classes on the dataset to a number, so you would need to add all the labels created for the images during the labeling phase.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc274149c2b873258cc2ed649dddfaea/href">https://medium.com/media/cc274149c2b873258cc2ed649dddfaea/href</a></iframe><h3><strong>CNN</strong></h3><p>Now you can create the model for the desired CNN, in any flavor you want. You may even use some of the model configurations provided by TensorFlow, many of which can be used for object detection: <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md">Tensor Flow CNN Models</a>.</p><p>In our case, we used 2 different models. At first we tried out <a href="http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2018_01_28.tar.gz">ssd_inception_v2_coco</a>, and then <a href="http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz">ssdlite_mobilenet_v2_coco</a>. They exhibit different performance and accuracy. We will discuss the trade-offs later on this blog. First let discuss how to configure everything for the training phase.</p><p>We recommend creating a new training folder. In there you will need to place the TFRecords, model.config (the one you created or downloaded from <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md">Tensor Flow CNN Models</a>), and the label map.</p><p>Yes, we haven’t talked about about the label map. But it’s easy, just create a new file with the following format, which contains the label name and the number id. This should map the IDs specified in the Python script to create the TFRecord.</p><pre>item {<br>    id: 1<br>    name: &#39;Pilsen&#39;<br>}<br>item {<br>    id: 2<br>    name: &#39;Imperial&#39;<br>}</pre><p>Now we need to change some of the fields on the model.config you chose:</p><p>First, change the variable <em>num_classes</em> to meet the correct amount of classes for your chosen dataset.</p><pre>num_classes: 2</pre><p>Also, find the <em>train_input_reader</em> and <em>eval_input_reader</em> tags and change the input path. The path should point to the train.record and test.record files you created. Moreover, <em>label_map_path</em> should point to the location of the label map.</p><pre>train_input_reader: {<br> tf_record_input_reader {<br>  input_path: “./train.record”<br> }<br> label_map_path: “./labelmap.pbtxt”<br>}</pre><pre>eval_input_reader: {<br> tf_record_input_reader {<br>  input_path: “./test.record”<br> }<br> label_map_path: “./labelmap.pbtxt”<br> shuffle: false<br> num_readers: 1<br>}</pre><p>Those particular changes are mandatory, but we also performed some other changes to the model in order to improve performance and tailor the model to our convenience, as explained below.</p><p>First we set the image size to a fixed lower resolution (360, 270):</p><pre>image_resizer {<br> fixed_shape_resizer {<br>  height: 360<br>  width: 270<br>  resize_method: AREA<br> }<br>}</pre><p>Additionaly, we reduced the number of possible detections in a single image from 1000 to only 1, to make training simpler.</p><pre>post_processing {<br> batch_non_max_suppression {<br>  score_threshold: 1e-8<br>  iou_threshold: 0.6<br>  max_detections_per_class: 1<br>  max_total_detections: 1<br> }<br> score_converter: SIGMOID<br>}</pre><p>After that, we also changed the <em>fine_tune_checkpoint</em> tag to point to the model of your selection:</p><pre>“./ssdlite_mobilenet_v2_coco_2018_05_09/model.ckpt”</pre><h3>Training</h3><p>Once you have the config file ready, you can start training the neural network! For this you may issue the following command:</p><pre>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssdlite_mobilenet_v2_coco.config</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*VgvkbnoccE5j81duxxmApA.jpeg" /></figure><p>We recommend using TensorBoard to follow the progress of the training.</p><pre>tensorboard --logdir=training/</pre><p>We trained our model on a basic laptop with an 8th Gen Intel i5, with no GPU, and it took about 20 hours to reach a reasonable total loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*D9og2n7coTEZy1PMZ-ZvZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*HFyPvEAg0pNl-VC__HgY6Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*bbJS3m0rtc-IOkPeRS16ow.png" /></figure><p>Once the training is ready, you can create a <em>.pb</em> file that will be used in the Jetson Nano as an input for the loading the model into a Tensor Graph.</p><pre>python export_inference_graph.py --input_type image_tensor --pipeline_config_path training/ssdlite_mobilenet_v2_coco.config --trained_checkpoint_prefix training/model.ckpt-XXXX --output_directory inference_graph</pre><p>Don’t drink the beers during training…. you will need them later.</p><h3>Models</h3><p>As mentioned before, we tried out two different models. The first one was SDD Inception V2. This model suggests to use several operations on each layer instead of only one, and then concatenate the results. Typically, on a layer we choose between doing filters of fixed size (1x1, 3x3, or 5x5), or a max pooling layer, or another operation. In the other hand, on an Inception model we would use all of them on a single layer. With the latter, a better accuracy score was obtained, and all the different beer brands were detected with an accuracy score between 85% and 95%. To accomplish this accuracy it was not necessary to train the neural network for much time (relatively), around 12 hours sufficed.</p><p>The second model used was the SSD Lite Mobilenet V2. This model was created to run on mobile devices; it splits all the convolutions on different substaks. At the end it does approximately the same thing as traditional convolution but faster. For the SSD Lite Mobilenet V2 the accuracy obtained was between 60% and 85%. To accomplish this accuracy it was necessary to train the neural network for around 20 hours. Furthermore, we decided to narrow down the amount of classes to 2 instead of 6, to reduce training time and to simplify the model.</p><p>Which model should you choose? Well, it would be pretty obvious to choose the SDD Inception V2 model because it presented the best performance. But bear in mind that several factors are to be considered if you want to run the model on the Jetson Nano.</p><p>The SDD Inception V2 needs more RAM to work than the SDD Lite Mobilenet V2. In our experience, the Jetson Nano could not even run the SDD Inception V2 model as it crashed upon running out of memory most of the times we tried it out. When it was possible to run the application, the best performance on the image display was around 0,9FPS. Meanwhile, the SDD Lite Mobilenet V2 presented a performance of around 9FPS. Because of this, we decided to stick with the SDD Lite Mobilenet V2 architecture instead of the SDD Inception V2.</p><h3>RPI Camera With Jetson Nano</h3><p>Interfacing the RPI camera with the Jeston Nano is not a big deal when using the Jetson Nano Developer kit, which already has the necessary drivers installed. Thus, by using the CSI-Camera GitHub repository, we can easily gain access to the camera, as NVidia’s Jetson Nano supports both MIPI and CSI cameras.</p><p>Upon loading the trained model into a TensorFlow Graph, we retrieved one frame at a time from the camera and fed it into the Tensorflow Session which has the Graph assigned to it. The prediction results (if any), are overlayed into the frame to be displayed, along with a box surrounding the detected object.</p><p>(The code referenced in this section may be found at <a href="https://github.com/memillopeloz/JetsonNano-RPICam">https://github.com/memillopeloz/JetsonNano-RPICam</a>)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2Fthoroughinformaladder&amp;url=https%3A%2F%2Fgfycat.com%2Fthoroughinformaladder&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FThoroughInformalAdder-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="600" height="336" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef25bc4e4f1a9f13d485b6fd3113151c/href">https://medium.com/media/ef25bc4e4f1a9f13d485b6fd3113151c/href</a></iframe><p>We ran a Python’s profiler against the aforementioned script, namely <em>cProfile</em>, so that we could have a better insight regarding time consumption. For a more robust and representative profiling, we modified the object detection code to stop by itself after processing a fixed amount of frames, and pointed the camera towards movement-free areas while profiling executed; this way we could get profile results for several runs that truly reflected the impact of code changes.</p><p>The preliminary results are presented next, thanks to the visual aid provided by Python’s <em>pstats</em>. Results were narrowed down to display the top fifteen most time-consuming functions.</p><pre>Welcome to the profile statistics browser.<br>no_opt.prof% strip<br>no_opt.prof% sort time<br>no_opt.prof% stats 15</pre><pre>Mon Aug 19 20:35:30 2019    no_opt.prof</pre><pre>8754726 function calls (8402269 primitive calls) in 129.808 seconds</pre><pre>Ordered by: internal time<br>   List reduced from 8001 to 15 due to restriction &lt;15&gt;</pre><pre>ncalls     tottime  percall  cumtime  percall filename:lineno(function)<br>       50      87.562    1.751   87.562    1.751 {built-in method _pywrap_tensorflow_internal.TF_SessionRun_wrapper}<br>        1       7.520    7.520    7.520    7.520 {built-in method _pywrap_tensorflow_internal.TF_NewSessionRef}<br>        1       2.504    2.504  129.828  129.828 Object_detection_picamera.py:21(&lt;module&gt;)<br>118135/82788    2.038    0.000    4.334    0.000 python_message.py:491(init)<br>       50       1.338    0.027    1.338    0.027 {imshow}<br>     1932       0.946    0.000    0.946    0.000 {built-in method marshal.loads}<br>       50       0.935    0.019    0.935    0.019 {built-in method _pywrap_tensorflow_internal.ExtendSession}<br>101359/36073    0.913    0.000    4.513    0.000 python_message.py:1046(ByteSize)<br>    58/38       0.683    0.012    1.234    0.032 {built-in method _imp.create_dynamic}<br>        1       0.619    0.619    0.619    0.619 {built-in method _pywrap_tensorflow_internal.TF_GraphImportGraphDefWithResults}<br>46278/155       0.603    0.000    6.398    0.041 python_message.py:1144(InternalParse)<br>     7052       0.596    0.000    0.736    0.000 pywrap_tensorflow_internal.py:1410(__init__)<br>   128023       0.515    0.000    0.765    0.000 python_message.py:1485(__init__)<br>53019/36809     0.504    0.000    1.788    0.000 python_message.py:1299(MergeFrom)<br>577706/577638   0.500    0.000    0.555    0.000 {built-in method builtins.isinstance}</pre><pre>no_opt.prof% quit<br>Goodbye.</pre><p>Given the results, we could clearly determine TensorFlow’s Session to be the main factor in time consumption, and so we investigated any possible optimization that could be applied, most likely via parameter. We stumbled upon tf.OptimizerOptions, which may be associated to a Session’s configuration, for them to be applied when constructing the Tensor Graph.</p><p>The optimizations are stated in the code with tf.GraphOptions, tf.OptimizerOptions, and tf.ConfigProto as shown in the snippet below.<br>(More details on optimization options may be found at <a href="https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/core/protobuf/config.proto">https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/core/protobuf/config.proto</a>)</p><pre>graph_options = tf.GraphOptions(<br>                    optimizer_options=tf.OptimizerOptions(<br>                        opt_level=tf.OptimizerOptions.L1,<br>                    )<br>                )</pre><pre>OptConfig = tf.ConfigProto(<br>                graph_options=graph_options<br>            )</pre><pre>#################################################<br># Code for loading model into memory goes HERE  #<br># graph is assigned to variable detection_graph #<br>#################################################</pre><pre>TFSess = tf.Session(graph=detection_graph, config=OptConfig)</pre><p>We chose the optimization L1, given that it performs common subexpression elimination and constant folding, which aims to reuse precalculated results on certain recurrent tensor operations. With this in mind we estimated an improvement of at least 15% in overall execution time with the usage of L1 optimizations. The results obtained by profiling the optimized code are shown below.</p><p>We chose to display fewer functions this time:</p><pre>Welcome to the profile statistics browser.<br>with_opt.prof% strip<br>with_opt.prof% sort time<br>with_opt.prof% stats 10<br>Mon Aug 19 22:59:45 2019    with_opt.prof</pre><pre>8754857 function calls (8402401 primitive calls) in 114.830 seconds</pre><pre>Ordered by: internal time<br>   List reduced from 8019 to 10 due to restriction &lt;10&gt;</pre><pre>ncalls     tottime  percall  cumtime  percall filename:lineno(function)<br>       50      69.425    1.389   69.425    1.389 {built-in method _pywrap_tensorflow_internal.TF_SessionRun_wrapper}<br>        1       6.629    6.629    6.629    6.629 {built-in method _pywrap_tensorflow_internal.TF_NewSessionRef}<br>    58/38       2.409    0.042    3.239    0.085 {built-in method _imp.create_dynamic}<br>118139/82794    2.053    0.000    4.418    0.000 python_message.py:491(init)<br>     1934       1.787    0.001    1.787    0.001 {method &#39;read&#39; of &#39;_io.FileIO&#39; objects}<br>       50       1.532    0.031    1.532    0.031 {imshow}<br>        1       1.323    1.323  114.851  114.851 OPT_Object_detection_picamera.py:21(&lt;module&gt;)<br>     1932       0.971    0.001    0.971    0.001 {built-in method marshal.loads}<br>101361/36075    0.896    0.000    4.511    0.000 python_message.py:1046(ByteSize)<br>     9828       0.705    0.000    0.705    0.000 {built-in method posix.stat}</pre><pre>with_opt.prof% quit<br>Goodbye.</pre><p>Eureka! We indeed managed to bring the execution time down from 129.8 sec to 114.8 sec, which represents an improvement of ~11.55% overall. It was not as high as initially expected, but it did pump up performance. Constant folding optimization replaces tensors whose values can be predetermined with constant nodes, this was probably the main reason behind the improvement shown.</p><h3>Conclusions</h3><p>Creating our own custom object detection neural network was quite fun and challenging. We had to take into account the limitations of the hardware where the model was set to run (Jetson Nano), and modify the models accordingly, so as to maximize performance.</p><p>Setting up the appropriate environment is crucial for a smooth execution both of the training process, and the object detection itself. Most of the problems encountered were because of a wrong/incomplete setup.</p><p>Great computer resources are required for the training phase, specially for highy complex models like Google’s Inception, or MobileNet. Moreover, training time could be reduced further, and accuracy heightened by utilizing specialized high-end equipment.</p><p>MobileNet proved to be the most efficient architecture for resource-limited mobile or embedded devices/modules. This was mostly because of memory limitations.</p><h3>Demo</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmkYTcMl5VY0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmkYTcMl5VY0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmkYTcMl5VY0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/36345a6b02244b1d0b4515f30139067a/href">https://medium.com/media/36345a6b02244b1d0b4515f30139067a/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=113fe4dba134" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/nvidia-jetson-nano-custom-object-detection-from-scratch-using-tensorflow-and-opencv-113fe4dba134">Nvidia Jetson Nano: Custom Object Detection from scratch using Tensorflow and OpenCV</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>