<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lila Mullany on Medium]]></title>
        <description><![CDATA[Stories by Lila Mullany on Medium]]></description>
        <link>https://medium.com/@lilamullany?source=rss-d55d0a812c15------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Pd3MAFy3UXrIzHCacMQuNg.jpeg</url>
            <title>Stories by Lila Mullany on Medium</title>
            <link>https://medium.com/@lilamullany?source=rss-d55d0a812c15------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:18:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lilamullany/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Use Computer Vision to Capture Real-World Events]]></title>
            <link>https://medium.com/data-science/use-computer-vision-to-capture-real-world-events-57dc890e82d5?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/57dc890e82d5</guid>
            <category><![CDATA[data-for-change]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Tue, 11 May 2021 06:00:59 GMT</pubDate>
            <atom:updated>2021-05-17T18:48:22.047Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/data-for-change">Data for Change</a></h4><h4>Leveraging computer vision to assist with COVID vaccine distribution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FjiEW3V-S6C4ZLNDl5LGA.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@kevin-ku-92347?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Kevin Ku</a> from <a href="https://www.pexels.com/photo/black-farmed-eyeglasses-in-front-of-laptop-computer-577585/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a>.</figcaption></figure><p>In this blog post, I’ll describe the general approach to using computer vision to capture different events, focusing on how computer vision was used to make a series of applications to help with vaccine distribution. This blog post is a longer one, but I cover a few different scenarios that should be helpful in many situations. If you want to just check out the code, it can be found <a href="https://github.com/alwaysai/vaccination-center-assistant">here</a>. This article will also be updated if improvements to the application are made!</p><p>The COVID-19 pandemic has impacted our lives for over a year in numerous ways and has presented us with a multitude of challenges. With vaccines now available, we have the ability to expedite the end of the pandemic, as long as the vaccine can be distributed safely and efficiently. Any effort that can assist in this process should be taken. Enter computer vision!</p><p>It is always best to develop an application with a specific use case and environment in mind. In order to determine how computer vision might help in the vaccination process, our team consulted a volunteer at a vaccination center to learn more about the process and find opportunities for computer vision to be applied. If you want to see a recording of this discussion and a demo of the application, you can watch it on <a href="https://youtu.be/ROOFvzPfR3I">YouTube</a>.</p><p>At the volunteer’s vaccination center, there were three main phases: waiting for your vaccine (and/or waiting to register), getting your vaccination, and post-vaccination monitoring. The main issues our volunteer identified that we thought could easily be helped with computer vision were: reducing prolonged wait times, helping individuals practice social distancing, and monitoring the post-vaccination room. It would also be useful to have a means of verifying how many vaccinations had taken place, as the current method was a manual clicker counter.</p><p>Some other obstacles to the vaccination process that were described, such as hard-to-navigate websites, are not suitable to be solved with computer vision. Other challenges, such as redundant form entering, could potentially be solved with a combination of computer vision solutions for license verification combined with a more efficient registration process online!</p><p>This solution uses three modular applications, which can be used individually or combined with a dashboard to help solve the issues outlined above. The first application monitors the waiting room, noting the distance between individuals, the proportion of individuals wearing masks, and how many chairs are unoccupied in a waiting room. The second application monitors how many vaccination events had taken place. The third application monitors when a patient raises their hand, indicating they need assistance from a volunteer. With these solutions integrated, and some general input data about the type of vaccine being used and the number of vaccination appointments that day, we can also help provide a means of verification for how many vaccine vials have been opened, how many doses are left in an opened vial, and whether the last appointment of the day has been reached. This data can help manage the wait times, by automatically texting upcoming appointments if the line is moving slower or faster than expected, as well as dialing the waiting list if there are vaccine doses remaining in an opened vial at the end of the day.</p><p>Because these applications were not being developed for a real site, just as close to the real scenario as possible, stock video data was used, and this data is what drove the implementation of the applications. If you build your own vaccination application, the exact app will most likely be very different; either the process will be slightly different, or the camera angles will be different.</p><h3>The Waiting Room Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PTFMCJUan0FiDiWLOMTn0A.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@paula-schmidt-353488?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Paula Schmidt</a> from <a href="https://www.pexels.com/photo/green-wooden-chair-on-white-surface-963486/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a>.</figcaption></figure><p><a href="https://github.com/alwaysai/vaccination-center-assistant/tree/main/waiting">The waiting room app</a> assists in analyzing social distancing and mask detection, as well as monitoring when waiting room chairs are unoccupied. People who are at least 6 feet (approximately) away from every other person and are wearing a mask had green bounding boxes; if someone is less than 6 feet away from anyone, or is not wearing a mask, red boxes were displayed.</p><p>To determine how full a waiting room is, we detect how many people are in chairs in the waiting room. This is done using a freely available stock alwaysAI model, yolov3, for object detection. The <a href="https://alwaysai.co/docs/edgeiq_api/tools.html#edgeiq.edge_tools.filter_predictions_by_label">results filtering function</a> is used to limit results to those with the label ‘person’ to ensure that only predictions for people are being detected. This is accomplished by creating a <a href="https://alwaysai.co/docs/edgeiq_api/bounding_box.html#edgeiq.bounding_box.BoundingBox">bounding box</a> around each chair and noting when a person’s <a href="https://alwaysai.co/docs/edgeiq_api/bounding_box.html#edgeiq.bounding_box.BoundingBox.compute_overlap">bounding box overlaps</a> with the closest chair bounding box over a certain threshold — say around 70%. This threshold is used because some people lean forward in a chair, or stick their foot out to the side, and so using a threshold &lt; 100% allows for a more generalizable way of detecting that a person is sitting in a chair. Reports about how many masks were detected, the average distance or distance between each pair of people, and how many chairs were occupied are sent to the server (see the last section).</p><p>The first step in developing this application was performing mask detection.</p><p>Todd Gleed, Product Manager at alwaysAI, trained a new model, using labels ‘mask’ (representing any face wearing a face covering) and ‘no-mask’ (which represented a face without any covering). By <a href="https://alwaysai.co/docs/alwaysai_workflow/working_with_projects.html#adding-collaborators-to-your-project">sharing the model on a project</a> with me, I was able to test out different versions. Testing the model in the application, it worked sporadically, but had difficulty on the stock video data, which had smaller instances of masks (the people were farther away from the camera than they were in the training data). The stock data also happened to only use masks that had more muted, neutral colors. I also noticed that when patients sat directly across from the check-in staff, the model could not always make any predictions at all (either as ‘mask’ or as ‘no-mask’); so images from that angle were likely not well represented in the dataset. We certainly could have trained the model more, using data that more closely matched the use case, however instead I modified the application code.</p><p>Because it is useful to know which people are wearing masks, either the bounding boxes of the people had to be compared to the bounding boxes of the mask detections, or the image corresponding to the person detection boxes could simply be used as input to the mask detection model, which is what I chose to do.</p><p>For each ‘person’ detection box, the portion of the original frame that corresponded to the person <a href="https://alwaysai.co/docs/edgeiq_api/tools.html#edgeiq.edge_tools.cutout_image">is cut out</a> and this image is run through the mask detection model. A similar method was used in <a href="https://github.com/alwaysai/gesture-audio-control">this application</a>. Additionally, if the model is not able to make any predictions, we gain the ability to say ‘there is a person here, but I can’t tell if they are wearing a mask’; this is useful if someone has their back turned to the camera, or mask detection is difficult for another reason. If a person is detected, but it isn’t possible to determine if they were wearing a mask, the person’s bounding box is marked up in red and is given a label of ‘no mask detected’. The application still had trouble with the angle directly across from the check-in staff.</p><p>This portion of the application did use <a href="https://alwaysai.co/docs/edgeiq_api/object_tracking.html#edgeiq.object_tracking.CentroidTracker">tracking</a>, however, it probably wasn’t necessary. Tracking, in this case, is useful for seeing how patients moved from chair to chair, but for the waiting room, we probably only care if all chairs are full, or noting how many chairs are empty, etc. This can be done just with object detection. Any time someone sits in a chair or leaves a chair, this information is sent to the dashboard application.</p><p>The distancing in this application is done simply by using a hard-coded standard measurement for a person, about three and a half feet, and using a pixel scale, such as is used in <a href="https://github.com/alwaysai/distance-detector">this sample application</a>. If you want to see a demo and explanation of this logic, you can check out this <a href="https://youtu.be/y5Kw3GN-u3s">video tutorial on distance measurement</a> on YouTube. Three and a half feet is used in this case because people are generally sitting down in the stock data that was used to develop the app. Using a hard-coded scale was possible because there was a single static camera, and whenever a person was detected and sitting, the entire person was in the frame. Depending on the camera angle and environment, more robust methods of distance measuring may be needed in your application, such as using other objects with known dimensions in the frame or checking for key points to see if a whole person is found, or using a different scale if just a partial person is found. You can also use a <a href="https://alwaysai.co/docs/edgeiq_api/real_sense.html">depth camera</a> for more precise measurements.</p><h3>The Vaccination Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Q8EldftNVg4-qipRtWZig.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@gustavo-fring?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Gustavo Fring</a> from <a href="https://www.pexels.com/photo/person-getting-vaccinated-3985170/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a>.</figcaption></figure><p>For the <a href="https://github.com/alwaysai/vaccination-center-assistant/tree/main/vaccination">vaccination application</a>, the scenario I modeled was a static view of a single table. The medical professional sits at the table, and periodically, a patient comes and sits down at the table. After some time, the medical professional walks around the table, stands next to the patient, and gives them their vaccination. So, a ‘vaccination event’ can be captured by noting when two people enter a portion of an image, noting when they leave, and then determining that they stayed in that region for an acceptable duration of time based on those time stamps. Every time a vaccination event occurs, a POST request is sent to the dashboard server.</p><p>This implementation just requires <a href="https://alwaysai.co/docs/edgeiq_api/object_tracking.html#edgeiq.object_tracking.CentroidTracker">tracking</a> and noting that the tracked bounding boxes are within a specific portion of an image for a set amount of time, exactly the same way as described in the section above. Depending on the camera angle, and the number of cameras per vaccination station, you may need to take an approach similar to the waiting room chairs, or maybe you need to use key points to note when someone is sitting or standing.</p><h3>The Post-Vaccination Monitoring Application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_cutcz8uwqtooxLwkV6Sg.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@rodnae-prod?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">RODNAE Productions</a> from <a href="https://www.pexels.com/photo/people-sitting-on-green-grass-waving-their-hands-7551760/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a>.</figcaption></figure><p>After individuals receive their vaccine, they are typically put into a large room for about 15 minutes, where they can sit and can notify healthcare professionals via a hand raise if they are feeling unwell. The third application is intended to assist in this <a href="https://github.com/alwaysai/vaccination-center-assistant/tree/main/post-vaccination">post-vaccination monitoring</a>. This is a pretty simple concept — capturing a hand raise. Something similar was done in this <a href="https://github.com/alwaysai/posture-corrector">application</a>. In this case, instead of defining what constitutes poor posture, I just captured different notions of a ‘hand raise’. In the stock video used for this application, a hand raise could be captured by noting when a person’s elbow was higher than the corresponding shoulder (note: this means the y-coordinate of the elbow is less than the y-coordinate of the corresponding shoulder).</p><p>This is an overly simplistic way to capture a hand raise, though. Look at all the individuals in the image at the top of this section, or think about how your arm looks in relation to your head, shoulders, and other key points when you raise your arm. Maybe your friend raises their arm a little differently — maybe your arm goes straight up, so your wrist is right above your elbow, which is right above your shoulder. Maybe your friend just brings their hand up by their face, so their wrist is over their shoulder, but their elbow is off to the side and in between the wrist and shoulder in terms of vertical placement. Maybe you can’t see a wrist, or an elbow, or a shoulder, in the camera view. You can test with real-world hand raises, and see what key point relationships reflect the most common hand raises given the angle of the camera!</p><p>To prevent capturing when someone just brushes the hair off of their face or maybe adjusts their glasses, motions that might result in very similar key point relationships to hand raises, we can use a ‘consensus’ logic in our application. This starts a timer when a hand raise is detected, and then counts the number of times the hand raise is detected within a certain timeout period. The number of total results is divided by the current number of people, and then this is compared to the total times a ‘hard raise’ was detected. For instance, if we run the loop 10 times, and one person is raising their hand, we should have 10 ‘hand raise’ results and 20 ‘no hand raise’ results. As long as the number of ‘hand raise’ events is greater than or equal to the total events divided by the number of people, we log it as a true hand raise. Every time a hand was raised, this was sent to the dashboard server. A similar approach was used in our <a href="https://github.com/alwaysai/gesture-audio-control">gesture-control application</a>. You can also watch the <a href="https://youtu.be/kxU761yA0TA">YouTube</a> recording on this topic!</p><h3>Bringing it all Together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GgKOkprmvYhsAABsTZQZ1A.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@mike-van-schoonderwalt-1884800?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Mike van Schoonderwalt</a> from <a href="https://www.pexels.com/photo/person-holding-white-puzzle-piece-5505150/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a>.</figcaption></figure><p>In my original implementation, three applications were tied together using a dashboard generated with <a href="https://dash.plotly.com/">Dash Plotly</a>. Because alwaysAI is Python-based, you can integrate these applications with existing Python packages. This <a href="https://github.com/alwaysai/dash-interactive-streamer">GitHub</a> repository provides a basic Dash dashboard as an alwaysAI application; you can send all of your data (using the ‘server_url’ attribute in each of the individual applications) to this dashboard, process the event log, and use this data to populate various graphs and tables in your dashboard! Note that in the individual applications, we sent data to a ‘route’ as well as the URL. In order to process the incoming data, you will need to create the routes you sent data to in the individual applications in dashboard_server/app.py, like so:</p><pre>@app.route(&quot;/event&quot;, methods=[&quot;POST&quot;])<br>def event():<br>    body = request.json<br>    # process body further, send to managing object</pre><p>You can make additional routes (like for<strong> ‘/setup’</strong>, or you can have different endpoints for each of your individual applications). It may be useful to make a specific class to parse and manage the incoming data. So, you can instantiate one ‘manager’ object (an instance of a class that you can define yourself) inside <strong>dashboard_server/app.py</strong> and then pass all the data to this object and process it. For instance, the Flask route could look like this:</p><pre>manager = DataManager()</pre><pre>@app.route(&quot;/event&quot;, methods=[&quot;POST&quot;])<br>def event():<br>    body = request.json<br>    manager.process_data(body)</pre><p>In fact, if you just want to aggregate data and send out alerts, or write data files based on the results, you could do this without Dash visualizations — just use these routes to process the incoming data and have some application logic use the data management instance to determine what action to take.</p><p>If you are using Dash, the callbacks in your <strong>dashboard_server/app.py</strong> can invoke methods on this managing object to get data to populate your visualizations. For instance, the callback in the above GitHub repository could be changed to look like this:</p><pre># Dash Callbacks<br>@dash_app.callback(<br>   output=[Output(&quot;logs&quot;, &quot;data&quot;), Output(&quot;logs&quot;, &quot;columns&quot;)],<br>   inputs=[Input(&#39;interval-component&#39;, &#39;n_intervals&#39;)]<br>)<br>def render_log_table(n_intervals):<br>   df = manager.get_log_table_df() <br>   return df.to_dict(&#39;records&#39;), [{&quot;name&quot;: i, &quot;id&quot;: i} for i in df.columns]</pre><p>Inside the ‘DataManager’ class you would define the function <strong>get_log_table_df() </strong>and implement it to return the proper data.</p><p>It may be useful to store reformatted data in <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html">pandas DataFrames</a>, as many components in Dash take a <a href="https://dash.plotly.com/layout">DataFrame as input</a>.</p><h3>Final Thoughts</h3><p>While all of the other applications are set up to run using the alwaysAI <a href="https://alwaysai.co/docs/edgeiq_api/streamer.html">streamer</a> functionality (a local webpage for viewing application output), this is mainly useful to show how the applications are working. If you need more control over the streamer, you can build your own; you can find various examples of that <a href="https://github.com/alwaysai?q=video-streamer&amp;type=&amp;language=&amp;sort=">here</a>. You can also use this template for your central server if you don’t need to incorporate visualizations. You can also run these applications without a streamer, and simply send requests to your server with information generated using computer vision services. If you want to use the streamer, but you want to hide faces, you can always use the face detection model (such as <a href="https://console.alwaysai.co/model-catalog?model=alwaysai/res10_300x300_ssd_iter_140000#">alwaysai/res10_300x300_ssd_iter_140000</a>) and then use the returned predictions from the results in a call to <a href="https://alwaysai.co/docs/edgeiq_api/tools.html#edgeiq.edge_tools.blur_objects">edgeiq.blur_objects()</a>. Of course, if you were ever developing a production application using health-related data, you would need to adhere to HIPAA and any and all requirements of the health organization.</p><p>The vaccination center use case will hopefully not be needed for much longer, but you can use computer vision to capture lots of different events!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57dc890e82d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/use-computer-vision-to-capture-real-world-events-57dc890e82d5">Use Computer Vision to Capture Real-World Events</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn Computer Vision Using the Python Interpreter]]></title>
            <link>https://medium.com/data-science/learn-computer-vision-using-the-python-interpreter-8cfb79041adb?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cfb79041adb</guid>
            <category><![CDATA[model]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Mon, 11 Jan 2021 23:16:11 GMT</pubDate>
            <atom:updated>2021-01-11T23:16:11.588Z</atom:updated>
            <content:encoded><![CDATA[<h4>Play with Semantic Segmentation in the Terminal</h4><figure><img alt="woman smiling with background replaced by mountains" src="https://cdn-images-1.medium.com/max/640/0*NPR7XPN1V3F0gLZ2.png" /><figcaption>Image of smiling woman with original background replaced with a mountains using semantic segmentation. Original image: Photo by <a href="https://unsplash.com/@michaeldam?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Michael Dam</a> on <a href="https://unsplash.com/s/photos/people?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>If you’re like me, you like to open up the Python interpreter and try out functions or working with new data structures in the terminal as you’re building your applications. For built-in modules, this is pretty easy, but for external packages, this can get a little more complicated, especially for working with computer vision model output! I made this tutorial to help provide a foundation for working with semantic segmentation in computer vision applications. And you can use the same general technique to work with other computer vision model output: object detection, key points, and image classification.</p><p><em>Note: You can find freely available images on </em><a href="https://unsplash.com/"><em>Unsplash</em></a><em>, where the image of the woman is from and </em><a href="https://www.pexels.com/"><em>Pexels</em></a><em>, where the mountain picture (shown below) is from. All other images in this tutorial were generated by myself, using image segmentation on those original images.</em></p><h3>Some Background</h3><p>This tutorial uses a set of free computer vision tools provided by <a href="https://alwaysai.co/?utm_campaign=Website%20Traffic&amp;utm_source=medium&amp;utm_medium=medium&amp;utm_content=learn_computer_vision_terminal">alwaysAI</a>, which is a Python-based platform for building and running computer vision applications, and with which I am affiliated. You can still reference the more general NumPy manipulations in this article if you don’t want to follow along using the tooling!</p><p>alwaysAI has written a Python <a href="https://alwaysai.co/docs/edgeiq_api/index.html?highlight=apis?utm_campaign=Website%20Traffic&amp;utm_source=medium&amp;utm_medium=medium&amp;utm_content=learn_computer_vision_terminal">API</a>, called edgeiq, to make working with computer vision model output a more standardized and simple process, and includes a number of popular packages, so you don’t need to install them individually. We’ll cover how to interact with the edgeiq and other useful computer vision modules, like OpenCV’s cv2, from the command line. This is useful for stepping through different manipulations of the segmentation masks and viewing the output.</p><p>This blog post was made after demoing some semantic segmentation applications in the YouTube video embedded below, in order to provide a better foundation before jumping in to more general applications. This tutorial will show how to replace parts of an image with semantic segmentation, step by step.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5mW_zlItoPw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5mW_zlItoPw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5mW_zlItoPw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d951377c1019bfac72dba2e5f472cf95/href">https://medium.com/media/d951377c1019bfac72dba2e5f472cf95/href</a></iframe><p>The video walks through a couple semantic segmentation applications, and the full code for both can be found on GitHub:</p><ol><li><a href="https://github.com/alwaysai/virtual-green-screen">The original virtual green screen repository</a></li><li><a href="https://github.com/alwaysai/smooth-semantic-segmentation">An application that smooths the edges returned by running the semantic segmentation</a></li></ol><p>These repositories show you how to use <a href="https://nanonets.com/blog/semantic-image-segmentation-2020/">semantic segmentation</a> results to replace pixels in one image that correspond to labels in another image and how you can create smoother edges using numpy array manipulation and other techniques.</p><h3>Set Up</h3><p>Note, you’ll have to have alwaysAI installed on your local machine to use the API and work through this tutorial. You can download alwaysAI for free from the alwaysAI <a href="https://alwaysai.co/dashboard?utm_campaign=Website%20Traffic&amp;utm_source=medium&amp;utm_medium=medium&amp;utm_content=learn_computer_vision_terminal">dashboard</a>. Or, just follow the NumPy manipulations if you prefer!</p><p>After you’ve installed alwaysAI, let’s set up your environment. You’ll need to configure and install any application that uses a semantic segmentation model; so using either of the above repositories, or any of the starter apps that use semantic segmentation will do. You can get the starter apps by typing aai get-starter-apps from within the terminal; this command will download the starter apps into your current working directory. You can then use ls to see all the starter app, and then use cd to navigate into either the ‘semantic_segmentation_cityscape’ directory, or the ‘semantic_segmentation_voc’ directory.</p><p>Choose your application, and from within the working directory run</p><pre>aai app configure</pre><p>You should select your local machine for the directory. Next, run</p><pre>aai app install</pre><p>This command builds the <a href="https://towardsdatascience.com/virtual-environments-104c62d48c54">virtual environment</a>, which contains the alwaysAI API, edgeiq, as well as the popular open source computer vision library from OpenCV, cv2. If you want to use any other packages, just add them to a requirements.txt file in the app directory and re-run aai app install.</p><p>Next, activate the python virtual environment, which will have all the required packages installed, by running</p><pre>source venv/bin/activate</pre><p>Finally, start up your python terminal by running</p><pre>python3</pre><p>You should see the terminal change to the python interpreter, which has the prompt</p><pre>&gt;&gt;&gt;</pre><p><em>Note: When you’re done, you can exit out of Python by typing </em><em>exit() and then get out of the virtual environment by typing </em><em>deactivate in the terminal.</em></p><p>Now you’re all set to work through this tutorial!</p><h3>Notes</h3><ul><li>You can always check the type of an output in python using type(variable)</li><li>You can try to view images with the following code, although I have issues with this from within the virtual environment and python interpreter:</li></ul><pre>def display(image):<br>    cv2.imshow(&quot;image&quot;, image)<br>    cv2.waitKey(0)<br>    cv2.destroyAllWindows()</pre><p>This will create a function you can call using display(image) to display any image. I’ve had trouble closing the window, but you can press any key then and be back in the terminal, although the display window may remain up.</p><ul><li>Another solution is to just write out images like so:</li></ul><pre>cv2.imwrite(&quot;test1.png&quot;, image)</pre><p>Any time you want to see the output of a step, I suggest writing an image out, which is what this tutorial will do.</p><h3>Tutorial</h3><p>I’ll splice some images and text in with the code that follows, but you can still copy paste the code text into your terminal to work through this tutorial. Most of the following code is taken from the <a href="https://github.com/alwaysai/virtual-green-screen">virtual green screen repository</a>.</p><p>First, we need to import the necessary libraries.</p><pre>import edgeiq<br>import cv2<br>import numpy as np<br>import sys</pre><p>The alwaysAI API, edgeiq, is what we use for semantic segmentation. We’ll use OpenCV’s library, cv2, for writing images in and out. Numpy is used for some array manipulation, and sys is used for changing the display format of the numpy arrays.</p><pre># use this if you want to see the entire numpy array (not very user <br># friendly, but can be helpful to seeing what labels are used)<br>np.set_printoptions(threshold=sys.maxsize)</pre><p>You’ll need an image in your working directory, I called mine ‘output.png’. I just got an image off of <a href="https://unsplash.com/">Unsplash</a>.</p><figure><img alt="Woman in red shirt and red lipstick smiling" src="https://cdn-images-1.medium.com/max/640/0*jCrky5U4xf5v63c0.jpg" /><figcaption>Photo by <a href="https://unsplash.com/@michaeldam?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Michael Dam</a> on <a href="https://unsplash.com/s/photos/people?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The following code reads in an image, then creates an instance of a SemanticSegmentation object using edgeiq, and specifies which inference engine to run input on. In the third block, it gets segmentation results from the input image.</p><pre># read in the image<br>image = cv2.imread(&quot;output.jpg&quot;)</pre><pre># instantiate a semantic segmentation object and load the engine<br>semantic_segmentation = edgeiq.SemanticSegmentation(&quot;alwaysai/fcn_alexnet_pascal_voc&quot;)<br>semantic_segmentation.load(engine=edgeiq.Engine.DNN)</pre><pre># run segmentation and get the results<br>segmentation_results = semantic_segmentation.segment_image(image)</pre><pre># to view the class_map of the resulting segmentation_results object:<br>segmentation_results.class_map</pre><p>The last line of code in the block above yeilds the following output (run before re-setting the display options, for the purposes of this tutorial):</p><p>array([[ 0, 0, 0, …, 0, 0, 0],<br> [ 0, 0, 0, …, 0, 0, 0],<br> [ 0, 0, 0, …, 0, 0, 0],<br> …,<br> [ 0, 0, 0, …, 11, 11, 11],<br> [ 0, 0, 0, …, 11, 11, 11],<br> [ 0, 0, 0, …, 11, 11, 11]], dtype=int32)</p><p>We can also write out an what this class map looks like, using:</p><pre># to write out an image of the class map (note, the class_map is a numpy array!)<br>cv2.imwrite(&quot;class_map.png&quot;, segmentation_results.class_map)</pre><figure><img alt="An image of a semantic segmentation mask, grey on black in shape of woman’s boundaries." src="https://cdn-images-1.medium.com/max/640/0*F03AkxW1XXOfycwl.png" /><figcaption>Semantic segmentation mask generated using original image of smiling woman.</figcaption></figure><p>Note, this is a bit hard to see, but there will be another example further along in the tutorial that will be a bit more obvious.</p><pre># get a label map, which is the same dimensions as the class map but # uses labels<br>label_map = np.array(semantic_segmentation.labels)[segmentation_results.class_map]</pre><p>The above code yields the following output:</p><p>array([[‘background’, ‘background’, ‘background’, …, ‘background’,</p><p>‘background’, ‘background’],</p><p>[‘background’, ‘background’, ‘background’, …, ‘background’,</p><p>‘background’, ‘background’],</p><p>[‘background’, ‘background’, ‘background’, …, ‘background’,</p><p>‘background’, ‘background’],</p><p>…,</p><p>[‘background’, ‘background’, ‘background’, …, ‘diningtable’,</p><p>‘diningtable’, ‘diningtable’],</p><p>[‘background’, ‘background’, ‘background’, …, ‘diningtable’,</p><p>‘diningtable’, ‘diningtable’],</p><p>[‘background’, ‘background’, ‘background’, …, ‘diningtable’,</p><p>‘diningtable’, ‘diningtable’]], dtype=’&lt;U11&#39;)</p><p>Now we’ll start building a mask that just has our desired labels in it.</p><pre># First just make a numpy array that is the same shape as the class # map but just zeros<br>filtered_class_map = np.zeros(segmentation_results.class_map.shape).astype(int)</pre><pre># print the filtered class map<br>filtered_class_map</pre><p>This yields the following:</p><p>array([[0, 0, 0, …, 0, 0, 0],</p><p>[0, 0, 0, …, 0, 0, 0],</p><p>[0, 0, 0, …, 0, 0, 0],</p><p>…,</p><p>[0, 0, 0, …, 0, 0, 0],</p><p>[0, 0, 0, …, 0, 0, 0],</p><p>[0, 0, 0, …, 0, 0, 0]])</p><p>And, just to show how the ‘type’ command works, we can check the type of the output like so:</p><pre>type(filtered_class_map)</pre><p>Which yields:</p><p>&lt;class ‘numpy.ndarray’&gt;</p><p>And here is what the current filtered_class_map looks like (don’t strain your eyes, it’s just all black for now!):</p><figure><img alt="A black screen" src="https://cdn-images-1.medium.com/max/640/0*6YBsbJzCmQdRYpq_.png" /></figure><pre># you can add labels here if you want to pick up more labels, <br># check out what you get from the <br># label_map print out to see what is being picked up <br># around the person<br>labels_to_mask = [&#39;person&#39;]</pre><pre># now toggle the cells that mapped to ‘person’<br>for label in labels_to_mask:<br>    filtered_class_map += segmentation_results.class_map * (label_map == label).astype(int)</pre><pre># Print out the map<br>filtered_class_map</pre><pre># Now, If we write out the image<br>cv2.imwrite(“new_filtered_class_map.png”, filtered_class_map)</pre><p>Again, this output is dim, but it is slightly different than the class_map output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*F0aM1HAxBrzvHkGc.png" /></figure><p>You can also check out the sizes of all the maps we’ve used:</p><pre>image.shape<br>label_map.shape<br>filtered_class_map.shape</pre><p>The above code yields (960, 640, 3), (960, 640), (960, 640) respectively for my input image’s results. Note they’re all the same size! The ‘3’ in the first tuple denotes a depth of three, which is used for the BGR (or RGB) color channels.</p><pre># Now we can make a map to pull out just the <br># labels of interest (i.e. people)<br>detection_map = (filtered_class_map != 0)</pre><p>Example output:</p><p>array([[False, False, False, …, False, False, False],</p><p>[False, False, False, …, False, False, False],</p><p>[False, False, False, …, False, False, False],</p><p>…,</p><p>[False, False, False, …, False, False, False],</p><p>[False, False, False, …, False, False, False],</p><p>[False, False, False, …, False, False, False]])</p><p>We can also check out the shape of the detection_map by running</p><pre>detection_map.shape</pre><p>Example output:</p><p>(960, 640)</p><p>Now we’ll choose a background image. Note that you can find some examples of images in the GitHub repositories! Download one of those and specify whatever name you choose in the example command below.</p><pre># Read in an image for your background. I found a picture of a mountain on Unsplash.<br>background = cv2.imread(&quot;mountain_pic.jpg&quot;)</pre><figure><img alt="picture of a mountain with trees and clouds" src="https://cdn-images-1.medium.com/max/1024/0*IpTkjPW2lj2V0aOM.jpg" /><figcaption>We’ll use this image of a mountain for our new background! Photo by <a href="https://www.pexels.com/@punttim?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Tim Gouw</a> from <a href="https://www.pexels.com/photo/panoramic-photography-of-alps-291732/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>Whenever we use a new image, it must be the same shape! So we can resize it with</p><pre>shape = image.shape[:2]<br>background = edgeiq.resize(background, shape[1], shape[0], keep_scale=False)</pre><p>Note that since the background is horizontal and the input image is vertical, we’ll have to use ‘keep_scale=False’, which will warp the image a bit. You could also choose a vertical image for your background, or keep it as is if your streamer/input image will be horizontal.</p><p>Finally, we replace the area in the background that corresponds to the ‘True’ sections in the detection map with the original image</p><pre>background[detection_map] = image[detection_map].copy()</pre><pre># And finally write out the new image!<br>cv2.imwrite(&quot;final_background.png&quot;, background)</pre><figure><img alt="Image of woman, with background replaced by mountains. Very large pixel borders." src="https://cdn-images-1.medium.com/max/640/0*eNC7YujjQYSaWq3B.png" /><figcaption>The original image with ‘person’ labels kept, and all others replaced by the mountain picture.</figcaption></figure><p>Now, just the portion of the original image that was labeled ‘person’ is used to replace the corresponding section in the background. This is a (rough) virtual green screen example!</p><p>We’ll see some more examples of segmentation masks, including some blurring, next.</p><p>Let’s change label colors to make the mask easier to view.</p><pre># iterate over all the desired items to identify, <br># labeling those white<br>for label in labels_to_mask:<br>    index = semantic_segmentation.labels.index(label)<br>    semantic_segmentation.colors[index] = (255,255,255)</pre><pre># build a new mask, and write a new image<br>new_mask = semantic_segmentation.build_image_mask(segmentation_results.class_map)<br>cv2.imwrite(&quot;new_mask.png&quot;, new_mask)</pre><p>This is the output:</p><figure><img alt="Label map with person in white, other label in red, and background in black." src="https://cdn-images-1.medium.com/max/640/0*On22278Gl2O-Wczy.png" /><figcaption>Now, the pixels corresponding to ‘person’ are in white.</figcaption></figure><p>Now that we have this mask that is easier to see, let’s do some image manipulations. If we want to blur the background, we could do the following:</p><pre># Create a blurred mask<br>blurred_mask = cv2.blur(new_mask, (100, 100))<br>cv2.imwrite(&quot;blurred.png&quot;, blurred_mask)</pre><p>This create the following image:</p><figure><img alt="The same image as previously show, but with edges blurred together." src="https://cdn-images-1.medium.com/max/640/0*KVx0TRuOGIRCl3CR.png" /><figcaption>The same image as shown previously, but with edges blurred.</figcaption></figure><p>If we want to make ONLY the labels we want white, and ALL others black, execute the following code and re-run the for loop above:</p><pre># this makes all the colors in the label map black<br>semantic_segmentation.colors = [ (0,0,0) for i in semantic_segmentation.colors]</pre><p>After running the for loop, re-building the mask and writing out the output, the result is:</p><figure><img alt="Just person labels in white, and all other labels in black." src="https://cdn-images-1.medium.com/max/640/0*R7_IAoK4A6eHSbUt.png" /><figcaption>Now, just person pixels are white, and all other labels are black.</figcaption></figure><p>Now that we have an idea of how the masks are used and manipulated, let’s use these techniques to smooth out those edges and create a more visually-appealing virtual green screen. The next section of the tutorial uses code from the following repository: <a href="https://github.com/alwaysai/smooth-semantic-segmentation">https://github.com/alwaysai/smooth-semantic-segmentation</a>.</p><p>The image we’re working with is the same as the one shown just above. However, if you want to make sure you have the proper set up, you can re-run the following code:</p><pre># read in the initial image again<br>image = cv2.imread(&quot;output.jpg&quot;)</pre><pre># build the color mask, making all colors the same except for background<br>semantic_segmentation.colors = [ (0,0,0) for i in semantic_segmentation.colors]</pre><pre># iterate over all the desired items to identify, labeling those white<br>for label in labels_to_mask:<br>   index = semantic_segmentation.labels.index(label)<br>   semantic_segmentation.colors[index] = (255,255,255)</pre><pre># build the color mask<br>mask = semantic_segmentation.build_image_mask(results.class_map)</pre><p>First, we’ll start off with enlarging the dilation mask (line 95 in the smoothing edges repo).</p><pre>dilatation_size = 15<br>dilatation_type = cv2.MORPH_CROSS<br>element = cv.getStructuringElement(<br>                        dilatation_type,<br>                        (2*dilatation_size + 1, 2*dilatation_size+1),<br>                        (dilatation_size, dilatation_size))<br>mask = cv2.dilate(mask, element)</pre><p>Now the mask looks like:</p><figure><img alt="Edges around the person are more gradual, and less blocky." src="https://cdn-images-1.medium.com/max/640/0*4MC_jNxe_Iksg5Yr.png" /><figcaption>Now the image has had a dilation applied to the person mask.</figcaption></figure><p>Next, we’ll apply smoothing to the mask:</p><pre>mask = cv2.blur(mask, (50, 50))</pre><p>Now the mask looks like this:</p><figure><img alt="The previous boundaries are now blurred." src="https://cdn-images-1.medium.com/max/640/0*db7Opj1RWSvKNUmb.png" /><figcaption>Smoothing (blurring) applied to the mask.</figcaption></figure><p>Next, we’ll set up the background image. To be safe, we’ll just re-import it and run the resizing code.</p><pre># Also import the background image again<br>background = cv.imread(&quot;mountain_pic.jpg&quot;)<br>background = edgeiq.resize(background, image.shape[1],<br>image.shape[0], keep_scale=False)</pre><p>Finally, we’ll use the ‘overlay’ function built in to edgeiq. This will use the blurred mask, the foreground image and the background image and interweave them to give nice, smoothed edges.</p><pre># Now overlay the background and foreground using the overlay function<br>frame = edgeiq.overlay_image(frame, background, mask)<br>cv.imwrite(&quot;overlayed_frame.png&quot;, frame)</pre><p>This is the resulting output:</p><figure><img alt="Final image, with original woman shown and mountains in the background." src="https://cdn-images-1.medium.com/max/640/0*QAnZ2HlK5ZPKu1dk.png" /><figcaption>Now the background from the original woman has been replaced with mountains.</figcaption></figure><p>Now, notice that center portion of the woman that is trees? Those look like they correspond to the red portion of the original class map we created. If we look at the label_map output, we see that diningtable and sofa are other labels that are printed out. It’s a bit of a hack, but let’s add those to labels_to_mask and re-run all of the above code for the smoothing edges section.</p><p>Now the output looks like this:</p><figure><img alt="Better boundaries between woman and moutains." src="https://cdn-images-1.medium.com/max/640/0*NPR7XPN1V3F0gLZ2.png" /><figcaption>Using different labels, we have filled in the gaps in the woman.</figcaption></figure><p>Much better!</p><p>I hope you found this tutorial helpful. We covered how to interact with edgeiq and cv2 from within the terminal, how to work with segmentation masks, and how to replace specific portions of images using these masks. This should set you up for working with semantic segmentation models in general. We also walked through how to smooth the output from more coarse-grained segmentation models, which should help you improve your applications when coarse-grained segmentation models are the only segmentation models available, or are the only models that best fit your use specific case.</p><p>Registering for a <a href="https://alwaysai.co/auth?register=true?utm_campaign=Website%20Traffic&amp;utm_source=medium&amp;utm_medium=medium&amp;utm_content=learn_computer_vision_terminal">free alwaysAI account</a> gives you access to different computer vision model types as well, so you can use the Python Interpreter to explore object detection, image classification, and pose estimation next!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cfb79041adb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/learn-computer-vision-using-the-python-interpreter-8cfb79041adb">Learn Computer Vision Using the Python Interpreter</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Halloween, but Make It Socially-Distanced]]></title>
            <link>https://lilamullany.medium.com/halloween-but-make-it-socially-distanced-f4e4d8b212c?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4e4d8b212c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[social-distance]]></category>
            <category><![CDATA[halloween]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Wed, 28 Oct 2020 01:22:28 GMT</pubDate>
            <atom:updated>2020-10-28T01:22:28.655Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use Computer Vision to Make your Halloween as Safe as it is Fun</h4><figure><img alt="Spooky picture of a skull, candle, and potion bottle." src="https://cdn-images-1.medium.com/max/1024/0*IFsZc8yUOCiQMzmi" /><figcaption>Image by <a href="https://pixabay.com/users/socialneuron-4098061/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=2217630">socialneuron</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=2217630">Pixabay</a></figcaption></figure><p>The CDC has put out <a href="https://www.cdc.gov/coronavirus/2019-ncov/daily-life-coping/holidays/halloween.html">guidelines and safe practices for Halloween</a> this year. Traditional trick-or-treating has been deemed ‘<a href="https://www.cdc.gov/coronavirus/2019-ncov/daily-life-coping/holidays.html#halloween">high-risk</a>’. Other ‘moderate’ risk activities, such as visiting social-distanced, open-air parties or mazes still require a mask and come with the caveat that revelers constantly sanitize their hands and don’t scream too much.</p><p>For those of you wanting a low-key Halloween this year, we’ve put together a treat in the form of three computer vision apps to help you stay safe while you celebrate!</p><p>Set the scene for your Halloween by adding the right backdrop. Maybe you want to be in a haunted castle or graveyard. Use the <a href="https://github.com/alwaysai/virtual-green-screen">virtual green screen application</a> with your favorite backdrops to have your Halloween in the destination of your choice. You can find some freely available <a href="https://pixabay.com/images/search/spooky/">spooky backdrops</a> on sites like Pixabay, Pexels, and Unsplash.</p><p>What’s Halloween without a costume? Checkout this <a href="https://www.youtube.com/watch?v=veLLscnrVdU">video tutorial</a> to find out how to add accessories to your costume like glasses or a moustache using facial landmarking. A basic facial landmarks repo can be found <a href="https://github.com/alwaysai/facial_landmarks_with_dlib">here</a>; the video tutorial uses a fork of this repo, found <a href="https://github.com/abhatikar/facial_landmarks_with_dlib">here</a>. Add your favorite accessories using PNG files with transparent backgrounds!</p><p>Maybe instead of accessories, you want to have an entirely new face for Halloween. You can use this <a href="https://github.com/alwaysai/face-swapper">face swapper app</a> to swap your face with your favorite Halloween character, or you can mix-and-match your face with anyone else in the same video stream as you.</p><h4>Need some other ideas?</h4><p>Maybe instead of the face detector, use a <a href="https://alwaysai.co/model-catalog?model=alwaysai/hand_detection">hand detection model</a> to give yourself a Hulk hand, or Mickey (or Minnie) Mouse gloves. Rig an app to play a spooky sound file (.wav format) when an object is detected. Or use this <a href="https://github.com/alwaysai/thermal-imaging">thermal imaging app</a> to make sure no one sneaks up on you in the dark. You can read more about how these apps were made on the <a href="https://alwaysai.co/blog?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=halloween-apps-social-distance">alwaysAI blog</a>. Join the alwaysAI <a href="https://discord.gg/rjDdRPT">Discord community</a> to collaborate on your computer vision projects!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4e4d8b212c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Computer Vision on the Edge]]></title>
            <link>https://medium.com/data-science/computer-vision-on-the-edge-1804257a20a?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/1804257a20a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[edge-computing]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Mon, 21 Sep 2020 22:43:10 GMT</pubDate>
            <atom:updated>2020-10-01T17:17:58.246Z</atom:updated>
            <content:encoded><![CDATA[<h4>Overcoming Challenges in Bringing CV Applications to Production</h4><figure><img alt="An image of gear cogs interleaved together." src="https://cdn-images-1.medium.com/max/1024/1*k4Ik2MXRZtNy-rs4fMWrKg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@bill_oxford?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Bill Oxford</a> on <a href="https://unsplash.com/s/photos/working-together-technology?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Developing a Computer Vision (CV) application and bringing it to production requires integrating several pieces of hardware and software. How can we ensure the pieces work seamlessly together? With the right methodologies, we can expedite development and deployment of CV applications. It is essential to find a platform with the goal of helping developers create computer vision applications from scratch quickly and easily with an integrated set of freely available tools. This article describes some of the challenges to developing and deploying CV applications, and how to mitigate them. If you would like more information, you can watch this free <a href="https://www.youtube.com/watch?v=l381u5WfAkc">webinar</a> on deployment challenges with computer vision.</p><h4>Computer Vision in the World</h4><p>First, some background. The term ‘computer vision’ refers to the process of a computer being able to analyze visual data as a human would, and make inferences about what that data contains. When integrated in an application, these inferences can be turned into actionable responses. For instance, take self-driving vehicles: computer vision can be used to analyze real-time video data to detect objects in the road, and then the software running the computer vision model can respond to this detection to stop the car or change its path to avoid the object.</p><p>CV is being leveraged more and more to solve diverse <a href="https://alwaysai.co/use-cases?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=challenges_edge_deployment">real world problems</a>, in fields ranging from security and health care, to manufacturing, smart cities, and robotics. CV can be used to detect cancerous cells in radiology reports, help analyze body movements, like proper gait and posture, or track production on a manufacturing line.</p><p>Throughout the rest of this article, we’ll explore the challenges of building and deploying a security camera CV application. You can find an implementation of such an application that has already been built on <a href="https://github.com/alwaysai/snapshot-security-camera">GitHub</a>. This application notices when a new person has entered a frame and records an image of the person. In this blog, we’ll cover the challenges to building such an application, as well as those involved with deploying it, potentially using multiple cameras, but if you’d like to read more about the code logic in the application itself, you can read this <a href="https://alwaysai.co/blog/detect-people-using-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=challenges_edge_deployment">blog.</a></p><h4>Prototyping and Developing CV Applications</h4><p>CV applications involve numerous components and can be extremely complicated to develop. Along with integrating the hardware that does the processing with the application software itself, the process of developing a production-ready computer vision application involves a multitude of steps: data collection and annotation, training computer vision models, integrating existing models made with different frameworks into your application, integrating existing computer vision libraries in the application, etc. Given the potential cost and effort it takes to develop a computer vision solution, ideally you also want to make sure your application is flexible, so that your application can evolve as your problem space evolves. This means ensuring you can use additional model types, move to a different model framework, or change the hardware without breaking your application.</p><p>Considering the security application, let’s say you just want to detect people. If you want a model that is guaranteed to work at different times of day, in different weather, etc., as you most likely would for a production-quality application, you may need to collect and annotate images taken from the angle of the camera you’ll use and in different environments and train your own model, iterating through this workflow a few times to achieve the desired performance. However, let’s say you just want a prototype application, in which case you could probably use an existing model that detects people, such as <a href="https://alwaysai.co/model-catalog?model=alwaysai/mobilenet_ssd?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=challenges_edge_deployment">mobilenet_ssd</a> or <a href="https://alwaysai.co/model-catalog?model=alwaysai/yolo_v2_tiny?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=challenges_edge_deployment">yolo_v2_tiny</a>. Ideally, you could try different models to see which one(s) work best, and maybe your application would benefit from using two models at once, to perhaps detect people at different distances, or in different conditions. Once you have your object detection model(s), you can build your application. You’ll need to use software packages that enable you to access a video stream from a camera, and you’ll need to know how to access the returned predictions from the model you’ve trained, so you can see if that person is new to the frame and save the portion of the image that corresponds to the bounding boxes of the detected person.</p><h4>Deploying Computer Vision Applications</h4><p>Another challenge in developing computer vision applications is deployment. Once you have a working prototype of a computer vision application, how do you get it out in the world where it can have an impact? While the cloud is often seen as a very flexible solution for machine learning applications, it can be prohibitively expensive, have higher latency, and the transmission of data poses a security risk. The alternative and increasingly popular option is to take your CV application to the edge.</p><h4>CV on the Edge</h4><p>What is the edge? In general, edge devices are small, lightweight, devices on which a computer vision application can be deployed and run. Many edge devices today even have a Graphical Processing Unit (GPU), or Visual Processing Unit (VPU), which enable usage of a greater range of models and application complexity. In the context of this article, an edge device is a device such as a Raspberry Pi, NVIDIA Jetson devices like the Jetson Nano or TX2, or various Internet of Things (Iot) devices, which are edge devices that have some capability to sense or assess, and possibly interact with, the environment in which they are used.</p><p>While an internet connection can be used to deploy the application, once the application is on the edge device, there is no need for it to have cloud connectivity to function. This means that any inferencing the application does is on the edge device itself, not in the cloud, drastically reducing the time it takes for those inferences to be turned into actions by the application. For certain use cases, such as self-driving vehicles or security cameras, this is imperative. Aside from the risk that data may be lost being sent to and from the cloud, the additional time required for using a cloud approach can mean not responding to a task in time, which can be catastrophic for tasks like autonomous driving.</p><p>In addition to edge devices, there are specialized peripheral devices, in particular cameras, that don’t have internet connectivity themselves and are used by edge devices to improve performance or expand application functionality. With devices like these, this concept of processing on the edge is taken even further. While edge devices typically connect to a USB or ribbon camera, which passes the image data to the device for processing, these devices incorporate a processor into the camera itself, reducing the inference time even further.</p><p>Additionally, because the data doesn’t need to travel to the cloud with edge deployment, all data can stay in a closed circuit on the device itself, which is much more secure, and, without the need for cloud processing, much less expensive.</p><figure><img alt="Comparison of deploying a computer vision application on the edge vs. in the cloud." src="https://cdn-images-1.medium.com/max/1024/1*l5TDQ_yAVBKVbWteH7S2sw.jpeg" /><figcaption>Comparison of deploying a CV application on the edge vs. in the cloud. Created in Lucidchart (<a href="http://www.lucidchart.com/">www.lucidchart.com</a>).</figcaption></figure><h4>Challenges with Edge Deployment</h4><p>Deploying a production-quality computer vision application on the edge comes with its own set of challenges. With cloud computing you can scale instances when computation demand ramps up, however with edge devices, you are limited to the power of the individual device. Additionally, with edge deployment you may not have access to device status without integrating additional packages.</p><h4>Making it All Work</h4><p>We’ve discussed the complexity of developing computer vision applications as well as deploying them; how can we mitigate these challenges and get CV applications to production as quickly as possible? If you look at a platform like alwaysAI, there are a couple key methodologies utilized to reduce the complexity of this workflow: containerization and Application Programming Interface (API) abstraction.</p><p><strong>Containerization</strong> refers to the process of bundling up software and dependencies in one unit, which can be executed in different environments. AlwaysAI leverages containerization using Docker images to enable deployment on different devices, including Raspberry Pi and Jetson Nano. It also incorporates open source computer vision tools, such as the <a href="https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/documentation/installation.md">Computer Vision Annotation Tool</a> (CVAT), that rely on containerization directly into our package to make installation and usage of these tools simpler.</p><p>alwaysAI has also developed an <strong>API</strong> to further accelerate development. With this API, users can interact with and manipulate computer vision related objects, such as keypoints, bounding boxes, labels, etc. that are returned from model inference, as well as hardware features like cameras, and external entities like files (images, videos, etc.) without having to know all the details behind these requests. How does this work? An API serves as a kind of negotiator between a complicated backend and an end user. It defines what kinds of requests can be made by the user and how data is returned. This simplifies usage for the user, by hiding some of the complexities involved in these calls, and helps ensure the only appropriate requests can be made.</p><p>The alwaysAI API wraps other APIs to help further facilitate application development. The OpenCV-Python API is a popular computer vision library; alwaysAI has incorporated some of the most common OpenCV-Python API calls into its library in order to simplify utilizing these functions and make sure they work with other aspects of its library. In addition, any alwaysAI app can directly import cv2, the OpenCV library, without needing to install the library or add any dependencies. This way, legacy code depending on cv2 will still work and users can default to the OpenCV-Python API if desired, but development can be done with one library going forward for simplicity.</p><p>APIs can also help give developers more fine-grained access to hardware as well. Some specialized cameras, such as the RealSense and OpenNCC Knight cameras, have their own software development kits (SDKs) to facilitate application creation. These SDKs also include APIs, which are likely sufficient for integrating these cameras with existing applications. While hardware features can be accessed directly by using these APIs, it is also possible to incorporate a camera-specific API into another library already being used for application development, which is what we’ve done with our API. This means that end users can access these camera features with the same library their application already uses and it is assured to work.</p><p>APIs and containerization can also help overcome the challenges associated with deployment outlined above. Using a container-based deployment service, you can deploy multiple edge devices and monitor their status. Both <a href="https://kubernetes.io/">Kubernetes</a> and <a href="https://docs.docker.com/engine/swarm/">Docker Swarm</a> enable monitoring and managing multiple Docker containers, as does <a href="https://www.balena.io/">balena</a>, which is a commercial service with a freemium option. Both of these methods utilize APIs, which can subsequently be used by third party applications like <a href="https://prometheus.io/">Prometheus</a>, which is open source, or <a href="https://www.datadoghq.com/blog/monitor-kubernetes-docker/">Datadog</a>, which is commercial, to gain more insights into raw metrics. While this approach does rely on some cloud connectivity, the actual computer vision data is not transmitted across the network, so you still maintain your speed and low cost features of deploying on the edge, and security is improved as authentication is needed to connect to the edge devices, and the application data remains on the edge.</p><p>Let’s return to our security camera application one final time. You’ve built your application, you can swap out different cameras easily and you can store images when objects are detected. You can now use the containerization techniques we’ve mentioned above to coordinate deployment of your application across multiple edge devices, as shown in the diagram below. With the right methodologies, platforms, libraries, and hardware can be seamlessly integrated, making CV applications easier to develop and deploy, as well as scalable for production.</p><figure><img alt="Overview of software and hardware components of a deployed computer vision application." src="https://cdn-images-1.medium.com/max/1024/0*zBkMrPfFJEM4Y6ZR" /><figcaption>Overview of deployed CV application. Created in Lucidchart (<a href="http://www.lucidchart.com/">www.lucidchart.com</a>).</figcaption></figure><p>If you would like to learn more about overcoming the deployment challenges of computer vision, you can watch this <a href="https://www.youtube.com/watch?v=l381u5WfAkc">webinar on YouTube</a>.</p><p>Contributions to the article made by Eric VanBuhler and Stephanie Casola</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1804257a20a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/computer-vision-on-the-edge-1804257a20a">Computer Vision on the Edge</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using alwaysAI’s Model Training Tool to Build a License Plate Tracker]]></title>
            <link>https://levelup.gitconnected.com/using-alwaysais-model-training-tool-to-build-a-license-plate-tracker-dd5f6185047a?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd5f6185047a</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 02:30:17 GMT</pubDate>
            <atom:updated>2020-08-21T15:57:53.699Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build a License Plate Tracker From Scratch</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vyl0yiI2e8IlwHyS" /></figure><p>In this tutorial, we’ll cover how to create your own license plate tracker using the license plate detection model that was created using alwaysAI’s model training tool. If you want to read more about how the license plate detection model was built, read <a href="https://alwaysai.co/blog/creating-a-license-plate-detection-model?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">this blog</a>. To read more about model training in general, you can read this <a href="https://alwaysai.co/blog/introduction-to-computer-vision-model-training?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">model training overview article</a>.</p><p>The model training tool will become available soon, however if you’re interested in training your own object detection model now, you can fill out this <a href="https://alwaysai.surveykiwi.com/model-training-signup">survey</a> to participate in our model training beta program now. You can use a freely available dataset to train your own license plate detection model, found <a href="https://www.alwaysai.co/docs/_static/beta/dataset_sample_584.zip?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">here</a>, which is 185MB, and consists of 584 image/annotation pairs. You can then test out your new model using the example app we’ll build in this tutorial! The finished code from this tutorial, as well as test videos and dataset links, is available on <a href="https://github.com/alwaysai/license-plate-detector">GitHub</a>.</p><p>To complete the tutorial, you must have:</p><ol><li>An <a href="https://alwaysai.co/auth?register=true?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">alwaysAI account</a> (it’s free!)</li><li>alwaysAI <a href="https://alwaysai.co/docs/getting_started/development_computer_setup.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">set up on your machine</a> (also free)</li><li>A text editor such as <a href="https://www.sublimetext.com/">sublime</a> or an IDE such as <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>, both of which offer free versions, or whatever else you prefer to code in</li></ol><p>This app includes a few useful features you can incorporate in your other projects, namely:</p><ul><li>Using a correlation tracker to reduce overhead</li><li>Using multiple files as input to your app</li><li>Labeling individual detected objects for readability</li></ul><p>Please see the alwaysAI <a href="https://alwaysai.co/blog?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">blog</a> for more background on <a href="https://alwaysai.co/blog/computer-vision-101?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">computer vision</a>, <a href="https://alwaysai.co/blog/computer-vision-application-lifecycles-explained?%20&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">developing models</a>, how to <a href="https://alwaysai.co/blog/change-computer-vision-models-alwaysai-platform?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">change models</a>, and more.</p><p><strong>Let’s get started!</strong></p><p>We’ll break down this tutorial in two parts:</p><ol><li>Set up</li><li>App.py</li></ol><h3>Set Up</h3><p>In this tutorial, we’ll build the app from scratch. After you’ve signed up and logged in, go to <a href="https://alwaysai.co/?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">https://alwaysai.co</a> and navigate to your Dashboard. You can follow the steps outlined <a href="https://alwaysai.co/docs/getting_started/working_with_projects.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">here</a> to create a new project. For this app, you’ll want to choose ‘create a project from scratch’. Once your project is created on the Dashboard, scroll down to the ‘Models’ section and click the ‘+’ sign to add a model. Browse the model category to find “alwaysai/vehicle_license_mobilenet_ssd”, which will be in the ‘object detection’ models. Add this model to your project by clicking on it, and selecting ‘add to project’ and choosing your project from the drop down menu. Navigate back to your project Dashboard to get the configuration hash code and then finish configuring your project locally as described in the <a href="https://alwaysai.co/docs/getting_started/working_with_projects.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">documentation page</a>.</p><p>Finally, create a folder inside the directory that app.py is in named ‘<strong>video</strong>’. This is where we’ll store the input videos. You can find some sample videos in the <a href="https://github.com/alwaysai/license-plate-detector">GitHub</a> repository, along with the completed code for this tutorial.</p><h3>App.py</h3><p>Now, we’ll add the content to <strong>app.py</strong>. There are a few different moving parts here, but we’ll walk through the code top to bottom and explain them as we go!</p><p>First, make sure the following lines are at the top of app.py:</p><pre>import time<br>import edgeiq</pre><p>This will import the necessary libraries (edgeiq is the alwaysAI python API). Then, replace the contents of ‘main()’ with the following:</p><pre>def main():<br>    # The current frame index<br>   frame_idx = 0</pre><pre>   # The number of frames to skip before running detector<br>   detect_period = 30</pre><pre>   obj_detect = edgeiq.ObjectDetection(<br>           &quot;alwaysai/vehicle_license_mobilenet_ssd&quot;)<br>   obj_detect.load(engine=edgeiq.Engine.DNN)</pre><pre>   print(&quot;Loaded model:\n{}\n&quot;.format(obj_detect.model_id))<br>   print(&quot;Engine: {}&quot;.format(obj_detect.engine))<br>   print(&quot;Accelerator: {}\n&quot;.format(obj_detect.accelerator))<br>   print(&quot;Labels:\n{}\n&quot;.format(obj_detect.labels))</pre><pre>   tracker = edgeiq.CorrelationTracker(max_objects=5)</pre><pre>   fps = edgeiq.FPS()</pre><p>What we’ve done here is pretty standard if you’ve used any of alwaysAI’s starter or example apps before: we’re just setting up the main method, creating an object detector using the <strong>alwaysai/vehicle_license_mobilenet_ssd</strong> model, and we’re printing the object detector’s configuration to the console.</p><p>We’re also creating three important variables: <strong>frame_idx</strong>, <strong>detect_period</strong>, and <strong>tracker</strong>.</p><p>The variable <strong>tracker</strong> is the correlation tracker object. Using a tracker, such as the <a href="https://alwaysai.co/docs/application_development/object_tracking.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">correlation tracker</a> in the edge IQ library, reduces the CPU usage and inference time.</p><p>The variable <strong>frame_idx</strong> tracks how many iterations we’ve done, and <strong>detect_period</strong> defines how often we’ll perform object detection. If the frame count is not evenly divisible by 30, we instead check if the tracker is currently tracking any objects (using ‘count’), and if so, we set the ‘predictions’ variable to be the tracked predictions. To reduce overhead, increase detect_period, and vice versa.</p><p><em>NOTE: comparing the value returned by using modulo (e.g. m % n) to an integer (e.g. 0) in a loop statement is a common way of performing a task every </em><strong><em>n </em></strong><em>iterations and can be applied in your other projects. If you set </em><strong><em>n</em></strong><em> to 2 and compare the result to 0, your loop task will execute every </em><strong><em>even</em></strong><em> iteration, for example.</em></p><p>Next, copy all the code contents directly under what you just added. This instantiates a ‘try’ block, along with a ‘finally’ counterpart, which will always be executed regardless of the ‘try’ execution and will bring down the streamer, stop tracking the frames per second and print closing messages.</p><pre>    try:<br>        # blank for now, will fill in later!</pre><pre>finally:<br>        fps.stop()<br>        streamer.close()<br>        print(&quot;elapsed time {:.2f}&quot;.format(fps.get_elapsed_seconds()))<br>        print(&quot;approx. FPS: {:.2f}&quot;.format(fps.compute_fps()))</pre><pre>        print(&quot;Program Ending&quot;)</pre><p>Now that the configuration is done and we have a skeleton of an app to work with, we’ll fill in the object tracking and file import portions.<strong> All of the rest of the code will go into the ‘try’ block we created in the last step.</strong></p><p><strong>Inside the ‘try’ block</strong>, paste the following code:</p><pre>        video_paths = edgeiq.list_files(base_path=&quot;./video/&quot;, valid_exts=&quot;.mp4&quot;)<br>        streamer = edgeiq.Streamer().setup()</pre><pre>        for video_path in video_paths:<br>                with edgeiq.FileVideoStream(video_path)as video_stream:</pre><p>This code uses the edge IQ command ‘list_files’ to get a list of all the files you’ve stored in the ‘video’ folder that have the file extension ‘mp4’. Then, it iterates over each of the file paths in that returned list and runs the nested code on each, which we’ll cover in the following section.</p><p>Add in as many .mp4 files to the ‘video’ folder as you would like to test your new app on!</p><p><strong>Next, we’ll add in the tracking logic. </strong>For every file in the ‘video’ folder, we’ll use that video as an input stream and detect and track license plates and vehicles.</p><p><em>NOTE: the variable called ‘predictions’ that is defined at the beginning of the ‘while’ loop in the code below is a list that will be used to store the predictions sent to the streamer. It will be updated every ‘detect_period’, otherwise it will hold the ‘tracker’ predictions.</em></p><p>Add the following code directly under the for loop statement initializing the streamer (for continuity, all the code is given at once, however the code comments provide additional information for each step):</p><pre>                # Allow Webcam to warm up<br>                time.sleep(2.0)<br>                fps.start()</pre><pre>                # loop detection<br>                while video_stream.more():</pre><pre>                    frame = video_stream.read()<br>                    predictions = []</pre><pre>                    # if using new detections, update &#39;predictions&#39;<br>                    if frame_idx % detect_period == 0:<br>                        results = obj_detect.detect_objects(frame, confidence_level=.5)</pre><pre>                        # Generate text to display on streamer<br>                        text = [&quot;Model: {}&quot;.format(obj_detect.model_id)]<br>                        text.append(<br>                                &quot;Inference time: {:1.3f} s&quot;.format(results.duration))<br>                        text.append(&quot;Objects:&quot;)<br>                        # Stop tracking old objects<br>                        if tracker.count:<br>                            tracker.stop_all()</pre><pre>                        # Set predictions to the new predictions<br>                        predictions = results.predictions</pre><pre>                        if not predictions:<br>                            text.append(&quot;no predictions&quot;)</pre><pre>                        # use &#39;number&#39; to identify unique objects<br>                        number = 0<br>                        for prediction in predictions:<br>                            number = number + 1<br>                            text.append(&quot;{}_{}: {:2.2f}%&quot;.format(<br>                               prediction.label, number, prediction.confidence * 100))<br>                            tracker.start(frame, prediction)</pre><pre>                        else:<br>                        # otherwise, set &#39;predictions&#39; to those<br>                        # stored in the correlation tracker object<br>                            if tracker.count:<br>                                predictions = tracker.update(frame)</pre><pre>                        # either way, use &#39;predictions&#39; to mark up the image and update text<br>                        frame = edgeiq.markup_image(<br>                              frame, predictions, show_labels=True, show_confidences=False, colors=obj_detect.colors)<br>                        streamer.send_data(frame, text)<br>                        frame_idx += 1</pre><pre>                        fps.update()</pre><pre>                        if streamer.check_exit():<br>                            break</pre><p><strong>That’s it!</strong></p><p>You can now build the app and start it (see <a href="https://alwaysai.co/blog/building-and-deploying-apps-on-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=license_plate_tracker_sample">this blog</a> if you need assistance), and you should see output similar to that shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aJZuJMRDETzU-nmk" /></figure><p>If you’d like to build your own license plate model and tracker from scratch you can find the survey link, dataset, test videos, and the app code on <a href="https://github.com/alwaysai/license-plate-detector">GitHub</a>.</p><p>Contributions to the article made by Todd Gleed and Jason Koo</p><h3>Level Up Coding</h3><p>Thanks for being a part of our community! <a href="https://www.youtube.com/channel/UC3v9kBR_ab4UHXXdknz8Fbg?sub_confirmation=1"><strong>Subscribe to our YouTube channel</strong></a> or join the <a href="https://skilled.dev/"><strong>Skilled.dev coding interview course</strong></a>.</p><p><a href="https://skilled.dev/">Coding Interview Questions + Land Your Dev Job | Skilled.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd5f6185047a" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/using-alwaysais-model-training-tool-to-build-a-license-plate-tracker-dd5f6185047a">Using alwaysAI’s Model Training Tool to Build a License Plate Tracker</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using alwaysAI to Build Your Own License Plate Detection Model]]></title>
            <link>https://levelup.gitconnected.com/using-alwaysai-to-build-your-own-license-plate-detection-model-5914c8b2068?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/5914c8b2068</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Thu, 13 Aug 2020 02:42:25 GMT</pubDate>
            <atom:updated>2020-08-13T02:42:25.040Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="cars on a highway, with object detection boxes around some cars and license plates." src="https://cdn-images-1.medium.com/max/800/1*LJ8MsrLmDjlQ7VPFTuHE_g.png" /><figcaption>Sample output from the license plate detection model.</figcaption></figure><p>In this tutorial, we’ll walk through the steps and decision points involved in the creation of the ‘<strong>alwaysai/vehicle_license_mobilenet_ssd</strong>’ model, an object detection model for identifying vehicles and license plates.</p><p>At alwaysAI, we want to make computer vision accessible for all developers. One of the more challenging components of computer vision is model training. Our aim with this tutorial is to provide a detailed overview of how custom models can be trained quickly and easily using the alwaysAI model training tool, starting with data collection and ending with testing your model’s performance, including some tips and tricks for model training.</p><p>For information about model training in general, check out this article: <a href="https://alwaysai.co/blog/introduction-to-computer-vision-model-training?&amp;utm_campaign=Open%20Beta&amp;utm_source=aai_blog&amp;utm_content=making_license_plate_model">Introduction to Computer Vision Model Training</a>.</p><p>If you’d like to join our model training beta program and train your own model, you can fill out <a href="https://alwaysai.surveykiwi.com/model-training-signup">this short survey</a> to sign up!</p><h3>Introduction to the License Plate Model</h3><p>This model was developed with the intent that it would be general enough to detect both American and European license plates and be applied to a few field use cases, including dashboard cameras and static traffic cameras at:</p><ul><li>intersections</li><li>overpasses</li><li>parking lots</li><li>store or house fronts</li></ul><p>The model detects four-wheeled vehicles, including</p><ul><li>sedans</li><li>trucks</li><li>buses</li></ul><h3>Data Collection</h3><p>We collected data from different vantage points that support the desired use cases mentioned above. This included images from highway overpasses, high-angle street views, low-angle street views, parking garages and parking lots. We aimed to capture images that represented readable plates from as many angles and distances as possible. Below are some examples of included images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gsh19l-_P88fydhF" /></figure><p>Most of the data were originally collected as videos, from which individual frames were then sampled.</p><p>Having a lot of very similar images won’t enhance a model’s performance, so we sampled all of the collected data to ensure that the input data was sufficiently varied. For photos, this simply meant excluding individual photos that were too similar. For videos, a little more work was involved. First, these videos were edited in an application, such as QuickTime Player, to remove large unusable portions of the video, such as sections of dead feed or footage with no cars, etc. Once the videos were cleaned up a bit, they were sampled using <a href="https://ffmpeg.org/">ffmpeg</a>.</p><p>The command that was used was</p><pre>ffmpeg -i movie_name.mov -r 2 -q:v 1 image_name_%4d.png</pre><p>This command sampled 2 frames per second (specified with ‘-r 2’), using the highest quality (specified with ‘-q:v 1’) and saved as png, which uses lossless compression (rather than jpg).</p><p>Once the videos were sampled, the resulting images were manually scanned through to eliminate near-duplicates or frames that did not contain objects of interest.</p><p>We’ve also created a data collection starter app, which enables you to collect data using your edge device and perform simple sampling and which is available for model training beta users. You can use ‘ffmpeg’, as described above, to perform more advanced sampling.</p><h3>Data Annotation</h3><p>The Computer Vision Annotation Tool (<a href="https://github.com/opencv/cvat">CVAT</a>) was used to perform data annotation, using the alwaysAI CLI annotation tool, which can be accessed with ‘aai annotate’. With CVAT, there is a limit to the amount of data that can be uploaded at a given time, so annotation was done in batches and the annotation sets were later merged, which we’ll cover later in this article. For each task (containing approximately 40–100 images each), the labels were kept constant.</p><p>Originally, we used labels ‘car’ and ‘license plate’, however we decided ‘vehicles’ would be more appropriate for a general model and ‘license_plates’ may avoid issues with whitespace down the road. A useful feature of CVAT is that you can log in as a super user, navigate to the ‘<a href="https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/user_guide.md#administration-panel">admin</a>’ panel, and modify the labels of an annotation task at any time!</p><p>When a series of images contained the same annotation objects, the <a href="https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/user_guide.md#track-mode-basics">interpolation mode</a> was used; this mode enables annotations to persist from one frame to the next, significantly reducing the time it takes to annotate a dataset.</p><p><em>NOTE: CVAT doesn’t automatically save your work, so remember to save often!</em></p><p>CVAT supports multiple annotation formats, but alwaysAI currently uses only the PASCAL VOC format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/0*S7gYcraL2_Tlo5Ji" /><figcaption>Example of exporting the datset in PASCAL VOC format.</figcaption></figure><h3>What to Annotate</h3><p>In general, we chose to annotate any license plate and vehicle that were discernible and around which a box could be drawn. Occasionally, this meant annotating a vehicle without a license plate or annotating a partially occluded vehicle, although we tried to limit these instances as much as possible. If a portion of an image clearly represented an object that we wanted to detect, even if it was very small, we annotated it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cyw_G6B_dWRF13ZJ" /></figure><h3>Merging Annotations</h3><p>As we mentioned above, you may not be able to upload all your images at once. Therefore, you’ll end up with multiple exported datasets that will need to be merged. The training tool will automatically merge your datasets if you pass in multiple files for training, however you can also easily do this with the alwaysAI CLI command ‘aai dataset merge’ if you want to work with just one dataset and simplify the training command.</p><pre>aai dataset merge dataset1.zip dataset2.zip dataset3.zip</pre><p>In the above command we showed three datasets being merged, however you can specify as many datasets to merge as necessary.</p><p><em>NOTE: we had the best success with merging ‘raw’ annotation sets exported directly from CVAT. If you need to train on datasets not exported from CVAT, try zipping the individual folders (‘Annotation’ and ‘JPEGImages’) within the parent annotation directory first. Then try merging the resulting zipped files.</em></p><h3>Training the Model</h3><p>We trained the model using alwaysAI’s CLI tool for model retraining. The tool is currently in closed beta and we are rapidly iterating on its features and functionality. More documentation on the training features will be made available soon.</p><p><em>NOTE: if you’re interested in joining the model training beta, fill out this </em><a href="https://alwaysai.surveykiwi.com/model-training-signup"><em>short survey</em></a><em>!</em></p><p>Currently, the tool can train an object-detection model by using transfer-learning on a mobilenet ssd network trained on the COCO dataset, using Tensorflow 1.14. There is support for either training using a CPU or using a GPU, and upon completion the model is instantly compatible with the alwaysAI platform.</p><p>We initially trained the model using CPU on a personal laptop, specifically a MacBook Pro. We began with batch size of 1 and increased it to 16, at which point the laptop ran out of memory. This was most likely the result of using large images in the dataset. We decided to use a batch size of 4 for the proof of concept models, varying the number of epochs and training images. Once we were satisfied with the progression of the model, we decided to drastically bump the number of epochs. The specifications of this training and the various configurations are shown in the table below, and training took approximately 20 hours.</p><p>Hardware</p><p><em>NOTE: There are much more powerful EC2 instances available; we could increase the number of GPUs to 24, and or use the more powerful Tesla V100 cores, but we wanted to test on more realistic machines.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9O-ee9Ktz83aBiE-5-fiEg.png" /><figcaption>Training configurations used in the creation of the license plate detection model.</figcaption></figure><h3>Analyzing Model Performance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*3vbaUroc79URC4svBTeo5Q.png" /><figcaption>Final performance metrics (loss and mAP) for the license plate detection model</figcaption></figure><p>The final statistics for this model indicate that we trained more steps than required. For the loss graph, you can see a long, relatively flat line. It is fluctuating up and down but it definitely flattens. In the graph of precision and recall, I highlight the area where the trend stops angling up. It seems we could have stopped training at about 35k steps and gotten similar results.</p><p>Here is the graph for loss for the training session. Number of steps is on the x-axis, and loss is plotted every 200 steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/0*bCf5ULw6G6nZksIp" /><figcaption>Loss vs. Steps</figcaption></figure><p>And here is the precision and recall statistics that are output during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nLm8zoHCHeilDxro" /><figcaption>mAP and Recall vs. Steps</figcaption></figure><p>While the graphs show a flattening, the model doesn’t appear to be overfit to our dataset. This could be confirmed by plotting validation loss, and that is a feature that we will soon implement. The test that we rely on the most is a visual test on novel data, which is easy to perform using the alwaysAI platform, and what we’ll describe next.</p><p>Models trained using the alwaysAI CLI can be published using</p><pre>aai app publish &lt;username/modelname&gt;</pre><p>Once you publish your model, you can add it to your app using</p><pre>aai app model add &lt;username/modelname&gt;</pre><p>You can also test it locally, without publishing it, by using</p><pre>aai app model add --local-version &lt;version&gt;</pre><p>We wanted to test how the model would perform in real-time, so we modified an existing starter app to use a file stream and used pre-collected video representing the desired use cases. You can see how this was done in <a href="https://alwaysai.co/blog/build-your-own-license-plate-tracker?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=making_license_plate_model">this blog</a>.</p><p>Some example output from testing the license plate detector model is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vyl0yiI2e8IlwHyS" /></figure><h3>Next Steps</h3><p>To access the license plate detector, and many other pre-trained models, sign up for a free <a href="https://alwaysai.co/auth?register=true?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=making_license_plate_model">account</a> to get started with computer vision today. If you’re interested in participating in our closed beta for model training and training your own object detection model, fill out the <a href="https://alwaysai.surveykiwi.com/model-training-signup">survey</a> to sign up!</p><p>You can find a subset of the dataset used in the creation ‘<strong>alwaysai/vehicle_license_mobilenet_ssd</strong>’ model <a href="https://www.alwaysai.co/docs/_static/beta/dataset_sample_592.zip?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=making_license_plate_model">here</a> and a larger version <a href="https://www.alwaysai.co/docs/_static/beta/dataset_sample_1186.zip?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=making_license_plate_model">here</a>, as well as all the code for the corresponding license plate tracker app to test your model on <a href="https://github.com/alwaysai/license-plate-detector">GitHub</a>.</p><p>Contributions to the article made by Todd Gleed and Jason Koo</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5914c8b2068" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/using-alwaysai-to-build-your-own-license-plate-detection-model-5914c8b2068">Using alwaysAI to Build Your Own License Plate Detection Model</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own Posture Corrector With Pose Estimation]]></title>
            <link>https://medium.com/better-programming/build-your-own-posture-corrector-with-pose-estimation-97009943e484?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/97009943e484</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 04:04:55 GMT</pubDate>
            <atom:updated>2020-07-14T16:03:08.700Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use alwaysAI and fix your posture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d6P7zbE0htaTluXe" /><figcaption>Photo by <a href="https://unsplash.com/@priscilladupreez?utm_source=medium&amp;utm_medium=referral">Priscilla Du Preez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>Many of us spend most of our days hunched over a desk, leaning forward looking at a computer screen, or slumped down in our chair. If you’re like me, you’re only reminded of your bad posture when your neck or shoulders hurt hours later, or you have a splitting migraine. Wouldn’t it be great if someone could remind you to sit up straight?</p><p>The good news is you can remind yourself! In this tutorial, we’ll build a posture corrector app using a pose estimation model available from alwaysAI.</p><p>To complete the tutorial, you must have:</p><ol><li>An <a href="https://alwaysai.co/auth?register=true?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">alwaysAI account</a> (it’s free!).</li><li>alwaysAI <a href="https://alwaysai.co/docs/getting_started/development_computer_setup.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">set up on your machine</a> (also free).</li><li>A text editor such as <a href="https://www.sublimetext.com/">sublime</a> or an IDE such as <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>, both of which offer free versions, or whatever else you prefer to code in.</li></ol><p>Please see the <a href="https://learn.alwaysai.co/?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">alwaysAI blog</a> for more background on computer vision, developing models, how to change models, and more. You can find step-by-step tutorials on the development machine set up for <a href="https://learn.alwaysai.co/setupformacusers?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">Mac</a> and <a href="https://learn.alwaysai.co/developing-alwaysai-windows?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">Windows</a> on the blog as well.</p><p>All of the code from this tutorial is available on <a href="https://github.com/alwaysai/posture-corrector?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">GitHub</a>.</p><h3>Let’s Get Started</h3><p>After you have your free account and have set up your developer environment, download the starter apps. Do so using <a href="https://alwaysai.co/docs/reference/starter_applications.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">this link</a> before proceeding with the rest of the tutorial. We’ll build the posture corrector by modifying the realtime_pose_detector starter app. You may want to copy the contents into a new directory so you retain the original code.</p><p>There will be three main parts to this tutorial:</p><ol><li>The configuration file</li><li>The main application</li><li>The utility class for detecting poor posture</li></ol><h3>Creation of the Configuration File</h3><p>Create the file as specified in <a href="https://learn.alwaysai.co/speed-up-development-with-a-json-configuration-file?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">this tutorial</a>. For this example app, we need one configuration variable (add more if you want): scale. It is an int and will be used to tailor the sensitivity of the posture functions.</p><p>Now the configuration is all set up!</p><h3>Creation of the App</h3><p>Add the following import statements to the top of your app.py file:</p><pre>import os<br>import json<br>from posture import CheckPosture</pre><p>We need json to parse the configuration file, and CheckPosture is the utility class for detecting poor posture that we’ll define later in this tutorial.</p><p><em>Note: You can change the engine and the accelerator you use in this app depending on your deployment environment. Since I am developing on a Mac, I chose the engine to be “DNN,” and so I changed the engine parameter to be </em><em>edgeiq.Engine.DNN. I also changed the accelerator to be “CPU.” You can read more about the accelerator and engine options in </em><a href="https://alwaysai.co/docs/edgeiq_api/constants.html"><em>alwaysAI’s documentation</em></a><em>.</em></p><p>Next, remove the following lines from app.py:</p><pre>text.append(&quot;Key Points:&quot;)<br>for key_point in pose.key_points:<br>    text.append(str(key_point))</pre><p>Add the following lines to replace the ones you just removed (right under the text.append statements):</p><pre><em># update the instance key_points to check the posture</em><br>posture.set_key_points(pose.key_points)</pre><pre><em># play a reminder if you are not sitting up straight</em><br>correct_posture = posture.correct_posture()<br>if not correct_posture:<br>   text.append(posture.build_message())</pre><pre>    # make a sound to alert the user to improper posture<br>    print(&quot;\a&quot;)</pre><p>We used an unknown object type just there and called some functions on it that we haven’t defined yet. We’ll do that in the last section!</p><p>Move the following lines to directly follow the end of the code above (directly after the for loop and right before the finally):</p><pre>streamer.send_data(results.draw_poses(frame), text)</pre><pre>fps.update()</pre><pre>if streamer.check_exit():<br>    break</pre><h3>Creating the Posture Utility Class</h3><p>Create a new file called posture.py. Define the class using the line:</p><pre>class CheckPosture</pre><p>Create the constructor for the class. We’ll have three instance variables: key_points, scale, and message.</p><pre>def __init__(self, scale=1, key_points={}):<br>   self.key_points = key_points<br>   self.scale = scale<br>   self.message = <strong>&quot;&quot;</strong></pre><p>We used defaults for scale and key_points in case the user doesn’t provide them. We just initialize the variable message to hold an empty string, but this will store feedback that the user can use to correct their posture. You already saw the key_points variable get set in the app.py section. This variable allows the functions in posture.py to make determinations about the user’s posture. Finally, the scale simply makes the calculations performed in posture.py either more or less sensitive when it is decreased or increased, respectively.</p><p>Now we need to write some functions for posture.py.</p><p>Create a getter and setter for the key_points, message, and scale variables:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b26fd81dd2a499efba515805013f7c8/href">https://medium.com/media/5b26fd81dd2a499efba515805013f7c8/href</a></iframe><p>Now we need functions to actually check the posture. My bad posture habits include leaning forward toward my computer screen, slouching down in my chair, and tilting my head down to look at notes, so I defined methods for detecting these use cases. You can use the same principle of coordinate comparison to define your own custom methods if you prefer.</p><p>First, we’ll define the method to detect leaning forward, as shown in the image below. This method works by comparing an ear and a shoulder on the same side of the body. So first it detects whether the ear and shoulder are both visible (i.e. the coordinate we want to use is not -1) on either the left or right side, and then it checks whether the shoulder’s <em>x</em>-coordinate is greater than the ear’s <em>x</em>-coordinate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8g2mm1VY5bXD1Jr6.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bcb69f53d1633fb8ff4753b258a5c20/href">https://medium.com/media/1bcb69f53d1633fb8ff4753b258a5c20/href</a></iframe><p><em>Note: The coordinates for </em><em>alwaysai/human-pose are 0,0 at the upper left corner. Also, the frame size will differ depending on whether you are using a Streamer to input video or images, and this will also impact the coordinates. I developed using a Streamer object and the frame size was (720, 1280). For all of these functions, you’ll most likely need to play around with the coordinate differences or modify the scale, as every person will have a different posture baseline. The principle of coordinate arithmetic will remain the same, however, and can be used to change app behavior in other pose estimation use cases!</em><strong><em> </em></strong><em>You could also use angles or a percentage of the frame so as to not be tied to absolute numbers. Feel free to rework these methods and submit a pull request to the GitHub repo!</em></p><p>Next, we’ll define the method for slouching down in a chair, such as in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OFdSTZbsTfcSJYiC.png" /></figure><p>In this method, we’ll use the <em>y</em>-coordinate neck and nose key points to detect when the nose gets too close to the neck, which happens when someone’s back is hunched down in a chair. For me, about 150 points represented the maximum distance I wanted to allow. If my nose is fewer than 150 points from my neck, I want to be notified. Again, these hardcoded values can be scaled with the scale factor or modified as suggested in the note above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9189a3c587c5555a76986fcf32c77172/href">https://medium.com/media/9189a3c587c5555a76986fcf32c77172/href</a></iframe><p>Now, we’ll define the method to detect when a head is tilted down, as shown in the image below. This method will use the ear and eye key points to detect when the <em>y</em>-coordinate of a given eye is closer to the bottom of the image than the ear on the same side of the body.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_T5l31M_20Dd5Vv0.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c643659bc8c795c4e41faa6c68b05dd1/href">https://medium.com/media/c643659bc8c795c4e41faa6c68b05dd1/href</a></iframe><p>Now, we’ll just make a method that checks all the posture methods. This method works by using Python’s all method, which only returns True if all iterables in a list return True. Since all of the posture methods we defined return False if the poor posture is detected, the method we define now will return False if any one of those methods returns False.</p><pre>def correct_posture(self):<br>    return all([self.check_slump(), self.check_head_drop(), self.check_lean_forward()])</pre><p>And finally, we’ll build one method that returns a customized string that tells the user how they can modify their posture. This method is called in app.py and the result is displayed on the streamer’s text:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3f4cecef2002812468a6a1fbe90ce50/href">https://medium.com/media/d3f4cecef2002812468a6a1fbe90ce50/href</a></iframe><p>That’s it! Now you have a working posture correcting app. You can customize this app by creating your own posture detection methods, using different key point coordinates, making the build_message return different helpful hints, and creating your own custom audio file to use instead of the print(“\a”). You could even alter the app to send you a <a href="https://alwaysai.co/blog/build-simple-computer-vision-texting-application-using-twilio?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">text message</a> instead of printing to the console!</p><p>If you want to run this app on a Jetson Nano, update your Dockerfile and the accelerator and engine arguments in app.py as described in <a href="https://learn.alwaysai.co/using-pose-estimation-on-the-jetson-nano-with-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">this article</a>.</p><p>Now, just start your app (follow <a href="https://learn.alwaysai.co/building-and-deploying-apps-on-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=posture-corrector">this tutorial</a> if you need a refresher on how to do this for your current setup) and open your web browser to localhost:5000 to see the posture corrector in action!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97009943e484" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/build-your-own-posture-corrector-with-pose-estimation-97009943e484">Build Your Own Posture Corrector With Pose Estimation</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Pose Estimation on the Jetson Nano with alwaysAI]]></title>
            <link>https://lilamullany.medium.com/using-pose-estimation-on-the-jetson-nano-with-alwaysai-140b0ab30b70?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/140b0ab30b70</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[pose-estimation]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[jetson-nano]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 03:59:11 GMT</pubDate>
            <atom:updated>2020-07-10T03:59:11.737Z</atom:updated>
            <content:encoded><![CDATA[<p>Many models, including those for pose estimation, may have much better performance when run on a GPU rather than a CPU. In this tutorial, we’ll cover how to run pose estimation on the Jetson Nano B01 and cover some nuances of running starter apps on this edge device.</p><p>To complete the tutorial, you must have:</p><ol><li>An <a href="https://alwaysai.co/auth?register=true?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">alwaysAI account</a> (it’s free!)</li><li>alwaysAI <a href="https://alwaysai.co/docs/getting_started/development_computer_setup.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">set up on your machine</a> (also free)</li><li>A text editor such as <a href="https://www.sublimetext.com/">sublime</a> or an IDE such as <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>, both of which offer free versions, or whatever else you prefer to code in</li><li>A Jetson Nano B01 (+keyboard, mouse, wifi adapter, &amp; monitor for setup)</li></ol><p>Please see the alwaysAI <a href="https://learn.alwaysai.co/?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">blog</a> for more background on <a href="https://learn.alwaysai.co/computer-vision-101?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">computer vision</a>, <a href="https://learn.alwaysai.co/blog/computer-vision-application-lifecycles-explained-0?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">developing models</a>, how to <a href="https://learn.alwaysai.co/blog/change-computer-vision-models-alwaysai-platform-0?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">change models</a>, and more.</p><p><strong>Let’s get started!</strong></p><p><em>NOTE: if you already have the </em><a href="https://alwaysai.co/docs/getting_started/development_computer_setup.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector"><em>alwaysAI CLI</em></a><em> installed on your machine, make sure you have the latest installed, at least version 0.3.1 or higher.</em></p><p>Make sure you have set up your Jetson Nano first: we’ve already developed a <a href="https://learn.alwaysai.co/getting-started-with-the-jetson-nano-using-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">tutorial</a> to help you do just that. Work your way through the tutorial at the previous link at least through the ‘Docker’ section. If you need help setting up your development machine, you can find step-by-step tutorials on <a href="https://learn.alwaysai.co/getting-started-with-the-jetson-nano-using-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">Mac</a> and <a href="https://learn.alwaysai.co/developing-alwaysai-windows?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">Windows</a> installation on the blog page as well.</p><p>If you have trouble testing your device connection with ‘ping’ you can also try to ssh. If you are on a Mac, ssh in with</p><pre>ssh username@device_name.local</pre><p>And if developing on Windows or Linux, use</p><pre>ssh username@device</pre><p><em>NOTE: your username can be found under ‘Settings-&gt;User Accounts’ and the device_name can be found under the main menu (upper right hand corner)-&gt; ‘About this computer’-&gt; ‘Device name’. You can edit username and device name here as well, if you would like. Finally, if you have trouble with the ssh timing out, just keep trying ‘aai app configure’, it should work within a few tries. If you get a ‘device not found’ type of error, try restarting the nano, especially if you just enabled wifi or booted up for the first time.</em></p><p>Once ssh is successful, you can type ‘exit’ into the command line to get out of the ssh.</p><p>Finally, as the Jetson Nano set up tutorial mentions, the Dockerfile should contain the ‘Nano’ base image. You must use the most current version of edge IQ for the nano to perform pose estimation; currently, this is nano-0.14.1.</p><p>Make sure the first line of the Dockerfile is</p><pre>FROM alwaysai/edgeiq:nano-0.14.1</pre><p><em>NOTE: you can find updates on the edge IQ releases at this </em><a href="https://alwaysai.co/docs/edgeiq_api/edgeiq_release_notes.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector"><em>link</em></a><em>.</em></p><p>Once your Jetson Nano is all set up and you’ve downloaded the starter applications, we can edit an existing starter app to run pose estimation on it. Either copy the contents of ‘<strong>realtime_pose_estimator</strong>’ into a new folder, or modify the existing starter app if you prefer.</p><p>The other main difference in this tutorial will be the engine, accelerator, and model we use. For using the Jetson Nano, use the NVIDIA accelerator and the TENSOR_RT backend engine. For more details on swapping out the engine and accelerator in general, visit this <a href="https://alwaysai.co/docs/application_development/changing_the_engine_and_accelerator.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">page</a>. To update the engine and accelerator, replace the following lines in app.py</p><pre>pose_estimator.load(engine=..., accelerator=...)</pre><p>with</p><pre>pose_estimator.load(engine=edgeiq.Engine.TENSOR_RT, accelerator=edgeiq.Accelerator.NVIDIA)</pre><p>Now we need to update the model. In the original example app code we used ‘human-pose’, but when using the Jetson Nano, we need to use ‘human_pose_nano’. To update the model, use the CLI, typing the following into the command line:</p><pre>aai app models add alwaysai/human_pose_nano<br>aai app models remove alwaysai/human-pose</pre><p>Now, just start your app (visit <a href="https://learn.alwaysai.co/building-and-deploying-apps-on-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">this page</a> if you need a refresher on how to do this for your current set up), and open your web browser to ‘localhost:5000’ to see the pose estimation in action!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*JddUbhDtlIwIP6EW.png" /></figure><p>You can see our <a href="https://learn.alwaysai.co/?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">blog</a> page for example applications using pose estimation including a <a href="https://learn.alwaysai.co/recognize-human-activity-using-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=jetson_slouch_detector">YMCA application</a> and a posture corrector.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=140b0ab30b70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Computer Vision Model Training]]></title>
            <link>https://medium.com/data-science/introduction-to-computer-vision-model-training-c8d22a9af22b?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8d22a9af22b</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[model-training]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Tue, 09 Jun 2020 01:00:17 GMT</pubDate>
            <atom:updated>2020-07-30T20:24:51.101Z</atom:updated>
            <content:encoded><![CDATA[<p>Training a computer vision model is one component of a complex and iterative undertaking, which can often seem daunting. At alwaysAI we want to make the process simple and approachable. To get you started, we have compiled a general overview of the training process of Deep Neural Networks (DNNs) for use in computer vision applications. We will focus on supervised learning in this overview, which uses labeled training data to teach the model what the desired output is. This article provides an introduction to each component of the model training process and will be the contextual basis for more in-depth articles in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CR9U8swozgIdtuTgKNf1zw.png" /><figcaption>The model training cycle.</figcaption></figure><h3>Overview</h3><p>The following sections are covered in this article:</p><ul><li>Types of Models</li><li>Dataset Generation</li><li>Dataset Annotation</li><li>Model Training</li><li>Testing &amp; Using</li></ul><h3>Types of Models</h3><p>Different computer vision models help us answer questions about an image. What objects are in the image? Where are those objects in the image? Where are the key points on an object? What pixels belong to each object? We can answer these questions by building different types of DNNs. These DNNs can then be used in applications to solve problems like determining how many cars are in an image, whether a person is sitting or standing, or whether an animal in a picture is a cat or a dog. We’ve outlined a few of the most common types of computer vision models and their use cases below.</p><p><em>NOTE: In general, computer vision model output consists of a label and a confidence or score, which is some estimate of the likelihood of correctly labeling the object. This definition is intentionally vague, as ‘confidence’ will mean very different things for different types of models.</em></p><p>In describing the different types of models and their use cases, we’ll outline an example use case of a virtual wardrobe: an application that lets users try on different clothing items virtually, before making a purchase for instance.</p><h4>Image Classification</h4><p><a href="https://learn.alwaysai.co/identify-objects-image-classification-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">Image classification</a> attempts to identify the most significant object class in the image; in computer vision, we refer to each class as a label. For example, we can use a general classification model, such as ‘<strong>alwaysai/googlenet</strong>’, to identify items of clothing, such as ‘running shoes’ or a ‘sweatshirt’, as shown below. The model would take an image as input and it would output a label along with the confidence the model has for the particular label compared to other labels. Additionally, DNNs for image classification tasks do not provide the location of the object in the image, so for use cases where we need this information, in order to track or count objects for example, we need to use an object detection model, which is the next model described.</p><figure><img alt="Example of a general classification model used to classify clothing items classifying running shoes." src="https://cdn-images-1.medium.com/max/662/1*hSHKdJn8zgfdlrxCshMWmQ.png" /><figcaption>Example of a general classification model used to classify clothing items classifying running shoes.</figcaption></figure><figure><img alt="Example of a general classification model used to classify clothing items classifying a sweater." src="https://cdn-images-1.medium.com/max/647/1*41pbw9IlSt6v3Tw-NHOEQw.png" /><figcaption>Example of a general classification model used to classify clothing items classifying a sweater.</figcaption></figure><h4>Object Detection</h4><p>When the location of the object is of importance, <a href="https://learn.alwaysai.co/build-real-time-object-dectector-starter-application?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">object detectio</a>n DNNs are typically used. These networks return a set of coordinates, called a bounding box, that specify an area of the input image containing an object, along with confidence value for that bounding box and a label. For our ‘virtual wardrobe’ application, we would need an input image of the person who wants to try on the virtual clothes, and then we would need to find the person in the image. To do this, we could use an <a href="https://learn.alwaysai.co/detect-people-using-alwaysai?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">object detection model for person detection</a>, such as ‘<strong>alwaysai/mobilenet_ssd</strong>’, that would return a bounding box around every person in an image, along with a label, ‘person’, and a confidence value for the output. An example of an object detection model that can distinguish people is shown below.</p><p><em>NOTE: Knowing the location of objects in a frame allows us to infer certain information about an image. For instance, we could count how many vehicles there are on a freeway to map out traffic patterns. We can also extend the functionality of an application by piggy-backing a </em><a href="https://learn.alwaysai.co/using-an-object-detection-model-with-a-classifier?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models"><em>classification model onto an object detection model</em></a><em>. For instance, we could feed the portion of the image corresponding to the bounding box from the detection model into the classification model so we could count how many trucks there are in the image versus sedans.</em></p><figure><img alt="An example of two detection model’s outputs. In this case, we can see that both models can detect the person, but one model i" src="https://cdn-images-1.medium.com/max/1024/1*O7Ro7Z7pJ95CUBNzycS2cw.png" /><figcaption>An example of two detection model’s outputs. In this case, we can see that both models can detect the person, but one model is able to detect both types of water bottles, whereas the other model has only found one.</figcaption></figure><p>Now we’ve seen how we can classify clothing into groups, such as shoes or sweatshirts, and we can detect a person in an image, but we still need to be able to let the user try on the clothes. This requires being able to distinguish the pixels that belong to the detected object from the pixels in the rest of the image, and in that case we would want to use segmentation, which we’ll cover next.</p><h4>Image Segmentation</h4><p>As we described above, in some tasks it is important to understand the exact shape of the object. This requires generating a pixel level boundary for each object, which is achieved through image segmentation. DNNs for image segmentation classify each pixel in an image by either object type, in the case of semantic segmentation, or by individual objects, in the case of instance segmentation.</p><p><em>NOTE: Currently, the alwaysAI platform supports semantic segmentation. We are always looking to grow the platform and are adding new models, including models that perform instance segmentation.</em></p><p>For our virtual wardrobe application, we could use semantic segmentation to distinguish the pixels that belong to the person, sweater, or shoe. We can then replace the pixels in one object with those from another object, such as replacing a sweater that the person in the original image was wearing with the pixels from the new sweater that the user wants to try on.</p><p><em>NOTE: a popular use case for semantic segmentation is the </em><a href="https://learn.alwaysai.co/create-your-own-virtual-green-screen?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models"><em>virtual background</em></a><em> used for tele-conferencing software like Zoom or Microsoft Teams: the pixels belonging to a person are distinguished from the rest of the image and are segmented out from the background.</em></p><p>In our examples for image segmentation applications for people, we saw that image segmentation enables us to distinguish which pixels belong to each object in an image. However, image segmentation doesn’t enable us to infer anything about the relative position of objects in the image, such as where a person’s hand is or where the taillights and bumper are on a car. For that, we would need information about specific areas on a person, say where a person’s hand is compared to their head. This requires tracking object landmarks, which is the last model type we’ll cover here.</p><h4>Object Landmark Detection</h4><p>Object landmark detection is the labeling of certain ‘keypoints’ in images that capture important features in the object. For our virtual wardrobe, we could use a pose estimation model, such as ‘<strong>alwaysai/human-pose</strong>’, that identifies body keypoints such as hips, shoulders and elbows, similar to the image shown below, to help our users accessorize. We could use eye keypoints to place glasses or a hat on the person in our virtual wardrobe, or use the ‘neck’ keypoint to let them try on a scarf.</p><p><em>NOTE: another useful application that uses keypoints would be one that checks for proper form during exercises and sports.</em></p><figure><img alt="Body keypoints detected using ‘alwaysai/human-pose’." src="https://cdn-images-1.medium.com/max/925/1*BswXDMqKaRLoabXXfu_EiQ.png" /><figcaption>Body keypoints detected using ‘alwaysai/human-pose’.</figcaption></figure><h4>Model Applications in General</h4><p>Computer vision models can be applied to a whole host of various applications. You could build a classification model for classifying types of dogs in a dog show, or build a detection model to find cancerous cells in biopsy slides. Conservation biologists could use one model to detect the presence of a particular family or genus, and then feed that output to a classification model to determine species, and then aggregate this data to track conservation efforts. <a href="https://learn.alwaysai.co/how-to-detect-pedestrians-and-bicyclists-in-a-cityscape-video?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">Semantic segmentation</a> is used for self-driving car technologies. An object detection model could be used to count items and generate inventory in a grocery store. The possibilities for computer vision applications are extensive!</p><figure><img alt="Example output from the starter app ‘semantic_segmentation_cityscape’." src="https://cdn-images-1.medium.com/max/964/1*-V_dzvAOzOCKHPAA-Jao9w.png" /><figcaption>Example output from the starter app ‘semantic_segmentation_cityscape’.</figcaption></figure><h3>Dataset Generation</h3><p>Computer vision model training begins with assembling a quality dataset. As the adage goes, “garbage in, garbage out”. But what constitutes “garbage” for a computer vision dataset? In computer vision, “inference” is the term we use for applying a trained model to an input to infer an outcome. We like to say <strong>“train as you will inference”.</strong> So, a good rule of thumb for a quality computer vision dataset is that it is similar to the real-world data that will be input into the trained model. To ensure this, consider the angle of the image, the lighting and weather, whether the desired objects are obscured, how far away the desired targets are, the resolution and scale of the image, as well as the background and foreground.</p><h4>Types of Dataset Generation</h4><p>There are a few different ways to generate a dataset, depending on your timeline and desired use cases. If you want a general detection model to fit into a prototype application right away, you may want to try to find an existing dataset complete with annotations. If you instead want a model that does one specific task very well, you’ll probably need to collect your own images that more closely resemble the environment the model will be used in. For instance, if you just want a model that will detect birds in general, you could probably go with a pre-existing dataset of images containing birds in different settings; however, if you are that conservation biologist we described earlier and you want a model that will be used to consistently detect birds appearing on the feed from a particular camera you’ve set up, you should collect images from the perspective of that camera to train the model on. We’ll describe both of these options, as well as a couple others, below. We’ll also cover data augmentation, which you can use to increase the size of any dataset.</p><h4>Using an Existing Annotated Dataset</h4><p>Depending on what you intend your model to detect there may be freely available annotated datasets for you to use, which can drastically cut down the time it takes to train and deploy your desired model. However, you will have much less control over the data quality this way, since the training data might not as closely resemble the data you’ll be using as input into the model, and, as such, the performance of your model may suffer. Therefore this approach may be best for proof of concept projects, and you will likely need to generate your own specific dataset at a later date, depending on your specific application.</p><p>Some popular existing datasets include Common Objects in Context (COCO), PASCAL Visual Object Classes (VOC), ImageNet, and Google’s Open Images Dataset V6. Some public datasets have non-commercial licenses, so remember to always check the license of any existing dataset you use.</p><h4>Using Existing Data or Collecting Your Own Data</h4><p>You can compile your own dataset by recording video, taking photos, or searching for freely available videos and images online. Unlike pulling data from an existing annotated dataset, you will need to annotate your collected images before you can use them for training. There are many popular sites for gathering photos including Unsplash, Pixabay, Pexels; and the latter two also provide videos. While gathering data, keep in mind the principles for data collection outlined earlier and attempt to keep your dataset as close to your inference environment as possible, remembering that ‘environment’ includes all aspects of the input images: lighting, angle, objects, etc.</p><p>Consider the following images for building a model that detects license plates. While every picture includes a car with a license plate, they have very different content. The first one depicts a nighttime scene with multiple cars; the second image shows a darker scene, but still daylight, with rain; the third one has more than one car, includes people and is more pixelated; and the fourth one has a different style of plate than the other two. These are all examples of variables that need to be considered when choosing images.</p><figure><img alt="Different environments for license plates and cars." src="https://cdn-images-1.medium.com/max/1024/1*3dmq8YlK6uWFkK9INhCDOg.png" /><figcaption>Environment examples for cars and license plates.</figcaption></figure><p><em>NOTE: As lots of images that are almost identical won’t enhance your model, if there are many frames per second in your video, sample your to select frames that will best suit your dataset needs.</em></p><h4>Using a Digitally Generated Dataset</h4><p>Another way to generate a dataset is to use synthetic, i.e. computer generated data. This technique is used if you are unable to gather enough data yourself. By using synthetic data, especially for training unusual circumstances, you can generate a much larger dataset than you would otherwise be able to gather from real-world occurrences, resulting in better performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h_0VYBVV0bAJL6au6UoXSg.jpeg" /><figcaption>A synthetic image.</figcaption></figure><h4>Data Augmentation</h4><p>Finally, a technique that can be used to boost your current dataset is data augmentation. This involves taking the existing dataset images and flipping, rotating, cropping, padding, or otherwise modifying them, to create images that are different enough to constitute a new data point; this can add variety to the training data and help avoid overfitting. Many deep learning frameworks (discussed below) include data augmentation capabilities.</p><h3>Dataset Annotation</h3><p>Generally, annotation is the process of selecting a portion of an image and assigning a label to that region. In the case of image classification, annotation is done by assigning a single label to the entire image, rather than a specific region. Annotated data are the input for model training via supervised learning. With many annotations and with enough variation in the underlying images, the model training identifies distinguishing characteristics of the images, and learns to detect the objects of interest, classify images, etc., depending on the type of model being trained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhlELJefzFhl39z_9e9OEg.png" /><figcaption>Examples of different annotation types. Instance segmentation of two cars on a highway in the upper left; 3D cuboid boxes in the upper right. Straightline annotations in a highway example are shown on the bottom left, and keypoint annotations in the bottom right.</figcaption></figure><p>Examples of annotation include: drawing bounding boxes or 3D cuboid boxes and assigning labels to objects for object detection, tracing around objects using polygonal outlines for semantic and instance segmentation, identifying key points and landmarks and assigning labels for object landmark detection, and identifying straight lines, such as is used in lane detection. For image classification, images are placed in groups, with each group corresponding to one label.</p><h3>Model Training</h3><h4>General Overview</h4><p>After the data are collected and annotated, they are used as input for model training. The data are fed into the DNN, which then outputs the prediction: a label for image classification, labels and bounding boxes for object detection, label maps for image segmentation, and keypoint sets for landmark detection, all of which are accompanied by a confidence. The model compares the prediction to the annotations, and makes adjustments to the DNN to produce better predictions. The process is repeated until you have good performance, based on a variety of metrics specific to each training session or model type. To build your own computer vision model for your application, you can start from scratch and build your own model architecture, or you can use an existing one.</p><h4>Transfer Learning &amp; Retraining</h4><p>Transfer learning leverages the knowledge gained from training a model on a general dataset, and applies it to other, possibly more specific, scenarios. You can take an existing model that has been trained on a large, general dataset, or a dataset that is similar to yours, and re-train on new labels that are specific to your use case. For instance, if you’re building a model that detects garbage, you could take a general object detection model and train it on specific labels relevant to your use case such as cans, bottles, and toilet paper rolls. Later, you realize you want to be able to detect take-out containers and cutlery. You could then collect and annotate additional images that contain these new objects, adding them to your original dataset, and train the model with these additional labels. This effectively updates the model so it can be applied to additional use cases.</p><p><em>NOTE: Model re-training has the extra advantage of requiring less training data!</em></p><h3>Testing Application</h3><p>Finally, once you have made your model using your training data, you can test it by feeding in new, unannotated images and seeing whether the model classifies, detects, etc. how you expect. For instance, continuing with the example of building a model to detect birds on an outside camera feed, you could collect some new video footage from the camera, perhaps a compilation of footage over the course of a typical day, or during different weather, and examine the output from the model in real time. While this isn’t an automated or quantified test, it enables you to quickly identify any shortcomings or edge cases that the model does not perform well on, such as during dawn and dusk, or when there it is raining. You can then re-train your model using annotations from images depicting these edge cases, or at least be aware of the specific environments in which the model performs best.</p><h3>Using Your Model on AlwaysAI</h3><p>AlwaysAI enables users to get up and running and deploy computer vision models to the edge quickly and easily. We’ve described a few models existing in the alwaysAI model catalog that can be used to build applications we highlighted as examples in this article; you can see descriptions of all the models in the catalog <a href="https://alwaysai.co/model-catalog?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">here</a>.</p><p>Our API, <a href="https://alwaysai.co/docs/edgeiq_api/image_classification.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">edge IQ</a>, enables you to tailor using these models in your applications independent of hardware choices and develop prototypes rapidly. You can filter by certain labels, markup images with predictions, change the markup color, add text, swap out media input and output, change the engine or accelerator, and much more, by using the same standard library.</p><p>With alwaysAI, you can also upload a custom model into the model catalog and add it to your testing app, or you can simply swap out the training output files in your local starter app, to quickly test out your new model.</p><p>Head to <a href="https://alwaysai.co/auth?register=true?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=general_making_cv_models">alwaysai.co</a>, sign up to use our platform, and get started in computer vision today!</p><p>Contributions to the article by Andres Ulloa, Todd Gleed, Eric VanBuhler, Jason Koo, and Vikram Gupta</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8d22a9af22b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/introduction-to-computer-vision-model-training-c8d22a9af22b">Introduction to Computer Vision Model Training</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Your Own Contraband Detector]]></title>
            <link>https://lilamullany.medium.com/create-your-own-contraband-detector-910ce0ee679b?source=rss-d55d0a812c15------2</link>
            <guid isPermaLink="false">https://medium.com/p/910ce0ee679b</guid>
            <category><![CDATA[object-tracking]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Lila Mullany]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 21:31:58 GMT</pubDate>
            <atom:updated>2020-04-15T17:37:04.310Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gZuQGtrZkqxVaR6IQBeHJw.jpeg" /></figure><p>Many people are now working or learning from home, introducing the new issue of enforcing professionalism and academic honesty by remote. Whether you are trying to prevent students from cheating on tests, want to double check that your kids aren’t just playing on their phones, or maybe you want to stop yourself from checking social media while you’re supposed to be working, computer vision can help. In this tutorial, I’ll show how you can create your own contraband object alert system using computer vision.</p><p>To complete the tutorial, you must have:</p><ol><li>An <a href="https://alwaysai.co/auth?register=true?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">alwaysAI account</a> (it’s free!)</li><li>alwaysAI <a href="https://alwaysai.co/docs/getting_started/development_computer_setup.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">set up on your machine</a> (also free)</li><li>A text editor such as <a href="https://www.sublimetext.com/">sublime</a> or an IDE such as <a href="https://www.jetbrains.com/pycharm/">PyCharm</a>, both of which offer free versions, or whatever else you prefer to code in</li></ol><p>Please see the alwaysAI <a href="https://learn.alwaysai.co/?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">blog</a> for more background on <a href="https://learn.alwaysai.co/computer-vision-101?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">computer vision</a>, <a href="https://learn.alwaysai.co/blog/computer-vision-application-lifecycles-explained-0?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">developing models</a>, how to <a href="https://learn.alwaysai.co/blog/change-computer-vision-models-alwaysai-platform-0?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">change models</a>, and more.</p><p>All of the code from this tutorial is available on <a href="https://github.com/alwaysai/contraband-detector">GitHub</a>.</p><p><strong>Let’s get started!</strong></p><p>After you have your free account and have set up your developer environment, download the starter apps; do so using this <a href="https://alwaysai.co/docs/reference/starter_applications.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">link</a> before proceeding with the rest of the tutorial.</p><p>There will be two main parts to this tutorial:</p><ol><li>Creation of the main app</li><li>Creation of the utility class to store and access a detection log</li></ol><p>To reduce overhead due to inference time as well as CPU usage, we will use a correlation<a href="https://alwaysai.co/docs/application_development/object_tracking.html?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector"> tracker</a> in this app. In this case, we will only look at every 50 frames for the target object, which is sufficient for our purposes, as we do not need to track the objects in every frame, just capture the occurrence.</p><p>We will be using two object detection models in order to maximize the amount of relevant detections. The two models we are using are ‘alwaysai/ssd_mobilenet_v2_oidv4’ and ‘alwaysai/ssd_inception_v2_coco_2018_01_28’. The first model, ‘alwaysai/ssd_mobilenet_v2_oidv4’, has a larger library of relevant labels and includes items such as ‘headphones’. The second model, ‘alwaysai/ssd_inception_v2_coco_2018_01_28’, detects certain objects, such as ‘cell phones’ and ‘books’, more readily and consistently. You can see the original post on using two detectors simultaneously for more information.</p><p><strong>Part I — Creation of app.py</strong></p><ol><li>Since we’re using an object tracker, we will modify the ‘detector_tracker’ starter app; cd into that directory and open up app.py.</li><li>Replace the lines declaring ‘obj_detect’, the subsequent call to ‘obj_detect.load’, and the print statements that follow with the following code:</li></ol><pre># to test an additional model, add it to the list below:<br>models = [“alwaysai/ssd_mobilenet_v2_oidv4”,”alwaysai/ssd_inception_v2_coco_2018_01_28&quot;]</pre><pre># include any labels that you wish to detect from any models <br>detected_contraband = [“Pen”, “cell phone”, “backpack”, “book”, “Book”, “Ring binder”, “Headphones”, “Calculator”, “Mobile phone”, “Telephone”, “Microphone”, “Ipod”, “Remote control”]</pre><pre># load all the models (creates a new object detector for each model)<br>detectors = []<br>for model in models:</pre><pre>    # start up a first object detection model<br>    obj_detect = edgeiq.ObjectDetection(model)<br>    obj_detect.load(engine=edgeiq.Engine.DNN)</pre><pre>    # track generated object detectors by storing them in detectors<br>    detectors.append(obj_detect)</pre><pre>    # print the details of each model to the console<br>    print(“Model:\n{}\n”.format(obj_detect.model_id))<br>    print(“Engine: {}”.format(obj_detect.engine))<br>    print(“Accelerator: {}\n”.format(obj_detect.accelerator))<br>    print(“Labels:\n{}\n”.format(obj_detect.labels))</pre><p>The declaration of models builds a list of all models we would like to include. The declaration of ‘detected_contraband’ specifies which labels from those models we want to check the video stream for. We then build a list of detection objects, stored in ‘detectors’ and perform the initialization, loading, and printing of the details from the original app on each detection object.</p><p>3. Instantiate a contraband tracker by adding the following line underneath the declaration of the ‘fps’ variable.</p><pre>contraband_summary = ContrabandSummary()</pre><p>4. Additionally, add the following declaration to the top of the code</p><pre>from contraband_summary import ContrabandSummary</pre><p>Don’t worry about the ContrabandSummary code just yet, we’ll come back to that in the second part of the tutorial.</p><p>5. Everywhere in the code, change references to the variable called ‘predictions’ to instead refer to ‘predictions_to_markup’. This is a small distinction, but we won’t be marking up every prediction and this reinforces that.</p><p>6. Add a declaration of a variable named ‘text’ and initialize it to be a list containing an empty string. The results from steps 5 and 6 should look like this:</p><pre>while True:<br>    frame = video_stream.read()</pre><pre>    predictions_to_markup = []</pre><pre>    text = [“”]<br> <br>    # only analyze every ‘detect_period’ frame (i.e. every 50th)<br>    if frame_idx % detect_period == 0:</pre><p>7. Remove the declarations of the ‘text’ variable from within the ‘if’ statement.</p><p>8. Replace all of the code inside the ‘for prediction in predictions’ loop with the following:</p><pre>if (prediction.label.strip() in detected_contraband):<br>    contraband_summary.contraband_alert(prediction.label, frame)<br>    predictions_to_markup.append(prediction)<br>    tracker.start(frame, prediction)</pre><p>This code only takes action if the detected object is in the list of contraband we defined in the beginning. If it finds a contraband item, it updates with contraband_summary with a call to contraband_alert() and adds the prediction to the ‘predictions_to_markup’. Finally, it starts tracking these new detections.</p><p>9. Finally, beneath the initialization of the ‘frame’ variable, add the following code to update the ‘text’ variable:</p><pre>text =contraband_summary.get_contraband_string()</pre><p>That’s all we need for the first part of this app! Now we will create the contraband_summary utility class.</p><p><strong>Part II — Creation of contraband_summary.py</strong></p><p>This separate utility class is used to abstract away how we handle contraband detections. By separating this logic away from the main app, we can change how we handle detections at any time without breaking the main application. For now, we simply store all detections and generate a log string that can be used to print to the console or returned to a calling class. Later, we could change contraband_summary to send an sms alert or save the images to disk and we wouldn’t need to alter app.py to do so.</p><ol><li>Create a new file called ‘contraband_summary.py’ from the command line or by creating a new file using the file browser, inside the same directory as app.py.</li></ol><p>2. Include the time module by adding the following line to the very top of the file:</p><pre>import time</pre><p>3. Declare a class called ‘ContrabandSummary’:</p><pre>class ContrabandSummary:</pre><p>4. Inside this class declaration, create a constructor for the ContrabandSummary class that initializes an empty list, which we will use to store contraband detections:</p><pre>def __init__(self):<br>    self.contraband_detections = []</pre><p>5. Create a ‘contraband_alert’ method. This method takes in a string, which is intended to be the label of the object, and the numpy array of the video frame. It calculates the local time and appends a tuple containing the string, the time, and the frame to the ‘contraband_detections’ list. The method is implemented in the following way:</p><pre>def contraband_alert(self, contraband, frame):<br>    detect_time = time.localtime()<br>    self.contraband_detections.append((contraband, detect_time, frame))<br>    items = self.get_contraband_string()</pre><pre>    print(*items)</pre><p>Notice the method calls another method in the ContrabandSummary class: get_contraband_string(). By separating the functionality of generating the log, we can change the way this log is presented in the future and we also gain the ability to call these two methods independently.</p><p>6. The get_contraband_string() method builds a log of all the detections of contraband items. We just want to know the item that was detected and the time that the detection occurred. Implement this method by copying the following code into ContrabandSummary class below the previous method:</p><pre>def get_contraband_string(self):<br>    contraband_string = [“”]</pre><pre>    for contraband, detect_time, frame in self.contraband_detections:<br>        string_time = time.strftime(‘%Y-%m-%d %H:%M:%S’, detect_time)<br>        contraband_string.append(contraband + “ detected at “ + string_time + “\n”)</pre><pre>    return contraband_string</pre><p><strong><em>NOTE</em></strong><em>: we don’t do anything with the collected numpy frame in this example, but you can see how this implementation offers the flexibility to change how to store or manipulate the data in however fashion best suits your needs. You could write the image to a folder, or send it to another application along with the text. You could also alter the alert function to relay the contraband detection in a </em><a href="https://learn.alwaysai.co/build-simple-computer-vision-texting-application-using-twilio?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector"><em>text message</em></a><em>.</em></p><p><strong>That’s it!</strong></p><p>Now, to see your app in action, first build the app by typing into the command line:</p><pre>aai app deploy</pre><p>And once it is done building, type the following command to start the app:</p><pre>aai app start</pre><p>Now open any browser to ‘localhost:5000’ to see your contraband detector in action. Check your console for a log of detections, and see the contraband items in detection boxes on the streamer, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VxtaxHx8NZ2a5Sk5HrRvPw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*o1HpymzsY_0DDthGMTYvfg.png" /><figcaption>The contraband detection log is printed to the console.</figcaption></figure><p>Visit <a href="https://alwaysai.co/?&amp;utm_campaign=Open%20Beta&amp;utm_source=medium&amp;utm_content=contraband_detector">alwaysai.co</a>, sign up to use our platform, and get started in computer vision today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=910ce0ee679b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>