<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tyler Hutcherson on Medium]]></title>
        <description><![CDATA[Stories by Tyler Hutcherson on Medium]]></description>
        <link>https://medium.com/@tyler.hutcherson?source=rss-b4f5908a7cac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*He0VP2sg_HBqzEfJ.</url>
            <title>Stories by Tyler Hutcherson on Medium</title>
            <link>https://medium.com/@tyler.hutcherson?source=rss-b4f5908a7cac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:19:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tyler.hutcherson/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[High-throughput Product Recommendations]]></title>
            <link>https://medium.com/@tyler.hutcherson/high-throughput-product-recommendations-8b9252c1969e?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b9252c1969e</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Thu, 16 Dec 2021 13:57:53 GMT</pubDate>
            <atom:updated>2021-12-16T13:57:53.120Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, <a href="https://medium.com/skafosai/interactive-product-search-engine-insights-from-a-ml-engineer-23c5462ba6df">I shared about Skafos’ interactive search engine</a> for product recommendations. It highlights the overall ML approach and focuses on why this works for our unique business and set of product offerings within Shopify.</p><p>In this post, I will open up the hood a bit further and share some tools that we use to make our product recommendations fast enough for modern web-based interactivity. As you might expect — it comes down to picking the right tools for the job and picking an architecture that is flexible and scalable.</p><h3>Annoy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*DGyX6pqfqN6jfXjF" /><figcaption><a href="https://github.com/spotify/annoy">https://github.com/spotify/annoy</a></figcaption></figure><p>Starting from the bottom of the stack, our machine learning models are wrapped in a popular Spotify-developed, open-source framework called <a href="https://github.com/spotify/annoy">Annoy</a> (Approximate Nearest Neighbors Oh Yeah). Annoy allows for efficient vector similarity calculations between items, in this case, eCommerce products. It’s fast, reliable, and accurate. The speed boost is in part due to the fact that you can hold the models in memory, even sharing them between processes. Additionally, as the name indicates, the framework <a href="https://github.com/spotify/annoy">performs an “approximate” search</a> using a tree, allowing it to cut some corners.</p><p>The beauty of using Annoy is that it’s just a vessel for delivering similarity recommendations. At the end of the day, you still need your own “special sauce” to create the model itself. At Skafos, we’ve invested countless hours into developing proprietary algorithms for transforming products into numerical vectors, a process called “embedding” or “encoding”: Image feature extraction, text processing, dimensionality reduction, matrix factorization…you name it.</p><h3>MultiThreading</h3><p>This piece is quite simple. As discussed in the last post, our search engine is composed of many models. We use multithreading to query asynchronously, and then reduce results into a final score vector. The Python <a href="https://docs.python.org/3/library/concurrent.futures.html">concurrent</a> library allows you to perform asynchronous execution with threads using a ThreadPoolExecutor.</p><p>Here’s a sample of code that might be useful:</p><pre># Submit search jobs to threadpool executor</pre><pre>with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:<br>    for model in models<br>        futures.append(ex.submit(model.search_by_item, product_id)</pre><pre>for future in concurrent.futures.as_completed(futures):<br>    result = future.result()</pre><h3>Redis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*dDQ48ay8968C-QXp" /></figure><p><a href="https://redis.com/">Redis</a> is the backbone of our architecture that makes the machine learning APIs soar. For those unfamiliar, it’s a “remote dictionary server” hosting an in-memory key/value store with some built-in datatypes and operations. We use Redis for the following on the backend:</p><ul><li>Result caching</li><li>Product attribute &amp; price filtering with <a href="https://redis.com/modules/redis-search/">RediSearch</a></li><li>Inter-service communication with <a href="https://redis.io/topics/pubsub">Pub/Sub</a></li></ul><h3>Microservices</h3><p>I’m not going to try to write anything deeply about microservices, as it is a widely debated and discussed topic in the software architecture world. Just do a bit of googling and you will see what I mean…</p><p>We build microservices because it gives the ability to release fixes and new features with ease and ultimate flexibility. It also benefits ML/AI applications because it can separate core functionality, resources, and business logic in a way that optimizes for inference + API throughput.</p><p>We use a tool called <a href="https://github.com/pnxtech/hydra">Hydra</a> to power our microservice architecture (fun fact: it’s also built on top of Redis).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/0*3urMOU98JyldwlWK.png" /></figure><p>These are just a few examples of powerful tools that you can use to speed up your machine learning APIs. The right choice depends on the use-case and functional requirements of the app and end-user. Hopefully, this gives you some ideas to start down the path.</p><p>Share below — What kinds of tools do you use to improve machine learning model speed + performance?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b9252c1969e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interactive Product Search Engine: Insights from a ML Engineer]]></title>
            <link>https://medium.com/@tyler.hutcherson/interactive-product-search-engine-insights-from-a-ml-engineer-23c5462ba6df?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/23c5462ba6df</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[shopify]]></category>
            <category><![CDATA[product-recommendations]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 12:47:28 GMT</pubDate>
            <atom:updated>2021-11-29T12:47:28.481Z</atom:updated>
            <content:encoded><![CDATA[<h3>Interactive Product Search Engine: Insights from a Machine Learning Engineer</h3><p>Product recommendations are everywhere in ecommerce. However, at <a href="http://skafos.ai/">Skafos</a>, we’ve taken a unique approach to assisting the online shopping journey. If you’ve experienced any of our recommendation widgets in the wild, they are unlike others you’ve probably seen elsewhere. <a href="https://youtu.be/lepbtTQWYkM">Watch this video to learn more</a>.</p><p>We’ll spend the rest of this article exploring exactly HOW we provide interactive product recommendations from the ML Engineer’s perspective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3xztw3R3TkqXZ7eB" /></figure><p>________</p><h3>The right tool for the right job</h3><p>Say you were trying to buy a new vacuum cleaner. A recommendation algorithm designed to get shoppers to fill up their cart with additional products might recommend a new vacuum filter, an extension cord, a Swiffer, or a matching broom. On the other hand, an algorithm designed to find visually similar vacuum cleaners will recommend alternatives that just look similar.</p><p>What’s also true is that the vacuum merchant may have different customer satisfaction goals adhering to their established brand.</p><p>Recognizing that the world of ecommerce is changing and also exploding, we’ve intentionally designed our recommendation system to be flexible, based on the goals of the merchant and quality of the training data.</p><h3>The Search Engine</h3><p>Our AI backend system is built as a product search engine, combining results from many models in parallel. Based on your “likes” (i.e. upvotes) and “dislikes” (i.e. downvotes) accumulated in a given shopping session, the frontend widgets use our ML API to make search requests with your explicit preferences.</p><p>For example, below is our “Product Finder” where you can upvote and downvote products as they flip, narrowing in on recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I4sCDAdA9mVh1sMA" /><figcaption>lightsonline.com</figcaption></figure><p>Another example is our “See Similar” widget that allows you to find similar products with one click.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vWgrJCoXmNHY1spx" /><figcaption><a href="https://zumaimports.com/collections/kilim-mules">https://zumaimports.com/collections/kilim-mules</a></figcaption></figure><p>On the backend, we have machine learning models that score products based on visual, text, or metadata similarity, as well as models for overall popularity, and frequently-bundled recommendations. On a search request, each individual model ranks the entire product catalog from <em>most</em> to <em>least</em> relevant based on the different facets described above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*teO0oL0xe07p9MEo" /></figure><p>Each model produces a “score” stored in item-wise vectors. Each value in the score vector represents a specific product/item in the catalog and its “score”. Higher scores are better (meaning a product is more relevant for a particular shopper). See example results from two arbitrary models, A and B, below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lPD7esfY-vcMcM7M" /></figure><p>Now the magic happens. Score vectors from each model are summed as a linear combination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pIMAkTiaLgrfyPQK" /></figure><p><strong>How are the model weights determined?</strong> Well… that’s our secret sauce. It’s also a constant work in progress. However, for some widgets, we may only care about visual similarity. For others, we may care only about popular products, or products frequently purchased together. The weighting is designed to be flexible. For applications where a mix of models is used, we start with a reasonable baseline for each shop. Then, over time, we start to adjust/tune the model weights based on interaction data and desired business outcomes.</p><p>After aggregating the product results, post-search filters are applied, final scores are normalized, and <strong>the response is returned to the client in under 300ms</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TRKQnSay_0Q5-LfR" /></figure><p>In summary, our product search engine gives ultimate flexibility to the merchant and agency to the shoppers. Want interactive product recommendations based on visual similarity, or some custom/magic combination? We can do that for you.</p><p>In a future post, I will share about specific technology choices and tools on the backend that allow us to provide recommendations in real-time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23c5462ba6df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training the DashLight Object Detection model in Create ML]]></title>
            <link>https://medium.com/@tyler.hutcherson/training-the-dashlight-object-detection-model-in-create-ml-5af96011c7c2?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/5af96011c7c2</guid>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[create-ml]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Tue, 01 Oct 2019 14:40:17 GMT</pubDate>
            <atom:updated>2019-10-09T15:11:33.218Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>, we built an iOS app called “DashLight” that locates and classifies icons on your car dashboard with the iPhone camera.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*f4il0PHJOfz89uyrxQSBJQ.png" /></figure><p>It has happened to all of us: we are driving along, and suddenly a dashboard light comes on. Instead of searching through a user manual by hand to find an icon’s explanation (tedious and time-consuming at best!), you can use DashLight to discover the identity and purpose of the illuminated icon. <em>This iOS app demonstrates a real world use-case that can be addressed with applied machine learning at the edge.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6VUWLRh_IUPDQTbOukcxw.jpeg" /><figcaption>DashLight overview.</figcaption></figure><p><a href="https://medium.com/skafosai/generating-training-images-for-object-detection-models-8a74cf5e882f">In a separate blog post</a>, I give an overview of object detection models and discuss the process of collecting training data for the type of model<strong> </strong>that’s powering the Dashlight application under the hood. If you are interested in exploring different ways to generate training data for object detection models, I recommend reading it! In the rest of this article I will show you the steps required to train the Dashlights object detection model with the new Create ML app, available in macOS 10.15.</p><h3>Gathering Training Data</h3><p>Summarizing the previously mentioned article, there are a number of different approaches to generating training data for object detection models. For Dashlight, since we didn’t already have a bunch of labeled training images of car dashboards and we couldn’t use an out-of-the-box method like <a href="https://apple.github.io/turicreate/docs/userguide/one_shot_object_detection/">Turi Create’s One Shot Object Detection</a>, we had to build our own.</p><p>Here is a <a href="https://github.com/tylerhutcherson/synthetic-images">link to the Github repository </a>with the Python code to generate your own synthetic training data given <em>backgrounds</em> and <em>objects</em>.</p><p>For the DashLight app, we generated thousands of training images like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qaTqRVBPKIRQQzI2JUo8ZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vu4Hbn-4aMK3Srx2pKsCMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j9aUnCE_-MsEv4cmYEn5kA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3WSWmGR3nfWgYmQ33FKiyA.png" /><figcaption>DashLight training data examples.</figcaption></figure><p>Surprisingly, this initial model did not perform very well! We had lots of issues getting it to properly recognize icons with the phone. So, we tried a different approach: we used a smaller set of data, collected by hand and labeled with <a href="https://rectlabel.com/">RectLabel</a>. <strong>Though we used a smaller dataset, the higher quality images produced a higher quality model.</strong> Soon I will release a separate blog post where I dive into that issue specifically as well as others.</p><h3>Training with Create ML</h3><p>Once you have a set of images for your training data, training a model with Create ML is a breeze and moves pretty quickly, especially if you have access to a GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*KG-ur4p4L68Nd5IC.png" /></figure><p>Before starting training, navigate to the CreateML app in the finder, right-click on the icon and select “Get Info”. This will open the panel shown below. Make sure that the option to prefer external GPU is selected. This option displays only if you have one connected. You’ll thank me later!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*u-pulb44C1-WfNC62BqULg.png" /><figcaption>Tell Create ML to prefer an external GPU.</figcaption></figure><p>Now, open the Create ML application and select Object Detection from the templates menu. Fill in a <em>Project Name</em>, <em>Author</em>, <em>License</em>, and <em>Description</em> as you see fit. Then, you’ll be presented with the following screen, where you can load in your data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*khtcsxBJE6EJapwLtaRWaA.png" /></figure><p>As shown above, the top panel inside the interface asks you to select a <a href="https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets">training, validation, and testing dataset</a>. Hit “Select Files…” and use the browser to select a folder that contains 2 things: annotations &amp; a folder of images. If your annotations file and images subdirectory do not have the structure shown below, Create ML will throw an error. Note that the annotations.json file contains the bounding box coordinates for each of the objects you are trying to detect. In this case, the bounding boxes delineate the location of each lit up icon on the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*kImwv7neeNAdcXodqyyfOA.png" /><figcaption>Data source structure for Create ML.</figcaption></figure><p>Once you’ve configured the data sources, you will need to set the max number of iterations you want the model to train for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*BM6--V0LNZm4LydLhbweEg.png" /><figcaption>Set the max iterations for model training in Create ML.</figcaption></figure><blockquote>If you need more control over model training, or would like to adjust parameters other than maximum iterations, I suggest using <a href="https://github.com/apple/turicreate/tree/master/userguide/object_detection">Turi Create’s Object Detection</a> toolkit in a Python environment instead of Create ML.</blockquote><p>Now we are ready to train the Object Detection model! <strong>Hit the Train “play button” in the top left corner of the app. </strong>For DashLight, I trained the model for ~1000 iterations, running 1.5 hours with an <a href="https://www.amd.com/en/products/graphics/radeon-rx-vega-64">AMD Radeon RX Vega 64</a> GPU connected to my laptop.</p><h3>Testing the Model</h3><p>Once training has completed, you should see a nice graph of training loss over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZqhtYkOxImTMK6SNQ2fIQ.png" /></figure><p>According to the plot, the model improved over the first few iterations and then leveled out. Unlike other model types offered by CreateML, we don’t get to see a trace of validation loss over time, meaning the risk of overfitting is high. A model that overfits the training data has a hard time generalizing to new data in the world it hasn’t already seen. This just means that we need to be particularly thorough in our model testing. Luckily, CreateML helps us to do this.</p><h4>Evaluating the Model on External Data</h4><p>Create ML provides two ways to check the Object Detection model’s accuracy before deploying it.</p><p><strong>Testing Accuracy</strong></p><p>Back in the first step, if you selected testing data, Create ML will unleash the model on the dataset and report an accuracy metric in the top right corner. This is a great way to get a model performance assessment in bulk.</p><p><strong>One-Off Tests</strong></p><p>Another way Create ML can test a model is through a drag and drop interface, where you can manually drag in images and see model predictions, right in CreateML. To get to this interface, click on the CoreML artifact in the top right corner:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*kFeh84wywRjf6VwB7dw9zQ.jpeg" /></figure><p>I selected a PNG image of a dashboard with three icons present: Emergency Brake, Tire Pressure, and Low Fuel. As you can see below, each were properly identified by the Object Detection model with high confidence!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*gRJVBOlhZCEbYSEomHLFgg.jpeg" /><figcaption>One-off testing interface in Create ML.</figcaption></figure><p>While this method isn’t necessarily a good measure of model strength over large sets of data, it provides a way to peel back the onion layers of the model and explore its tendencies with specific test cases.</p><h3>The Output</h3><p>At the end of this training process, we now have a <a href="https://developer.apple.com/documentation/coreml">Core ML</a> model that we deployed to the DashLight app with <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>. Using these steps in CreateML, you can also build an Object Detection model. Directly from the Create ML interface, you can drag this model to your desktop, Xcode, or elsewhere, enabling you to easily take the first steps towards building an ML-powered app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*kFeh84wywRjf6VwB7dw9zQ.jpeg" /><figcaption>Core ML model artifact.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*pECvS5IgKiIXig6J7KwN6w.jpeg" /></figure><p>At <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>, we are committed to helping you build the next generation of ML-powered apps. <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Sign-up</a> today for a free account and use our dashboard to manage, deploy, and monitor machine learning models at the edge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5af96011c7c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating Training Images for  Object Detection Models]]></title>
            <link>https://medium.com/@tyler.hutcherson/generating-training-images-for-object-detection-models-8a74cf5e882f?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a74cf5e882f</guid>
            <category><![CDATA[skafos]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[turi-create]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Tue, 01 Oct 2019 14:34:21 GMT</pubDate>
            <atom:updated>2019-10-01T14:45:11.036Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generating Training Images for Object Detection Models</h3><p><em>Making synthetic data for the DashLight iOS App</em></p><p>At <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=dashlight">Skafos</a>, we’ve built an iOS application called “DashLight” that locates and classifies icons on your car dashboard with the iPhone camera. Do you have a light that is on in your dash, but you don’t know what it is or how serious it is, and you don’t want to search through the manual? DashLight is for you. Under the hood, DashLight is powered by an Object Detection Machine Learning model. Interested in building your own? Read on to learn how we generated the training data to power this app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*QsBQ4QiSWJ5z7k4Nkg5ARQ.png" /><figcaption>Fig 1. DashLight app leveraging an object detection ML model.</figcaption></figure><p>Object Detection, a hot-topic in the machine learning community, can be boiled down to 2 steps:</p><ul><li><strong>Localization</strong>: Find a region (<em>bounding box</em>) in an image wherein lies some object of interest.</li><li><strong>Classification</strong>: Classify the identified object as one of <em>N</em> pre-defined classes from model training.</li></ul><p>In order to build one, we need training data that includes both the images themselves, and annotations (bounding boxes) that tell us precisely where in the objects are located. In the photo below, we show a more generic example of of several objects on a table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*foARJbfi09Rc6tYpmT1nyg.jpeg" /><figcaption>Fig 2. <a href="https://github.com/apple/turicreate/tree/master/userguide/object_detection">Object Detection bounding box example from Turi Create Docs.</a></figcaption></figure><p>In this example, we care about predicting balls and cups, so our annotations look like this:</p><pre>[{&#39;coordinates&#39;: {&#39;height&#39;: 104, &#39;width&#39;: 110, &#39;x&#39;: 115, &#39;y&#39;: 216},<br>  &#39;label&#39;: &#39;ball&#39;},<br> {&#39;coordinates&#39;: {&#39;height&#39;: 106, &#39;width&#39;: 110, &#39;x&#39;: 188, &#39;y&#39;: 254},<br>  &#39;label&#39;: &#39;ball&#39;},<br> {&#39;coordinates&#39;: {&#39;height&#39;: 164, &#39;width&#39;: 131, &#39;x&#39;: 374, &#39;y&#39;: 169},<br>  &#39;label&#39;: &#39;cup&#39;}]</pre><p>Once you have a set of training images and annotations, building an object detection model like the one used in DashLight is straightforward with <a href="https://github.com/apple/turicreate">Turi Create</a>.</p><h3>The Challenge</h3><p>Unfortunately, creating a dataset of images with bounding boxes can be quite challenging. How in the world do we make this kind of dataset? Generally, if we already have a bunch of images, we could use a tool like <a href="https://labelbox.com/">Labelbox</a> or <a href="https://makeml.app/">MakeML</a> to “draw” bounding boxes around each object, generating annotations like you see above. This would be quite tedious, but it is doable, and once we have these images and annotations, the <a href="https://apple.github.io/turicreate/docs/userguide/object_detection/">Turi Create Object Detection Toolkit</a> (or another tool of your choice) could handle the rest with ease.</p><p><strong>For the DashLight app, this is NOT the case.</strong></p><ol><li>We don’t have an image dataset of vehicle dashboards.</li><li>Even if we had one, building a production quality Object Detection model typically requires hundreds or thousands of images for training.</li><li>The users of the DashLight app will be pointing their iPhone camera at a car dashboard from varying distances and angles. Therefore, the dataset images must vary in size, coloring, and orientation for the model to be robust enough for the real-world.</li></ol><h3>Solutions</h3><p>Given that we didn’t have the training data we needed, and Labelbox or MakeML weren’t really options, where did that leave us?</p><h4>Go Collect It</h4><p>We could manually snap hundreds of images of car dashboards OR scrape the web. After this, we’d still have to annotate the images by hand to make them ready for modeling.</p><p>If you have access to multiple cameras, team members, and time, this method is a pretty brute-force approach, but very effective.</p><h4>One-Shot Object Detection</h4><p>Turi Create made a toolkit to help address this issue called <a href="https://apple.github.io/turicreate/docs/userguide/one_shot_object_detection/">One-Shot Object Detection (OSOD)</a>. If we supply <em>object</em> images, the toolkit will automatically create thousands of training images on random <em>backgrounds </em>and generate the <em>annotations</em> so we don’t have to!. This can be quite useful for many generic object detection applications; go try it out!</p><p>However, it doesn’t exactly meet DashLight’s needs:</p><ul><li>The OSOD data augmentation pipeline performs extraneous perturbations without any user control. Many of the image transformations don’t really fit our use-case, like <a href="https://helpx.adobe.com/photoshop-elements/key-concepts/skew.html">skew</a>.</li><li>Our dashlight icon images are extremely small compared to the background images provided by Turi Create. Size matters and we need a way to better control that aspect of the data.</li><li>Even if we supply our own background images, potentially fixing the size issue, we still require more control. We must be able to tune the location, scale, and orientation of the icons on each of the dashboards.</li></ul><p>I am fairly confident that the OSOD toolkit will soon be expanded to support our use-case, but for now, we’re on our own.</p><h4>Generate Synthetic Data</h4><p>Since we don’t have images for our DashLight model, we need to make some. Our approach will be similar to that of OSOD, but with more control.</p><p>Creating our dataset requires the following:</p><ul><li><strong>Backgrounds</strong>: Car dashboard images used as the backdrop for which <em>objects</em> will be superimposed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*R5KLiDTlB5eJkQ2D_Wxx6w.png" /><figcaption>Fig 3. Example vehicle dashboard image used as a background.</figcaption></figure><ul><li><strong>Objects</strong>: Smaller dashlight icons, placed on <em>backgrounds</em> in different locations, that we want our model to detect.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/25/1*5tc6a74tScLOlQoBCMgyPg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/30/1*JQMkcxc9JqH2ExnCjxan5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/30/1*oi3w0o-jq1CoO2mRTUUdUQ.png" /><figcaption>Fig 4. Example dashlight icons that will be superimposed on background images.</figcaption></figure><ul><li><strong>Perturbations</strong>: A set of (optional) transformations applied to our training images including resizing, scaling, and placement.</li></ul><p><strong>We will focus on this approach for the rest of the article.</strong></p><h3>Steps for Creating Synthetic Images</h3><p>The algorithm is straightforward: Grab a <em>background</em> dashboard image, place single or groups of icon <em>objects</em> in the frame, perform necessary <em>perturbations</em>, write out the <em>annotations</em>, and save the image. Next.</p><p>Our goal is to make thousands synthetic images like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*4SkIS-p7FI0y6ha7nnFZdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*iowhUPOcZtenB0VKazT8UQ.png" /><figcaption>Fig 5. Example synthetic images generated by our algorithm.</figcaption></figure><p>Clearly, these don’t perfectly resemble the exact location of a dashlight icon on a vehicle’s dashboard. But that’s 100% OK. What’s important is that <em>we can control</em> the amount and variety of data created.</p><p>We wrote a Python script (<a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#examples">leveraging the incredible Pillow library</a>) to automate the workflow and the image annotation.</p><blockquote>With this approach, we don’t need to manually take hundreds of pictures or scrape images — nor do we need to manually annotate them!</blockquote><h4><strong>Step 1: Download the Python Script</strong></h4><p>If you want to give this a shot, grab the script by cloning the code repository from <a href="https://github.com/tylerhutcherson/synthetic-images">github</a>. Enter the directory and activate the <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html">conda environment</a> provided. As an aside, if you’ve never used conda to manage Python environments, now is a great time to start!</p><pre>$ git clone git@github.com:tylerhutcherson/synthetic-images.git<br>$ cd synthetic-images<br>$ conda env create -f environment.yml<br>$ conda activate images</pre><h4><strong>Step 2: Gather Backgrounds and Objects</strong></h4><p>We put 3–4 dashboard images in the Backgrounds/ folder and 12 car dashlight icons in the Objects/ folder, ensuring that the backgrounds and objects had the appropriate <a href="https://en.wikipedia.org/wiki/Aspect_ratio_(image)"><em>aspect ratio</em></a> and <a href="http://paulbourke.net/dataformats/dimensions/"><em>dimension</em></a>. We also named each of the icon images with their class name. Do the same if you are following along with your own images!</p><h4><strong>Step 3: Run the Script</strong></h4><pre>$ python create.py --annotate True --sframe True --groups True</pre><p>There are other <a href="https://github.com/tylerhutcherson/synthetic-images#args">entrypoint options</a> available at runtime. Additionally, the Image class of the <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html">Pillow library</a> contains other methods that provide deeper functionality.</p><p>The script put all output images in the TrainingImages/ folder.</p><h4>What’s Happening Here?</h4><p><em>If you’re not interested in the specifics, feel free to skip this section!</em></p><p>The python script is a big loop that iterates through dashboards, icons, and pre-determined icon sizes. It creates images with single icons and groups of icons in an attempt to prepare the model for multiple real-world scenarios.</p><p><strong>Single Icon Images</strong></p><p>For each dashboard image (<em>d in D</em>), icon image (<em>i in I</em>), and icon size (<em>s in S</em>), we randomly generate 4 (x, y) coordinates for the upper-left corner of the icon with respect to the background image.</p><p><em>Total Images</em> = |D|*|I|*|S|* 4 = 4 * 12 * 5* 4 = 960 synthetic images</p><p><strong>Multiple Icon Images</strong></p><p>For each dashboard image (<em>d in D</em>), we randomly generate 24 separate groups of 2–4 icons that are paired together.</p><p><em>Total Images</em> = |D| * 24 = 96 synthetic images</p><p>The script also implements some logic to ensure the following:</p><ul><li>no icons overlap/intersect (<em>see line 59</em>)</li><li>no icons are partially out-of-bounds (<em>see line 72</em>)</li></ul><p>Lastly, it creates/writes annotations and saves the data as a <a href="https://apple.github.io/turicreate/docs/api/generated/turicreate.SFrame.html">Turi Create SFrame</a>.</p><blockquote>Please feel free to <a href="https://github.com/tylerhutcherson/synthetic-images">submit a PR to the repo</a> with any improvements you think are possible. Use this tool however you need. Once it is robust enough, we will consider publicizing the project more formally.</blockquote><p>Finally, we have a folder TrainingImages/ filled with a synthetic dataset AND a Turi Create SFrame 100% ready for modeling. <strong>Who says manual data creation is the only option!?</strong></p><p>After we trained the Object Detection model using Turi Create, we tested it out on some holdout images below in Fig 6:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*AcZXkKgU_n2AdjNKVKwqzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*JgK1fJpHPj_30QF7JCKGBQ.png" /><figcaption>Fig 6. Model predictions on holdout images.</figcaption></figure><h3>Conclusion</h3><p>Preparing data for Object Detection training can be a real pain. However, using some automated tools like <a href="https://apple.github.io/turicreate/docs/api/turicreate.toolkits.one_shot_object_detector.html#id1">Turi Create’s OSOD</a> or our own <a href="https://github.com/tylerhutcherson/synthetic-images">custom solution</a><strong> </strong>we were able to get moving much more quickly to begin prototyping a model. <a href="https://medium.com/skafosai/training-the-dashlight-object-detection-model-in-create-ml-5af96011c7c2">Read this post to learn about how we built the object detection model itself with CreateML.</a> <strong>What Object Detection use-cases are you working with?</strong></p><p>At <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=dashlight">Skafos</a>, we’re here to help YOU deliver ML capabilities to your apps. This is one example of many, including the <a href="https://apps.apple.com/us/app/boot-finder/id1472790615">Bootfinder</a> iOS app, that we are building to demonstrate the power and utility of machine learning at the edge. <a href="https://join.slack.com/t/metismachine-skafos/shared_invite/enQtNTAxMzEwOTk2NzA5LThjMmMyY2JkNTkwNDQ1YjgyYjFiY2MyMjRkMzYyM2E4MjUxNTJmYmQyODVhZWM2MjQwMjE5ZGM1Y2YwN2M5ODI">Don’t be a stranger, come say hi!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a74cf5e882f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Activity Classification with Create ML, CoreML3, and Skafos: Part 2]]></title>
            <link>https://medium.com/@tyler.hutcherson/activity-classification-with-create-ml-coreml3-and-skafos-part-2-734f1ea2f6e?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/734f1ea2f6e</guid>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[skafos]]></category>
            <category><![CDATA[activity-classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Fri, 27 Sep 2019 17:54:00 GMT</pubDate>
            <atom:updated>2019-10-21T16:17:19.744Z</atom:updated>
            <content:encoded><![CDATA[<p>iOS App Integration</p><blockquote>This tutorial leverages a model built using Create ML, now available as part of Xcode on macOS 10.15, Catalina.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*shNq6IadIcQK44g9.jpg" /><figcaption>Fig 1. <a href="https://cdn.mos.cms.futurecdn.net/UchWW83pHgDe7jUZyUk2bM-768-80.jpg">https://cdn.mos.cms.futurecdn.net/UchWW83pHgDe7jUZyUk2bM-768-80.jpg</a></figcaption></figure><p><a href="https://medium.com/skafosai/activity-classification-with-create-ml-coreml3-and-skafos-part-1-8f130b5701f6">In a previous post</a>, we trained and tested an activity classification model in Create ML. In this post, we will integrate the Core ML model in an iOS application, using Core Motion sensor data to classify your activity as either <em>standing</em>, <em>sitting</em>, or <em>moving</em>.</p><h3>Getting Started</h3><p>Our goal is to create an app that displays the current activity classification on the main user interface. Under the hood, the app will stream through sensor data reported by the <a href="https://developer.apple.com/documentation/coremotion">Core Motion Framework</a> and forward it to the Core ML model for activity classification.</p><h4>Setup the iOS App</h4><p>At <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>, we’ve built a few <a href="https://github.com/skafos/example-ml-apps">example iOS applications</a> that are powered by machine learning. However, <a href="https://github.com/skafos/ActivityClassifier">our activity classification app</a> is built to use models trained by Turi Create, and the one we are working with here was built with Create ML. This is no problem at all! We’ll make a few code adjustments to get it working with our new model.</p><ol><li><strong>Grab the code from Github</strong></li></ol><pre>$ git clone git@github.com:skafos/ActivityClassifier.git<br>$ cd ActivityClassifier/</pre><p>2. <strong>Pods</strong></p><p>We use <a href="https://cocoapods.org/">CocoaPods</a> to manage some dependencies. Run the following commands:</p><pre>$ pod repo update &amp;&amp; pod install<br>$ open ActivityClassifier.xcworkspace/</pre><p>This will install some frameworks and launch Xcode!</p><h4>Replace the Model</h4><p>In the Create ML app, drag the output to your desktop and rename the artifact ActivityClassifier.mlmodel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*vsn8B4d4LOHSjFbvSePWNg.png" /><figcaption>Fig 2. CoreML output.</figcaption></figure><p>We also need to remove the existing model in the app and replace it with the one we just dragged out of Create ML .(<em>This is the newly trained model we are trying to test!</em>) Do this in Xcode by performing the following steps:</p><ul><li><strong>Delete</strong> the file called ActivityClassifier.mlmodel on the left-side file browser. Do this by right-clicking and selecting “<em>Delete</em>” and “<em>Move to trash</em>”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*2ATNwNKIdT7iSe9PjqkBqw.png" /><figcaption>Fig 3. Xcode left panel file browser.</figcaption></figure><ul><li><strong>Drag-n-drop</strong> the newly created artifact from your desktop to the same location and select “<em>Create groups</em>” when Xcode prompts you. Model replaced!</li></ul><h4>Modify the View Controller</h4><p>Now we need to make a few code adjustments in MainViewController.swift to make everything work together.</p><p><strong>Class Initialization</strong>: Update the view controller class initialization section to match below (this is everything above the viewDidLoad() method). Here we break out each sensor component (acceleration + rotation X, Y, Z) into separate <a href="https://developer.apple.com/documentation/coreml/mlmultiarray"><em>MLMultiArray</em></a><em>s. </em>Delete or comment out the existing code and replace it with what’s below.</p><pre><strong>class</strong> ViewController: UIViewController {</pre><pre>// Define some ML Model constants for the recurrent network<br>  <strong>struct</strong> ModelConstants {<br>    <strong>static</strong> <strong>let</strong> numOfFeatures = 6<br>    // Must be the same value you used while training<br>    <strong>static</strong> <strong>let</strong> predictionWindowSize = 30<br>    // Must be the same value you used while training<br>    <strong>static</strong> <strong>let</strong> sensorsUpdateFrequency = 1.0 / 10.0<br>    <strong>static</strong> <strong>let</strong> hiddenInLength = 20<br>    <strong>static</strong> <strong>let</strong> hiddenCellInLength = 200<br>  }</pre><pre>// Initialize the model, layers, and sensor data arrays<br>  <strong>private</strong> <strong>let</strong> classifier = ActivityClassifier()<br>  <strong>private</strong> <strong>let</strong> modelName:String = &quot;ActivityClassifier&quot;<br>  <strong>var</strong> currentIndexInPredictionWindow = 0</pre><pre><strong>let</strong> accX = <strong>try</strong>? MLMultiArray(<br>    shape: [ModelConstants.predictionWindowSize] <strong>as</strong> [NSNumber],<br>    dataType: MLMultiArrayDataType.double)</pre><pre><strong>let</strong> accY = <strong>try</strong>? MLMultiArray(<br>    shape: [ModelConstants.predictionWindowSize] <strong>as</strong> [NSNumber], <br>    dataType: MLMultiArrayDataType.double)</pre><pre><strong>let</strong> accZ = <strong>try</strong>? MLMultiArray(<br>    shape: [ModelConstants.predictionWindowSize] <strong>as</strong> [NSNumber],<br>    dataType: MLMultiArrayDataType.double)</pre><pre><strong>let</strong> rotX = <strong>try</strong>? MLMultiArray(<br>    shape: [ModelConstants.predictionWindowSize] <strong>as</strong> [NSNumber],<br>    dataType: MLMultiArrayDataType.double)</pre><pre><strong>let</strong> rotY = <strong>try</strong>? MLMultiArray(<br>    shape: [ModelConstants.predictionWindowSize] <strong>as</strong> [NSNumber],<br>    dataType: MLMultiArrayDataType.double)</pre><pre><strong>let</strong> rotZ = <strong>try</strong>? MLMultiArray(<br>    shape: [ModelConstants.predictionWindowSize] <strong>as</strong> [NSNumber],<br>    dataType: MLMultiArrayDataType.double)</pre><pre><strong>var</strong> currentState = <strong>try</strong>? MLMultiArray(<br>    shape: [(ModelConstants.hiddenInLength +  <br>      ModelConstants.hiddenCellInLength) <strong>as</strong> NSNumber],<br>    dataType: MLMultiArrayDataType.double)</pre><pre>// Initialize CoreMotion Manager<br>  <strong>let</strong> motionManager = CMMotionManager()</pre><pre>// Initialize the label that will get updated<br>  <strong>private</strong> <strong>lazy</strong> <strong>var</strong> label:UILabel = {<br>    <strong>let</strong> label           = UILabel()<br>    label.text          = &quot;&quot;<br>    label.font          = label.font.withSize(35)<br>    label.textAlignment = .center<br>    label.numberOfLines = 0<br>    label.lineBreakMode = .byWordWrapping<br>    label.textColor = .white<br>    <strong>self</strong>.view.addSubview(label)<br>    <strong>return</strong> label<br>  }()</pre><p><strong>Stop Motion Data</strong>: Update the stopDeviceMotion()function to reset the currentState array when motion data streaming stops. Again, delete the existing code and replace it with what follows.</p><pre><strong>func</strong> stopDeviceMotion() {<br>  <strong>guard</strong> motionManager.isDeviceMotionAvailable <strong>else</strong> {<br>    debugPrint(&quot;Core Motion Data Unavailable!&quot;)<br>    <strong>return<br>  </strong>}</pre><pre>// Stop streaming device data<br>  motionManager.stopDeviceMotionUpdates()</pre><pre>// Reset some parameters<br>  currentIndexInPredictionWindow = 0<br>  currentState = <strong>try</strong>? MLMultiArray(<br>    shape: [(ModelConstants.hiddenInLength +<br>      ModelConstants.hiddenCellInLength) <strong>as</strong> NSNumber],<br>    dataType: MLMultiArrayDataType.double)<br>}</pre><p><strong>Add Motion Data To Arrays: </strong>Update the addMotionDataSampleToArray() function to assign motion data to the proper array.</p><pre>func addMotionDataSampleToArray(motionSample: CMDeviceMotion) {<br>  // Using global queue for building prediction array<br>  DispatchQueue.global().async {<br>    self.rotX![self.currentIndexInPredictionWindow] = motionSample.rotationRate.x as NSNumber<br>    self.rotY![self.currentIndexInPredictionWindow] = motionSample.rotationRate.y as NSNumber<br>    self.rotZ![self.currentIndexInPredictionWindow] = motionSample.rotationRate.z as NSNumber<br>     self.accX![self.currentIndexInPredictionWindow] = motionSample.userAcceleration.x as NSNumber<br>     self.accY![self.currentIndexInPredictionWindow] = motionSample.userAcceleration.y as NSNumber<br>      self.accZ![self.currentIndexInPredictionWindow] = motionSample.userAcceleration.z as NSNumber<br>            <br>       // Update prediction array index<br>       self.currentIndexInPredictionWindow += 1<br>            <br>       // If data array is full - execute a prediction<br>       if (self.currentIndexInPredictionWindow == ModelConstants.predictionWindowSize) {<br>         // Move to main thread to update the UI<br>         DispatchQueue.main.async {<br>           // Use the predicted activity<br>           self.label.text = self.activityPrediction() ?? &quot;N/A&quot;<br>         }<br>         // Start a new prediction window from scratch<br>         self.currentIndexInPredictionWindow = 0<br>       }<br>     }<br>   }</pre><p><strong>Make Predictions</strong>: Update the activityPrediction()function to generate an activity prediction from the feature arrays and re-assign the state vector.</p><pre><strong>func</strong> activityPrediction() -&gt; String? {<br>  // Perform prediction<br>  <strong>let</strong> modelPrediction = <strong>try</strong>? classifier.prediction(<br>    acceleration_x: accX!,<br>    acceleration_y: accY!,<br>    acceleration_z: accZ!,<br>    rotation_x: rotX!,<br>    rotation_y: rotY!,<br>    rotation_z: rotZ!,<br>    stateIn: currentState)</pre><pre>// Update the state vector<br>  currentState = modelPrediction?.stateOut</pre><pre>// Return the predicted activity<br>  <strong>return</strong> modelPrediction?.label<br>}</pre><h3>Build and Run</h3><p>Let’s play with the activity classifier!</p><ul><li>Plug in an iOS device OR use one of the simulator devices listed in the top left of the Xcode window:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*VcwY59B2woaHO4JLbpasHQ.png" /><figcaption>Fig 4. Xcode build device.</figcaption></figure><ul><li>Hit the play button in the top left corner to build and run. The app may take a few minutes to build.</li></ul><p>Assuming everything builds properly, you should see the app install and open on the device. <strong>And here we have it: the app shows the current activity classification: moving, standing, or sitting.</strong> Try shaking it or setting it on the table to see predictions change.</p><h3>Skafos Integration</h3><p>As is typical with all software, we periodically release updates that improve the application experience for users, or fix existing bugs. When it comes to machine learning, this is even more true. All models degrade over time, requiring frequent updates (retraining) to stay fresh. Additionally, say we wanted to expand our classifier model to also predict whether or not someone is walking or running. Updating these models in your app, as often as you want to, is made easy with Skafos.</p><p><a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> allows you to deploy models over-the-air to your applications managed by two environments: <strong>Dev</strong> and <strong>Prod. </strong>This means you don’t have to re-release the app to the App Store to deliver a new model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m0KA1cgz74aPvZLA.png" /><figcaption>Fig 5. Model environment cards.</figcaption></figure><p>When your app is running in DEBUG mode (i.e. during development or testing), Skafos will ship the model deployed to the <strong>Dev</strong> environment. Otherwise, Skafos will ship the model deployed to the <strong>Prod</strong> environment. This way you can test a new model on a subset of users before releasing it to your entire user base.</p><p>If you followed the steps above, your application already has most of the code to support this, we just need to do a few things.</p><h4><a href="https://dashboard.skafos.ai/sign-up">Create an Account</a></h4><p><em>If you already have a </em><a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml"><em>Skafos</em></a><em> account, you can skip this part!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*Gw-aImpxprmRR94z.png" /><figcaption>Fig 6. Sign up for Skafos.</figcaption></figure><h4>Create an App &amp; Model in the Dashboard</h4><p>Once you’ve logged into the dashboard, create a <em>New App Integration </em>and name it ActivityClassification. Create a New Model and name it ActivityClassifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*FKQgOP-X00fXS6kM.png" /><figcaption>Fig 7. Create a new App and new Model.</figcaption></figure><p>You will be taken to the integration tab. This page has detailed instructions on how to get an iOS app setup to receive model deliveries from <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>. We already did most of this earlier, so feel free to skim it. The only part you need to follow is the section about <em>Environment Keys</em>.</p><h3>Delivering New Models</h3><p>As noted above, ML models need to be periodically retrained as new data is collected and the world continues to change around us. We can use <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> to manage, deploy, and monitor updated models over-the-air. Remember back when you had to manually drag the .mlmodel and add drag it into Xcode? No more! Skafos can deliver your model updates remotely from the dashboard.</p><p>To do this, you simply need to upload your new models to the Skafos dashboard. There are three ways to do this and each is described below. Regardless of the method, once you’ve uploaded a model, it will appear in the model versions table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e9tYmnDsJDboxzYd.png" /><figcaption>Fig 8. Model versions table.</figcaption></figure><h4>Upload in the Dashboard</h4><p>Under the “Manage” tab in Skafos, you will see a section near the bottom designated for model drag-n-drop. Zip up the.mlmodel file, drag onto the screen and enter a description. <em>Below is an example of uploading a zip archive to the dashboard!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pyGaxdsAY-ClNqLz.png" /><figcaption>Fig 9. Model upload on the dashboard.</figcaption></figure><h4>Upload with the Python SDK</h4><p><a href="http://sdk.skafos.ai/en/stable/install.html">Install the Python SDK</a> and use it to upload the model artifact. Typically, this method is most useful if you are training a model in a Python environment, so I will omit the details here. But read through the rest of the <a href="http://sdk.skafos.ai/en/stable/index.html">documentation</a> if you’re interested in using this tool!</p><h4>Upload directly from Create ML</h4><p>Create ML offers several ways to distribute Core ML models by clicking the Share button — including Skafos!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*bfy4ZNiVZlFMYC1KJUybxQ.png" /></figure><p>To enable this functionality, <a href="https://docs.skafos.ai/macos-app?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">simply download our share sheet app</a> and follow the instructions. This will enable you to upload models directly from Create ML to Skafos.</p><p>When you are ready — deploy your newly uploaded model to the app by hitting the “Deploy” button on the manage tab of the dashboard. <strong>Select the Dev environment</strong>. Re-run the app and you should see the new model create predictions based on input text.</p><p>Now you’ve got your activity classifier pipeline setup end-to-end. You have a model, an iOS application, and a seamless way to update models in your app as you retrain. Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=734f1ea2f6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Activity Classification with Create ML, CoreML3, and Skafos: Part 1]]></title>
            <link>https://medium.com/@tyler.hutcherson/activity-classification-with-create-ml-coreml3-and-skafos-part-1-8f130b5701f6?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f130b5701f6</guid>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[create-ml]]></category>
            <category><![CDATA[activity-classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Fri, 27 Sep 2019 17:45:50 GMT</pubDate>
            <atom:updated>2019-10-09T15:17:13.138Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Preparation and Model Training</em></p><blockquote>This tutorial leverages Create ML, now available as part of Xcode on macOS 10.15, Catalina.</blockquote><p>Your iPhone is more powerful than you probably realize. <em>How is that even possible?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B3Gdz5s2oXwnSxON.jpg" /><figcaption>Fig 1. Apple’s iPhone.</figcaption></figure><p>When Apple announced improvements to the <a href="https://developer.apple.com/documentation/coremotion">Core Motion Framework</a> at the <a href="https://developer.apple.com/videos/play/wwdc2016/713/">2016 WWDC</a>, it sparked a wave of data-driven applications centered around health, fitness, and other motion-related tasks. Core Motion reports accelerometer, gyroscope, and other sensor readings from the on-device hardware buried in iPhones, iPads, and Apple Watches.</p><p>With a means to obtain rich motion data on-device, Apple’s release of CreateML, including the anticipated activity classification toolkit, brings new capabilities to developers. For example, building an app that tracks your distinct exercises during a workout or counts the number of frisbee throws you make in a game is now easier than ever.</p><p><em>This is part 1 of a two-part series on activity classification at the edge. In this post, we will prepare sensor data obtained from Apple watches and train an activity classifier model in Create ML.</em></p><h3>Gathering Training Data</h3><p>Building an activity classifier model requires labeled sensor data. On a mobile device, two commonly used sensors are the <a href="https://en.wikipedia.org/wiki/Accelerometer">accelerometer</a>, tracking (x, y, z) acceleration, and <a href="https://en.wikipedia.org/wiki/Gyroscope">gyroscope</a>, tracking rotation around the (x, y, z) axes. Below is an example of this kind of data captured while walking for 3 seconds:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/0*yfzNpaan0M8Eqft6.png" /><figcaption>Fig 2. <a href="https://github.com/apple/turicreate/raw/master/userguide/activity_classifier/images/walking.png">https://github.com/apple/turicreate/raw/master/userguide/activity_classifier/images/walking.png</a></figcaption></figure><p>Depending on the use-case, gathering this type of data is manageable thanks to some apps like <a href="https://apps.apple.com/us/app/sensorlog/id388014573">SensorLog</a> that allow you to collect, label, and export. <a href="https://medium.com/skafosai/activity-classification-for-watchos-part-1-542d44388c40">In a previous blog post</a>, we walked through the process of collecting data through the app with a few volunteers wearing Apple watches. The data contains 3 basic activities: <em>sitting</em>, <em>standing</em>, <em>moving</em>.</p><p>For this example,<strong> </strong>our goal is to use the previously collected data to build a model in Create ML that can recognize and differentiate these activities/motions from one another.</p><h4>Data Download</h4><p>Download the raw CSV dataset <a href="https://s3.amazonaws.com/skafos.example.data/ActivityClassifier/cleaned/watch_activity_data.csv"><strong>here</strong></a>.</p><p>If you want to <em>skip over the data cleaning section</em>, download a zip archive of the fully-prepared dataset <a href="https://s3.amazonaws.com/skafos.example.data/ActivityClassifier/cleaned/WatchData.zip"><strong>here</strong></a>. You will be able to use the data instantly in Create ML.</p><p>If you want to see how to prepare the raw data for Create ML, keep reading!</p><h4>Data Cleaning and Prep for Create ML</h4><p>First, we need to break up the large CSV file into training + testing sets. Also, we need to break the data into subfolders where each folder name corresponds to one of the activities mentioned above. Inside each activity-specific folder, we will write out chunks of data broken up by session_id. A single session is one continuous data collection period by a single volunteer.</p><p>Open a Python 3.6+ console in the same directory as the watch_activity_data.csv file and run the following code:</p><pre>import os<br>import turicreate as tc</pre><pre><em># Column name mappings</em><br>cols = {<br>    &quot;motionRotationRateX(rad/s)&quot;: &quot;rotation_x&quot;,<br>    &quot;motionRotationRateY(rad/s)&quot;: &quot;rotation_y&quot;,<br>    &quot;motionRotationRateZ(rad/s)&quot;: &quot;rotation_z&quot;,<br>    &quot;motionUserAccelerationX(G)&quot;: &quot;acceleration_x&quot;,<br>    &quot;motionUserAccelerationY(G)&quot;: &quot;acceleration_y&quot;,<br>    &quot;motionUserAccelerationZ(G)&quot;: &quot;acceleration_z&quot;,<br>    &quot;sessionId&quot;: &quot;session_id&quot;,<br>    &quot;activity&quot;: &quot;activity&quot;<br>}<br>csv_cols = [&quot;rotation_x&quot;, &quot;rotation_y&quot;, &quot;rotation_z&quot;, &quot;acceleration_x&quot;, &quot;acceleration_y&quot;, &quot;acceleration_z&quot;]</pre><pre><em># Load csv data and rename columns<br></em>sf = tc.SFrame.read_csv(&quot;watch_activity_data.csv&quot;)[list(cols.keys())].rename(cols)</pre><pre><em># Remove missing activies<br></em>sf = sf[sf[&#39;activity&#39;] != &#39;&#39;]<br>acts = sf[&#39;activity&#39;].unique()</pre><pre><em># Split data into training and testing sets<br></em>train, test = tc.activity_classifier.util.random_split_by_session(sf, session_id=&#39;session_id&#39;, fraction=0.8)</pre><pre><em># Write out training data<br></em>path = &quot;train/&quot;<br>os.mkdir(path)</pre><pre>for a in acts:<br><em>  # Check for folder<br></em>  cls_path = path + a + &quot;/&quot;<br>  if not os.path.exists(cls_path):<br>    os.mkdir(cls_path)</pre><pre><em># Split data by activity &amp; session and write to file<br></em>  sf_act = train[train[&#39;activity&#39;] == a]<br>  for s in sf_act[&#39;session_id&#39;].unique():<br>    sf_act[sf_act[&#39;session_id&#39;] == s][csv_cols].save(cls_path + str(s) + &quot;.csv&quot;)</pre><pre><em># Write out testing data<br></em>path = &quot;test/&quot;<br>os.mkdir(path)</pre><pre>for a in acts:<br><em>  # Check for folder<br></em>  cls_path = path + a + &quot;/&quot;<br>  if not os.path.exists(cls_path):<br>    os.mkdir(cls_path)</pre><pre><em># Split data by activity &amp; session and write to file<br></em>  sf_act = test[test[&#39;activity&#39;] == a]<br>  for s in sf_act[&#39;session_id&#39;].unique():<br>    sf_act[sf_act[&#39;session_id&#39;] == s][csv_cols].save(cls_path + str(s) + &quot;.csv&quot;)</pre><p>Afterward, you should have a file structure that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/169/1*b8wOO73qFBzYseM2mdIdgA.png" /><figcaption>Fig 3. Data file structure for Create ML.</figcaption></figure><p>The file train/sitting/9.csv is sensor data in the training set, recorded while sitting, from one volunteer during a single session. You would also have this structure if you skipped ahead and downloaded/unzipped the <a href="https://s3.amazonaws.com/skafos.example.data/ActivityClassifier/cleaned/WatchData.zip">pre-processed dataset</a>. Now we are ready for model training in Create ML.</p><h3>Training in Create ML</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*_gYiTGVfbFDf3ow0.png" /><figcaption>Fig 4. Create ML.</figcaption></figure><p>Open the Create ML application and select the Activity Classifier from the templates menu. Fill in a <em>Project Name</em>, <em>Author</em>, <em>License</em>, and <em>Description</em> as you see fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loKN8ZXoP5qG16CGEvNtgA.png" /><figcaption>Fig 5. Create ML Activity Classifier.</figcaption></figure><p>After hitting “Next”, you will be shown an interface with several panels and options. <strong>First, select the data inputs.</strong> Select folders for both training data and testing data and select all 6 feature columns (autodetected by Create ML): acceleration_x, acceleration_y, acceleration_z, rotation_x, rotation_y, rotation_z. This will launch a Finder window where you can select the folders created earlier: train/ and test/.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMytg6Di-LGZxMBmZCC2GA.png" /><figcaption>Fig 6. Activity Classifier data inputs.</figcaption></figure><p>Before we can start training, we have to define a few parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*J6WcmWZtAqwvAAP3leWleQ.png" /><figcaption>Fig 7. Activity Classifier parameters.</figcaption></figure><p><strong>Maximum Iterations</strong> and<strong> Batch Size</strong> (30 and 64 respectively), were determined after some trial and error. If you are unsure, you can start with these values and adjust as needed. <strong>Sample Rate </strong>refers to<strong> </strong>the number of observations recorded per second by the on-device hardware. The training data was collected at <strong>10Hz</strong>, so we set the sample rate to 10. <strong>Prediction Window Size</strong> is the number of samples the model will use to make a classification. With a value of 30 as shown above, the model uses 30 samples (3 seconds of data) to classify an activity.</p><p>Now it’s time to train the Activity Classifier model! <strong>Hit the Train “play button” in the top left corner of the app. </strong>Be prepared to wait several minutes while this model trains. You should see progress displayed under the “Activity” panel like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCnTEk_4_xIPrNecxb_rJw.png" /><figcaption>Fig 8. Activity Classifier training accuracy progress.</figcaption></figure><h4>What’s Happening?</h4><p>Create ML abstracts all training details from the user. Therefore, it’s tough to infer which algorithm is used under-the-hood. Because <a href="https://github.com/apple/turicreate">Turi Create</a> (TC) and Create ML are both flagship machine learning tools from Apple, looking to Turi Create for more details is a good place to start. Further, the model parameters presented by Create ML are <em>equivalent</em> to those supported by the Turi Create library. As a part of their toolkit, TC wrote an <a href="https://github.com/apple/turicreate/blob/master/userguide/activity_classifier/how-it-works.md">excellent user guide</a> on time-series classification problems handled by deep learning models. Read the guide to learn more about how activity classifier models are constructed and get a better sense of what Create ML is doing under-the-hood.</p><h3>Testing the Model</h3><p>Create ML provides a few ways for us to evaluate the quality of the model we just trained.</p><p>As displayed in Fig 8 above, the report of training accuracy over time indicates that the model <em>steadily improved on training data</em> from start to finish. However, this is only somewhat useful since what we are curious about is how the model performs on holdout data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qOH8J6dMbbIDNBRZSGNmcw.png" /><figcaption>Fig 9. Activity Classifier testing metrics.</figcaption></figure><p>The true test of model performance is on holdout testing data! After training, Create ML automatically computes <a href="https://towardsdatascience.com/precision-vs-recall-386cf9f89488">precision and recall</a> metrics on the testing dataset we selected (shown in Fig 9).</p><p>In Figure 9, we observe that the model is pretty good at differentiating movement from the stationary activities: sitting and standing. This seems reasonable and suggests that the model wasn’t able to pick up enough information to distinguish sitting from standing. <em>Further tuning of model parameters like Prediction Window Size, or higher volumes of training data could alleviate this gap in performance.</em></p><h4>Evaluating on External Data</h4><p>Create ML also provides an interface to live-test the model. Give it a shot by clicking on the CoreML artifact in the upper right corner. It will take you to a screen that looks like this, where you can drag in some sensor data with the same features used during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjbGldwSQ0h_wf9JWaIUKQ.png" /><figcaption>Fig 10. Create ML model testing interface.</figcaption></figure><p>In Figure 10 above, the model classified the first 3 seconds of sensor observations (highlighted portion) as “moving” with a confidence of 57%.</p><h3>The Output</h3><p>After all training and testing, Create ML is ready to export the model to <a href="https://developer.apple.com/documentation/coreml">Core ML</a> format, enabling it to run on any iOS device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*vsn8B4d4LOHSjFbvSePWNg.png" /><figcaption>Fig 11. Core ML artifact.</figcaption></figure><p>In the next post, we will integrate this artifact in an iOS app capable of determining your current activity from Core Motion data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f130b5701f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying Spam Messages with Create ML, CoreML3, and Skafos: Part 2]]></title>
            <link>https://medium.com/@tyler.hutcherson/classifying-spam-messages-with-create-ml-coreml3-and-skafos-part-2-6fe3177c5d62?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fe3177c5d62</guid>
            <category><![CDATA[skafos]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 20:40:48 GMT</pubDate>
            <atom:updated>2019-10-21T16:21:41.267Z</atom:updated>
            <content:encoded><![CDATA[<p><em>iOS Application and Skafos Deployment</em></p><blockquote>This tutorial leverages a model built using Create ML, now available as part of Xcode on macOS 10.15, Catalina.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qJEBJbsFntzyk4tzJ9UZ6w.png" /></figure><p>This is Part 2 of a two-part series. <a href="https://medium.com/skafosai/classifying-spam-messages-with-create-ml-coreml3-and-skafos-part-1-137e7f6cf037">Part 1: Classifying Spam Messages with Create ML, CoreML3, and Skafos</a> was an introduction to Text Classification in Create ML. We trained a model to identify “spammy” text and tested it out all within the Create ML interface.</p><p>However, even the world’s best model creates absolutely zero value when left on the shelf. That’s why in this post we will show you how to integrate that model in a sample iOS application, leveraging <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> for deployment and monitoring.</p><h3>Getting Started</h3><p>Our goal is to create an iOS app that uses the spam classifier model trained with Create ML back in <a href="https://medium.com/skafosai/classifying-spam-messages-with-create-ml-coreml3-and-skafos-part-1-137e7f6cf037">Part 1</a>. While you can also train text classifiers with the Turi Create library (<a href="https://github.com/skafos/colab-example-models#text-classification">see examples here</a>), the artifacts produced by the Create ML tool are slightly different and require different Swift code to integrate into the app.</p><p>To build an iOS app, we will leverage Apple’s <a href="https://developer.apple.com/documentation/naturallanguage">Natural Language Framework</a> in addition to <a href="https://developer.apple.com/documentation/coreml">Core ML</a> to integrate the spam classifier</p><h4>Create the App</h4><p>At <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>, we’ve taken time to build out a few example iOS applications that are powered by machine learning. However, our text classification app is built to use models trained by Turi Create. No problem at all! We will make a few code adjustments to get it working with our spam classifier.</p><ol><li><strong>Grab the code from Github</strong></li></ol><pre>$ git clone git@github.com:skafos/TextClassificationIOS.git<br>$ cd TextClassificationIOS/</pre><p>2. <strong>Update the Podfile</strong></p><p>We will use <a href="https://cocoapods.org/">CocoaPods</a> to manage some dependencies. A small change is required if you’re running on Xcode 11+ (with Catalina OS). Open the Podfile and change the Skafos version to the following:</p><pre>pod &#39;Skafos&#39;, &#39;~&gt; 4.1.1-beta&#39;</pre><p>Otherwise, leave it be and run the following commands:</p><pre>$ pod install<br>$ open TextClassification.xcworkspace/</pre><p>This will launch Xcode!</p><h4>Replace the Model</h4><p>In the Create ML app, drag the output to your desktop and rename the artifact TextClassifier.mlmodel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*f9wljRlwPgSxaBSd78gaTg.png" /></figure><p>We need to remove the existing model that came with the app. In Xcode, delete the file called TextClassifier.mlmodel under the Resources/ folder on the left-side file browser. Do this by right-clicking and selecting “<em>Delete</em>” and “<em>Remove all references</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*KIGR5JmzsQHetmI99LHjHw.jpeg" /></figure><p>Now drag the newly created artifact from your desktop to the same location and select “<em>Create groups</em>” when Xcode prompts you. BOOM! Model replaced!</p><h4>Modify the View Controller</h4><p>Now we need to make a few code adjustments in MainViewController.swift to make everything work together.</p><p><strong>Imports</strong>:<strong> </strong>include the<strong> </strong>NaturalLanguage<strong> </strong>framework to the list of imports.</p><pre>import Foundation<br>import UIKit<br>import Skafos<br>import CoreML<br>import NaturalLanguage<br>import Snapkit</pre><p><strong>Class Initialization</strong>: wrap the Core ML text classifier as an NLModel so the Natural Language framework can use it.</p><pre>class MainViewController : UIViewController {</pre><pre>// Core ML model setup<br>  private let classifier:TextClassifier! = TextClassifier()<br>  private let modelName:String = &quot;TextClassifier&quot;</pre><pre>// Natural Language model wrapper<br>  private lazy var textClassifier:NLModel! = try! NLModel(mlModel: classifier.model)</pre><pre>...<br>...<br>...</pre><p><strong>Skafos Model Updates</strong>: update the viewDidAppear method.</p><pre>override func viewDidAppear(_ animated: Bool) {<br>  super.viewDidAppear(animated)<br>  let _ = field.becomeFirstResponder()</pre><pre>  // Skafos load<br>  Skafos.load(asset: self.modelName) { (error, asset) in<br>    // Log the asset in the console<br>    console.info(asset)<br>    guard error == nil else {<br>      console.error(&quot;Error: \(String(describing: error))&quot;)<br>      return<br>    }<br>    if let model = asset.model {<br>      // Assign the updated model<br>      self.classifier.model = model<br>    }<br>  }<br>  self.textClassifier = try! NLModel(mlModel: self.classifier.model)<br>}</pre><p><strong>Model Inference: </strong>hand off<strong> </strong>all text processing to the NLModel.</p><pre>// replace the initial processText() method with the one below</pre><pre>@objc func processText(_ sender:Any? = nil) {<br>  guard let text = field.text else { return }<br>  let prediction = self.textClassifier.predictedLabel(for: text)!<br>  label.text = &quot;Classification: \(prediction)&quot;<br>}</pre><blockquote>Now you have a functional text classification app in place!</blockquote><h3>Build and Run</h3><p>Let’s play with the spam classifier!</p><ul><li>Plug in an iOS device running 13+ OR use one of the simulator devices listed in the top left of the Xcode window:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*EJ4-O8cHsRxIkai43_HQvg.png" /><figcaption>Select a build device to run the app</figcaption></figure><ul><li>Hit the play button in the top left corner to build and run. The app may take a few minutes to build.</li></ul><p>Assuming everything builds properly, you should see the app install and open on the device. <strong>And here we have it: the app allows you to type in some text and get a prediction as to whether it’s considered spam or not.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*OcpmzxDEgMQKJmyFRXQFVA.png" /></figure><p>With the provided code and logic available, you should be able to integrate a Core ML text classifier into any iOS solution.</p><h3>Skafos Integration</h3><p><a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> allows you to deploy models over-the-air to your applications managed by two environments: <strong>Dev</strong> and <strong>Prod. </strong>This means you don’t have to re-release the app to the App Store to deliver a new model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2djurRmsa3uDPk7Z.png" /></figure><p>When your app is running in DEBUG mode (i.e. during development or testing), Skafos will ship the model deployed to the <strong>Dev</strong> environment. Otherwise, Skafos will ship the model deployed to the <strong>Prod</strong> environment. This way you can test a new model on a subset of users before releasing it to your entire user base.</p><p>Your application already has most of the code to support this, we just need to do a few things.</p><h4><a href="https://dashboard.skafos.ai/sign-up">Create an Account</a></h4><p><em>If you already have a </em><a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml"><em>Skafos</em></a><em> account, you can skip this part!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*gbPsHhNQyDuJvTnN.png" /><figcaption>Fig 5. Environment cards on the dashboard.</figcaption></figure><h4>Create an App &amp; Model in the Dashboard</h4><p>Once you’ve logged into the dashboard, create a <em>New App Integration </em>and name it TextClassification. Create a New Model and name it TextClassifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*IACYy12upG80P8WTunnGRg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*e3UWULcWRlhSgamUNn55SQ.png" /></figure><p>You will be taken to the integration tab. This page has detailed instructions on how to get an iOS app setup to receive model deliveries from <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>. We already did most of this earlier, so feel free to skim it. The only part you need to follow is the section about the <strong>Environment Keys</strong>.</p><h4>Set Environment Keys</h4><p>Open up the AppDelegate.swift file in Xcode and look for the section prompting you to enter your environment keys. It should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*OHrp3ijDvpHdGtEX.png" /><figcaption>Fig 6. App delegate code where environment keys are set.</figcaption></figure><p><strong>Your environment keys should be prominently displayed in the Integration Guide. </strong>Add them here. Now the app is ready for <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> model delivery!</p><h3>Delivering New Models</h3><p>ML models need to be periodically retrained as new data is collected and the world continues to change around us. We can use <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> to manage, deploy, and monitor updated models over-the-air. Remember back when you had to manually drag the .mlmodel and add drag it into Xcode? No more! Skafos can deliver your model updates remotely from the dashboard.</p><p>But first, you need to upload the models. There are three ways and each is described below. Regardless of the method, once you’ve uploaded a model, it will appear in the model versions table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YpI-2o0xsX2XSL9pThJAJg.png" /></figure><h4>Upload in the Dashboard</h4><p>Under the “Manage” tab, you will see a section near the bottom designated for model drag-n-drop. Zip up the.mlmodel file, drag onto the screen and enter a description. <em>Below is an example of uploading a zip archive to the dashboard!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HomON1E-HUNndIFs.png" /><figcaption>Fig 10. Model upload on the dashboard.</figcaption></figure><h4>Upload with the Python SDK</h4><p><a href="http://sdk.skafos.ai/en/stable/install.html">Install the Python SDK</a> and use it to upload the model artifact. Typically, this method is most useful if you are training a model in a Python environment, so I will omit the details here. But read through the rest of the <a href="http://sdk.skafos.ai/en/stable/index.html">documentation</a> if you’re interested in using this tool!</p><h4>Upload directly from Create ML</h4><p>Create ML offers several ways to distribute Core ML models by clicking the Share button — including Skafos!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*bfy4ZNiVZlFMYC1KJUybxQ.png" /></figure><p>To enable this functionality, <a href="https://docs.skafos.ai/macos-app?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">simply download our share sheet app</a> and follow the instructions. This will enable you to upload models directly from Create ML to Skafos.</p><p>When you are ready — deploy your newly uploaded model to the app by hitting the “Deploy” button on the manage tab of the dashboard. <strong>Select the Dev environment</strong>. Re-run the app and you should see the new model create predictions based on input text.</p><p>Now you’ve got your text classifier pipeline setup end-to-end. You have a model, an iOS application, and a seamless way to update models in your app as you retrain. Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fe3177c5d62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying Spam Messages with Create ML, CoreML3, and Skafos: Part 1]]></title>
            <link>https://medium.com/@tyler.hutcherson/classifying-spam-messages-with-create-ml-coreml3-and-skafos-part-1-137e7f6cf037?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/137e7f6cf037</guid>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[skafos]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[create-ml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Fri, 20 Sep 2019 20:38:24 GMT</pubDate>
            <atom:updated>2019-10-21T16:19:49.667Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Prep and Model Training in Create ML</em></p><blockquote>This tutorial leverages Create ML, now available as part of Xcode on macOS 10.15, Catalina.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*XgMW0NGbML4IwElT.jpg" /></figure><p>At one time or another, we’ve all received text messages, emails, or DMs that are suspicious, no matter how “convincing” the sender’s attempt. Here’s an example:</p><blockquote>WINNER!! As a valued network customer you have been selected to receive a £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.</blockquote><p>If you’ve ever wondered how tools like <a href="https://www.google.com/gmail/">Gmail</a> now classify incoming messages as spam, removing them from the user’s view, then keep reading! This article is the first of a 2-part series from <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> about text classification at the edge, more specifically <a href="https://en.wikipedia.org/wiki/Email_filtering">spam filtering</a>.</p><p>In this post, we will focus on text data collection and model training in <a href="https://developer.apple.com/videos/play/wwdc2018/703/">Create ML</a>. <a href="https://medium.com/skafosai/classifying-spam-messages-with-create-ml-coreml3-and-skafos-part-2-6fe3177c5d62">In the next post, we will upload the trained Core ML model to Skafos for iOS app delivery.</a></p><h3>Gathering Training Data</h3><p>Building a text classifier model requires training data with text and labels. In our case, the model will be trained to predict whether or not a piece of text is legit, <em>ham</em>, or fake/bad, <em>spam,</em> with some measure of likelihood between 0–100. Fortunately, we found a free dataset that already contains all of the information we need! Let’s get started.</p><p>First, <a href="http://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip">download the dataset for this example</a>. Before you can do anything in Create ML, two things must be done:</p><ul><li>Split the data into training (80%) and testing (20%) sets</li><li>Convert the data to folders representing each class spam and ham, with individual text files for each observation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*Jk30y8E36PCoCfeW0IfPRA.png" /><figcaption>Fig 2. The file structure we need for Create ML training.</figcaption></figure><p>If the model was predicting more than just 2 classes, you would include those data samples in additional subfolders per class.</p><p>Unzip the downloaded archive and launch a Python console inside the spsspamcollection/ folder. Execute the code below, or copy the code to a Python file named prep.py and run as a script: python prep.py. If you need to install the pandas library, use <a href="https://pip.pypa.io/en/stable/installing/">pip</a>: pip install pandas.</p><pre>import os<br>import pandas as pd</pre><pre>df = pd.read_csv(&quot;smsspamcollection/SMSSpamCollection&quot;, sep=&quot;\t&quot;)<br>df.columns = [&quot;label&quot;, &quot;text&quot;]</pre><pre># Split data into training and testing sets by label<br>train = df.sample(frac=0.8, random_state=42)<br>train_spam = train[train.label == &quot;spam&quot;]<br>train_ham = train.drop(train_spam.index)</pre><pre>test = df.drop(train.index)<br>test_spam = test[test.label == &quot;spam&quot;]<br>test_ham = test.drop(test_spam.index)</pre><pre># Create folders for data<br>os.mkdir(&quot;train&quot;); os.mkdir(&quot;test&quot;)<br>os.mkdir(&quot;train/spam&quot;); os.mkdir(&quot;train/ham&quot;)<br>os.mkdir(&quot;test/spam&quot;); os.mkdir(&quot;test/ham&quot;)</pre><pre># Write out data<br>def write_text(path, df):<br>    for i, r in df.iterrows():<br>        with open(path + str(i) + &quot;.txt&quot;, &quot;w&quot;) as f:<br>            f.write(r[&quot;text&quot;])</pre><pre>write_text(&quot;train/spam/&quot;, train_spam)<br>write_text(&quot;train/ham/&quot;, train_ham)<br>write_text(&quot;test/spam/&quot;, test_spam)<br>write_text(&quot;test/ham/&quot;, test_ham)</pre><p>Now you should have a file structure the same as that in Figure 2. We will come back to this data shortly!</p><h3>Training with Create ML</h3><p>Let’s get rolling with the Create ML app!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*DoqcnwN8lKrcJ5y3.png" /><figcaption>Fig 3. Create ML Logo.</figcaption></figure><p>Open the Create ML application and select the Text Classifier from the templates menu. Fill in a <em>Project Name</em>, <em>Author</em>, <em>License</em>, and <em>Description</em> as you see fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZjdJdnfr7XKwCp49qBgjw.png" /><figcaption>Fig 4. Create ML setup.</figcaption></figure><p>After hitting “Next”, you will be presented an interface with several different panels and options. <strong>First, select the data inputs.</strong> Select folders for both training data and testing data. This will launch a Finder window where you can select the folders you created earlier: train/ and test/.</p><blockquote>Leave the validation data field set to “automatic”. At the time of writing this post, that was the only option available in the beta Create ML app.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrDHzGktdG1FE7CgWUg6Uw.png" /><figcaption>Fig 5. Create ML training setup.</figcaption></figure><p>Create ML automatically determines the 2 classes you have (spam and ham) based on the selected data.</p><p>Now select the type of algorithm for Create ML to use when training the ML model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*cZ2oWvbyrHqXLJgEnGTraw.png" /><figcaption>Fig 6. Create ML algorithm choice.</figcaption></figure><p>The two algorithms at the top of the list (<a href="https://en.wikipedia.org/wiki/Principle_of_maximum_entropy">maximum entropy</a> and <a href="https://en.wikipedia.org/wiki/Conditional_random_field">conditional random field</a>) are commonly used to build this type of model; however, the third, called transfer learning, is a new offering with this release of Create ML. <a href="https://developer.apple.com/videos/play/wwdc2019/428/">It was featured in this announcement video back at WWDC 2019</a>. So I’d recommend using this one to see how it performs! We will discuss what’s happening here in more detail below.</p><p>Now it’s time to train the Text Classifier model! <strong>Hit the Train “play button” in the top left corner of the app. </strong>Be prepared to wait several minutes while this model trains.</p><p>You should see progress displayed under the “Activity” panel like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxRPiefMjb58v_1Qb9Pa1g.png" /><figcaption>Fig 7. Training progress.</figcaption></figure><h4>What’s Happening?</h4><p>The selected algorithm, called transfer learning, leverages features learned from pre-built text embedding models that have been trained on large text datasets. <a href="https://machinelearningmastery.com/what-are-word-embeddings/">Here’s a great introductory resource on text embedding</a>.</p><p>The benefit here is that we can quickly use these text embeddings (features) to build a classifier model. The “features” themselves are just vectors of numbers representing the text in a way that a machine can process:</p><blockquote>“hello” = [0.3, 0.45, 0.6, 0.95, 0.56, …….]</blockquote><p>Embeddings encode semantic word relationships, making them more powerful than anything we could garner from our small spam dataset. Words that are used in similar ways throughout a corpus of text end up with similar embeddings.</p><p>In summary, we are transferring textual “knowledge” learned elsewhere and applying it to our specific use-case. <a href="https://towardsdatascience.com/deep-transfer-learning-for-natural-language-processing-text-classification-with-universal-1a2c69e5baa9">Read more about this here.</a></p><h3>Testing the Model</h3><p>Once training has completed, you should see a nice graph of training and validation accuracy over time (Fig 7) and a report of accuracy, <a href="https://towardsdatascience.com/precision-vs-recall-386cf9f89488">precision &amp; recall </a>metrics. Based on the training progress in Figure 7, the model improved over the first few iterations and then leveled out quickly. We don’t see a significant drop in validation accuracy, indicating <strong>our model didn’t </strong><a href="https://en.wikipedia.org/wiki/Overfitting"><strong>overfit</strong></a><strong> the training data</strong>. Therefore, we’d expect it to perform well on our testing dataset!</p><p>In Figure 9 below, we show the final accuracy numbers over training data, validation, and testing sets. The<strong> validation and testing metrics are much more useful </strong>for diagnosing the model’s performance. 99/<em>98% — not too bad!!</em> :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-s2yzFp3H-uSO4wtjeiArw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ut8lBwo4kiNbYUZI9ycnTw.png" /><figcaption>Fig 9. Accuracy report and precision &amp; recall metrics on testing data.</figcaption></figure><h4>Evaluating the Model on External Data</h4><p>Create ML also provides an interface to live-test the model. Give it a shot by clicking on the CoreML artifact in the upper right corner. It will take you to a screen that looks like this, where you can type in some example text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07YnfBkTanBOz_E23UVaWw.png" /><figcaption>Fig 10. Model testing interface.</figcaption></figure><h3>The Output</h3><p>After all training and testing, notice that Create ML went the extra mile and <strong>converted the model to </strong><a href="https://developer.apple.com/documentation/coreml"><strong>Core ML</strong></a><strong> format</strong>, enabling it to run on any iOS device. We will use this artifact in the next post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*f9wljRlwPgSxaBSd78gaTg.png" /><figcaption>Fig 11. Core ML model artifact.</figcaption></figure><p>That’s all for Part 1! <a href="https://medium.com/skafosai/classifying-spam-messages-with-create-ml-coreml3-and-skafos-part-2-6fe3177c5d62">In the next post</a>, we will share a sample iOS app that you can use to integrate your new Core ML artifact and get you set up with <a href="https://www.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> for model delivery and management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=137e7f6cf037" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting Bird Sounds with Create ML, CoreML3, and Skafos: Part 2]]></title>
            <link>https://medium.com/@tyler.hutcherson/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-2-60bb0ba61975?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/60bb0ba61975</guid>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[skafos]]></category>
            <category><![CDATA[create-ml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Mon, 09 Sep 2019 15:21:11 GMT</pubDate>
            <atom:updated>2019-10-21T16:13:55.127Z</atom:updated>
            <content:encoded><![CDATA[<p><em>iOS Application and Skafos Deployment</em></p><blockquote>Announced at WWDC 2019 this year, and new to iOS 13 &amp; macOS 10.15 is the ability to analyze and classify sound with machine learning in Core ML 3!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*q3SawWirEwvkZfqTSNeXMA.png" /></figure><p>This is Part 2 of a two part series. <a href="https://medium.com/skafosai/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-1-f11ddb7fbee1">Part 1: Detecting Bird Sounds with Create ML, CoreML3, and Skafos</a> was an introduction to Sound Classification in Create ML. We trained a model to identify bird sounds in an audio file or stream. In this post, we will integrate that model in an example iOS app, using <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> for deployment and monitoring. Who’s ready to listen for birds?</p><h3>Getting Started</h3><p>At the end of <a href="https://medium.com/skafosai/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-1-f11ddb7fbee1">Part 1</a>, we finished with a Core ML model shown below. In the rest of this post we will work through getting this model wired up with a companion iOS app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*v9kVeOaryQP394Euy28-Yg.png" /></figure><h4>Skafos Setup</h4><p><em>If you already have a Skafos account and an App that uses a sound classification model, you can skip these setup instructions!</em></p><p><a href="https://dashboard.skafos.ai/sign-up"><strong>Create an Account</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*oPzfoVfAxIwigwBg97USQg.png" /><figcaption>Sign up for Skafos if you haven’t already done so.</figcaption></figure><p><strong>Create a new App and Model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*BZrWg8e0kdgZyeil7n_mmQ.png" /><figcaption>Navigate through these steps in the dashboard to create a new Skafos App and Model</figcaption></figure><ul><li>Create a new app integration and name it BirdDetector.</li><li>Inside the app, create a new model and name it BirdSoundClassifier.</li><li>Click on the “Manage” tab.</li></ul><h3>Building the App and Integrating your Model</h3><h4>BirdDetector App Setup</h4><p>In full transparency, I am not an iOS developer by trade; I’m a machine learning engineer. Thankfully, building an app to use a sound classifier model is fairly straightforward thanks to Apple’s <a href="https://developer.apple.com/documentation/soundanalysis/analyzing_audio_to_classify_sounds">Core ML &amp; SoundAnalysis frameworks</a>. We’ve shared the project for you to fetch on Github and use to complete this tutorial:</p><pre>$ git clone git@github.com:skafos/BirdDetector.git<br>$ cd BirdDetector/</pre><p>Use <a href="https://cocoapods.org/">CocoaPods</a> to install the Skafos iOS Framework and then open the xcworkspace to launch the app in Xcode:</p><pre>$ pod install --repo-update<br>$ open BirdDetector.xcworkspace/</pre><p>After opening you should see a few files including the AppDelegate.swift and ViewController.swift. These are the two files that we will investigate further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*hwdCBCqjaUJAdEgR9fYzPA.png" /><figcaption>Xcode project navigator for BirdDetector.</figcaption></figure><h4>Set Environment Keys</h4><p>Skafos allows you to deploy models over-the-air to your applications managed by two environments: <strong>Dev</strong> and <strong>Prod. </strong>This means you don’t have to re-release the app to the App Store to deliver a new model.<strong> </strong>When your app is running in DEBUG mode (i.e. during development or testing), Skafos will ship the model deployed to the Dev environment. Otherwise, Skafos will ship the model deployed to the Prod environment. This way you can test a new model on a subset of users before releasing it to your entire user base. <em>For the remainder of this tutorial, we will be updating to the dev environment only, since we won’t be releasing this app to the app store.</em></p><p>On the “Manage” tab of the <a href="https://dashboard.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos dashboard</a> you can see the two environment cards that are initially empty:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_AkNICHpeSEToUyB99JKQ.png" /></figure><p>Open up the AppDelegate.swift file in XCode and look for the section prompting you to enter your environment keys. It should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*nw32GL7JhuUXjNa1rUHjEQ.png" /></figure><p>Find the two environment keys under the “Integrate” tab on the <a href="https://dashboard.skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos Dashboard</a> and add them here. Now the app is ready for Skafos model delivery!</p><h4>Quick App Overview</h4><p>Next, open up the ViewController.swift file. We won’t dive too deeply into what’s going on under-the-hood here, but I will emphasize a few high-level points to help as you work through the code base:</p><ul><li><strong>Lines 17–23</strong>: We instantiate the initial Core ML model BirdSoundClassifier(), the audio engine and signal analyzer, the results callback, and the processing queue.</li><li><strong>Line 26</strong>: When the UI view first loads, we make a call to Skafos to check and update the model in the app if you’ve deployed a new one on the dashboard. This is captured in the reloadModel() function on line 51.</li><li><strong>Line 40</strong>: When the UI view appears, we immediately start capturing audio from the sensors onboard your iOS device. This is detailed in the startCapturingAudio() function on line 67.</li><li><strong>Line 120</strong>: When the model has classified incoming audio as either hasbird or nobird, the observer processes that result and passes it on to the UI for you to see. There’s even a fun animation built-in on line 105.</li></ul><h4>Build and Run</h4><p>Enough explanation! Let’s play with our bird sound monitoring app!</p><ul><li>Plug in an iOS device running 13+ OR use one of the simulator devices listed in the top left of the Xcode window:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*TntI7Jv1P5Etrj4ok3HPqw.png" /></figure><ul><li>Hit the play button in the top left corner to build and run. The app may take a few minutes to build.</li></ul><p>Assuming you don’t run into any build issues, you should see the app install and open on your device. And here we have it: The app is listening to your environment and scrolling model predictions across your screen bottom to top.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*DUwzcxjZXSFMmqNZHKlIWQ.png" /><figcaption>BirdDetector UI view.</figcaption></figure><p>Now, don’t get caught up by the sophisticated UI/UX :). From my perspective, simpler is better for this use-case because you can swap out any other Create ML sound classifier model and the app will still function just as you see here, but with different class labels. <em>Here’s to standardization! </em>Once you’ve got the ML integration down, you can modify the app to your heart’s content.</p><h3>Delivering New Models</h3><p>Shipping an initial model with the iOS App is recommended. The <a href="https://github.com/skafos/BirdDetector">BirdDetector app</a> comes with a .mlmodel file that contains the Core ML specification for our sound classifier trained in <a href="https://medium.com/skafosai/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-1-f11ddb7fbee1">Part 1</a>. However, ML models need to be periodically retrained as we gather additional data and have increased understanding of the world around us. Fortunately, we can use <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> to manage, deploy, and monitor updated models over-the-air.</p><p>Skafos provides three ways to upload new models, and we will describe them below. Regardless of the method you use, once you’ve uploaded a model, you will see it in the model versions table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drXPrwSpAJJOIOeDYuTTtQ.png" /><figcaption>Model versions table after a model upload to the Skafos dashboard.</figcaption></figure><h4>Upload in the Dashboard</h4><p>Under the “Manage” tab, you will see a section near the bottom designated for model drag-n-drop. Zip up your .mlmodel file, drag onto the screen, and enter a description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zgbZtdElVtE7BNg4REu4w.png" /></figure><h4>Upload with the Python SDK</h4><p><a href="http://sdk.skafos.ai/en/stable/install.html">Install our Python SDK</a> and use it to upload the model artifact. Typically, this method is most useful if you are training a model in a Python environment, so I will omit the details here. But read through the rest of the <a href="http://sdk.skafos.ai/en/stable/index.html">documentation</a> if you’re interested in using this tool!</p><h4>Upload directly from Create ML</h4><p>Create ML offers you a number of helpful ways to distribute your Core ML models by clicking the Share button — including Skafos!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*bfy4ZNiVZlFMYC1KJUybxQ.png" /></figure><p>To enable this functionality, <a href="https://docs.skafos.ai/macos-app?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">simply download our share sheet app</a> and follow the instructions. This will enable you to upload models directly from Create ML to Skafos.</p><h4>Model Retraining</h4><p><strong><em>Why would you retrain another model?</em></strong></p><p>Great question. There are a few reasons why. Say you wanted to improve this model by actually classifying the species of bird instead of merely indicating the presence of bird calls. While <strong>you would need to find a different dataset</strong> than the one we used, you could still use Create ML to build the model as described in <a href="https://medium.com/skafosai/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-1-f11ddb7fbee1">Part 1</a>. Create ML is designed for this sort of thing and makes it extremely easy to iterate, so you can try out different combinations of data and parameters to get the best model for the data set at hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*UYq6vrfkl2cI0nx87R8rQw.png" /><figcaption>Add another model source by clicking the plus sign in the top left corner of the Create ML interface.</figcaption></figure><p>Now you’ve got your sound classifier pipeline connected end-to-end. You have a model, an iOS application, and a seamless way to update models in your app as you retrain. Go conquer the world!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60bb0ba61975" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting Bird Sounds with Create ML, CoreML3, and Skafos: Part 1]]></title>
            <link>https://medium.com/@tyler.hutcherson/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-1-f11ddb7fbee1?source=rss-b4f5908a7cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/f11ddb7fbee1</guid>
            <category><![CDATA[sound-classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[create-ml]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[coreml]]></category>
            <dc:creator><![CDATA[Tyler Hutcherson]]></dc:creator>
            <pubDate>Fri, 06 Sep 2019 14:22:50 GMT</pubDate>
            <atom:updated>2019-09-16T14:55:47.451Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Prep and Model Training in Create ML</em></p><blockquote>Announced at WWDC 2019 this year, and new to iOS 13 &amp; macOS 10.15 is the ability to analyze and classify sound with machine learning in Core ML 3!</blockquote><p>Recently, while looking for open-source sound datasets, I stumbled upon an interesting one: <a href="http://machine-listening.eecs.qmul.ac.uk/bird-audio-detection-challenge/#downloads">Bird Audio Detection Challenge</a> provided by the Machine Listening Lab from the Queen Mary University of London.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZX7SLAf3V1QGxz8sUiuXoQ.jpeg" /></figure><p>Each audio file in the dataset contains some ambient wildlife noise with a handful of them also containing bird sounds (chirps? tweets?). The dataset was provided as part of a challenge where engineers tried to build the best bird sound detection algorithm. <strong>So What? Why is this particular dataset useful?</strong></p><ol><li>A performant model that’s trained to recognize bird sounds in any environment would help automate wildlife monitoring services and contribute to the work done by wildlife conservation services globally.</li><li>The dataset can help us better understand how to build a sound classification model that we could also extend to other use-cases.</li><li>The dataset will allow us to demonstrate building a sound classifier with <a href="https://developer.apple.com/documentation/createml">Create ML</a> and deploying it to a simple iOS app with <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>.</li></ol><p>In the rest of this article, I will walk through the dataset preparation and model training steps using Create ML.</p><h3>Gathering Training Data</h3><p>Download the bird sounds dataset, both the <a href="https://archive.org/download/ff1010bird/ff1010bird_wav.zip">audio files</a> &amp; <a href="https://ndownloader.figshare.com/files/6035814">data labels</a>. Don’t be surprised if it takes several minutes for the files to download!</p><ul><li>Audio Files: ff1010bird_wav.zip</li><li>Data Labels: ff1010bird_metadata.csv</li></ul><p>After unzipping the audio files, you should have one folder that contains ~7690 separate wav recordings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zvGn2KHMED5JiTc08pL7IQ.png" /></figure><p>The data labels csv contains an identifier for each audio file paired with a ground truth label, hasbird, as 1 or 0. Our model will be trained to predict this value. <em>A value of 1 means that the respective audio file contains bird sounds.</em></p><h4>Preparing the Data for Create ML</h4><p>We need to reorganize our audio file structure to look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*4kruORlq5-DZqj8u4MQhhw.png" /><figcaption>New file structure ready for Create ML training.</figcaption></figure><ul><li>Instead of one big folder containing all audio clips, we need to split the dataset into <a href="https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets">training and testing groups</a>.</li><li>Using the data in ff1010bird_metadata.csv, we need to move audio clips into folders grouped by their respective “class”: hasbird &amp; nobird. This way Create ML can infer the class names of the dataset.</li></ul><p>Below is a small Python program, data_prep.py, to move audio files around. Run this script from within the same directory that holds the audio folder and metadata csv.</p><pre>$ python data_prep.py</pre><p><em>You could also choose to do this all manually if you have loads of free time!</em></p><pre>## data_prep.py ##</pre><pre>import os<br>import pandas as pd</pre><pre># Setup Audio File Paths<br>path = &#39;wav/&#39;<br>hasbird = &#39;hasbird/&#39;<br>nobird = &#39;nobird/&#39;<br><br>def make_path(r):<br>    if r[&#39;train&#39;]:<br>        new_path = &#39;train/&#39;<br>    else:<br>        new_path = &#39;test/&#39;<br>    if not os.path.exists(path + new_path):<br>        os.mkdir(path + new_path)<br>    if r[&#39;hasbird&#39;]:<br>        new_path += hasbird<br>    else:<br>        new_path += nobird<br>    if not os.path.exists(path + new_path):<br>        os.mkdir(path + new_path)<br>    return new_path + r[&#39;audio_fn&#39;]</pre><pre># Load Meta Data CSV into a Pandas DF<br>birds = pd.read_csv(&#39;ff1010bird_metadata.csv&#39;)</pre><pre># Assign to train/test and create new path<br>train_ix = birds.sample(frac=0.8, random_state=42).index<br>birds[&#39;train&#39;] = [i in train_ix for i in birds.index]<br>birds[&#39;audio_fn&#39;] = birds[&#39;itemid&#39;].apply(lambda x: str(x) + &#39;.wav&#39;)<br>birds[&#39;new_path&#39;] = birds.apply(lambda r: make_path(r), axis=1)</pre><pre># Move audio files<br>for i, r, in birds.iterrows():<br>    os.rename(path + r[&#39;audio_fn&#39;], path + r[&#39;new_path&#39;])</pre><pre>print(&#39;Done!&#39;)</pre><h3>Training with Create ML</h3><p>Now for the fun part!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*aGZKPvLAkxK3C5e4.png" /></figure><p>Open the Create ML application and select the Sound Classifier from the templates menu. Fill in a <em>Project Name</em>, <em>Author</em>, <em>License</em>, and <em>Description</em> as you see fit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GcYVwI_ROvjOeLN0VqMqXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-ORdNDD72oYwR5K7eOdDg.png" /><figcaption>Create ML Setup.</figcaption></figure><p>After hitting next, you will be presented an interface with several different panels and options. <strong>First, configure the data inputs.</strong> Select files for both training data and testing data. This will launch a Finder window where you can select the train/ folder and test/ folder of audio clips respectively.</p><blockquote>Leave the validation data field set to “automatic”. At the time of writing this post, that was the only option available in the beta Create ML app.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Le2a_ZY6deHtKhZtzA6lwQ.png" /><figcaption>Sound Classifier interface on Create ML.</figcaption></figure><p>Create ML automatically determines the 2 classes we have (hasbirds and nobirds) based on the selected data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zzq-uJr7T5qplR2vJXfIPQ.png" /><figcaption>Training and Testing data loaded. Ready for training to begin.</figcaption></figure><p>Now we are ready to train the Sound Classifier model! <strong>Hit the Train “play button” in the top left corner of the app.</strong></p><p>Then, wait for a few minutes while Create ML is training the model. You should see progress displayed under the “Activity” dropdown panel. <a href="https://github.com/apple/turicreate/blob/master/userguide/sound_classifier/how-it-works.md#how-does-this-work">Read more from our friends at Turi Create</a> about what’s happening here.</p><ul><li><strong>Step 1: Feature Extraction</strong> — Create ML scans through all of the audio files, performing signal processing and automatically extracting features to use during training.</li><li><strong>Step 2: Training</strong> — Create ML iteratively trains the sound classifier neural network layers under the hood.</li></ul><p>Connect your laptop to an external GPU if you want to speed up the training process! In the Finder, right click Create ML app, select “Get Info” and make sure that the app is configured to prefer an external GPU if available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*PGCl2-2ehAaVQJhr_ymD9g.png" /></figure><h3>Testing the Model</h3><p>Once training has completed, <strong>you should see a nice graph of training and validation accuracy over time and a table of </strong><a href="https://towardsdatascience.com/precision-vs-recall-386cf9f89488"><strong>precision &amp; recall </strong></a><strong>metrics</strong>. According to the plot, the model improved over the first few iterations and then leveled out. What’s really great is that we don’t see a significant drop in validation accuracy, indicating <strong>our model didn&#39;t </strong><a href="https://en.wikipedia.org/wiki/Overfitting"><strong>overfit</strong></a><strong> the training data</strong>. <em>Therefore, we expect it to perform well on our testing dataset!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvInYdWnkApsNtGHTdCA7A.png" /><figcaption>Training and validation accuracy for each iteration during model training.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_a69SzKiYpeCAdqg14b5w.png" /><figcaption>Precision and recall metrics post-training.</figcaption></figure><p>On holdout testing data, <strong>the model was 85% accurate</strong> with the following precision &amp; recall breakdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ye92erMwkiBG_LQ9FLSeg.png" /><figcaption>Precision and recall metrics on testing data.</figcaption></figure><p>According to the results, the model is better at properly classifying audio samples without birds. While there is room for improvement, we recommend actually doing something with this model before jumping straight to retraining. Deploy it to an app, test it out in the wild, get feedback… Then, once you’ve gone end-to-end, you can continue to refine (and re-deploy) these new models with <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a>. We will cover that in the next part of this series — stay tuned!</p><h4>Evaluating the Model on External Data</h4><p>Create ML streamlines our ability to train and test a sound classifier model. It also provides an interface to live-test the model on audio picked up by the computer’s microphone or from uploaded audio files. Give it a shot!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvPfAXJF_CPunGQn3ENhqA.png" /><figcaption>Evaluate the model on external data from a file or from the computer’s mic.</figcaption></figure><h4>The Output</h4><p>After all training and testing, notice that Create ML went the extra mile and <strong>converted the model to </strong><a href="https://developer.apple.com/documentation/coreml"><strong>Core ML</strong></a><strong> format</strong>, enabling it to run on any iOS device. We will use this artifact in the next post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*jWtdvCbQ6y5ggh97xKLYmQ.png" /><figcaption>The output Core ML artifact.</figcaption></figure><p>That’s all for Part 1! <a href="https://medium.com/skafosai/detecting-bird-sounds-with-create-ml-coreml3-and-skafos-part-2-60bb0ba61975">In the next post</a>, we will share a sample iOS app that you can use to integrate your new Core ML artifact and get you setup with <a href="https://skafos.ai/?utm_source=medium&amp;utm_medium=content&amp;utm_campaign=create-ml">Skafos</a> for model delivery and management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f11ddb7fbee1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>