<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Prevision.io - Medium]]></title>
        <description><![CDATA[https://prevision.io/ - Medium]]></description>
        <link>https://medium.com/prevision-io?source=rss----72b644e841a2---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Prevision.io - Medium</title>
            <link>https://medium.com/prevision-io?source=rss----72b644e841a2---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:27:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/prevision-io" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[All you need is AutoML … and love! — Prevision.io]]></title>
            <link>https://medium.com/prevision-io/all-you-need-is-automl-and-love-prevision-io-cf7a9828497d?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf7a9828497d</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[automl]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Tue, 25 Jan 2022 10:52:21 GMT</pubDate>
            <atom:updated>2022-01-25T10:52:21.424Z</atom:updated>
            <content:encoded><![CDATA[<h3>All you need is AutoML … and love! — Prevision.io</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*Y-Tc456w5Gx28T6g.png" /></figure><p>Organized by Kaggle Days, last week’s hackathon <a href="https://www.kaggle.com/c/now-you-are-playing-with-power"><strong><em>Now You Are Playing With Power</em></strong></a> concluded. Here is a recap of the project and a bit on how using Prevision.io’s AutoML capability performed. The hackathon:</p><p>How much electricity can a steam turbine generate? In this competition, your aim is to predict the unit load power generation based on the given factors of a steam turbine in specific working environments.</p><p>Steam turbines are still used continuously today due to their successful stability in electric power generation. Other important factors are that the steam conditions have been improving and that steam turbines have been evolving and responding to electric power energy demand by rapidly adopting the latest analysis technologies for their design and applying up-to-date technologies for higher efficiency and reliability to actual products.</p><p>The steam power plant consists of a boiler, steam turbine and generator and other fillers. The boiler produces high-pressure vapor and high temperature. Steam turbine converts heat energy steam to mechanical energy. The generator then converts mechanical energy to electrical energy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*pedorYOCwns0SsH9.jpg" /></figure><p>At normal atmospheric pressure [0.10 MPa] water increases at 100 ° C when the pressure increases, the boiling point of the water increases. When the pressure is increased to 22.12 MPa, and at a temperature of 374ºC, water does not boil but is directly converted into steam. This is called the critical point, and the pressure above this critical point is called supercritical pressure. Supercritical pressure with a temperature equal to or more than 593ºC is called ultra-supercritical pressure.</p><p>Some parts of turbines working in a high-pressure environment face specific challenges over other turbines. The reason why is simple: these parts are exposed to higher pressures and higher temperatures, particularly in the high-pressure (HP) section.</p><p>Coordinated control system (CCS) coordinates boiler and turbine systems to track load demand and ensure safe and economical operation simultaneously.</p><p>The evaluation metric for this competition is Root Mean Squared Error (RMSE). <a href="https://prevision.io/blog/blog-post-3-metrics-for-regression">Click here if you want to know more about regression metrics.</a></p><p>🚀 Only using Prevision.io AutoML capabilities, you would have completed 16th out of 130 submissions !</p><p>Simple Steps To Reproduce:</p><ol><li>Start the competition <a href="https://www.kaggle.com/t/df24a2b9ddc94bbcad6badeacab7f1f5">here</a></li><li>Sign up / Login to <a href="https://cloud.prevision.io/">https://cloud.prevision.io</a></li><li>Upload train.csv and test.csv. You can find data <a href="https://www.kaggle.com/c/now-you-are-playing-with-power/data">here</a>.</li><li>Run experiment using full models + blend</li><li>40 minutes later the experiment is done, make predictions on test.csv file</li><li>Submit to Kaggle. Your score is 1.94 on Kaggle Public Leaderboard (top 17%)</li><li>Add predictions to test file and merge together train &amp; test</li><li>Upload traintest.csv</li><li>Run experiment using full models + blend on this new dataset</li><li>Wait 1h, make predictions on test.csv file</li><li>Submit to Kaggle. Your score is 1.55 on Kaggle Public Leaderboard (top 12%)</li></ol><p>Why use Prevision.io ?</p><p>✔️ In first 2 minutes, with a fast and easy Decision Tree, your score is 2.0 on Kaggle Public Leaderboard (top 19%)</p><p>✔️ No data preparation needed, Prevision.io manages missing values, target encoding, creation of statistics by row and normalization of continuous variables.</p><p>✔️ Prevision.io manages the search for hyper parameters</p><p>✔️ Prevision.io uses Linear Regression, Random Forest, Gradient Boosting (XGB, LGB, CAT), neural network (tensorflow/keras)</p><p>✔️ Prevision.io makes up to 3 levels of stacking!</p><p>Results</p><p>I launched a second experiment with additional feature engineering to obtain the final score / ranking. Prevision.io allows you to compare versions of experiments, keeping track of results in a simple UI. What did it take to make this happen:</p><p>🔥 Less than 30 minutes of effort on my side to achieve a top 12% finish.</p><p>👉 Try It Now, it’s FREE ! <a href="https://cloud.prevision.io/">https://cloud.prevision.io</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf7a9828497d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/all-you-need-is-automl-and-love-prevision-io-cf7a9828497d">All you need is AutoML … and love! — Prevision.io</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What does it take to offer a product on Google Cloud?]]></title>
            <link>https://medium.com/prevision-io/what-does-it-take-to-offer-a-product-on-google-cloud-ba1413774e56?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/ba1413774e56</guid>
            <category><![CDATA[saad]]></category>
            <category><![CDATA[gcp-marketplace]]></category>
            <dc:creator><![CDATA[Olivier Biesmans]]></dc:creator>
            <pubDate>Fri, 21 Jan 2022 08:47:26 GMT</pubDate>
            <atom:updated>2022-01-21T08:47:29.216Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>What does it take to offer a product on Google Cloud? A recap of Prevision.io’s experience bringing our AI Management platform into GCP.</strong></h3><h3>The process with Google</h3><p>The GCP Marketplace team will grant you access to the Producer Portal, and you’ll be guided through your entire application by a Cloud Partner Engineer. A submission to the Marketplace goes to three distinct reviews: the product details, the pricing, and the technical integration.</p><p>In this article, we’ll go through what the technical integration to the Google Cloud Marketplace actually is, and what to expect from the reviews.<br>Hopefully this will help you go through your own Marketplace application!</p><h3>Integration</h3><p>The technical integration with the Google Marketplace is documented in an <a href="https://cloud.google.com/marketplace/docs/partners/integrated-saas/">onboarding guide</a> and in a <a href="https://developers.google.com/gcp-marketplace-saas#0">codelab</a>. Both are very important reads. In any case, here is a big picture view of the integration process.</p><h3>Managing users</h3><p>When users buy your product on the Google Cloud Marketplace, they have to create an account in your app. The first integration is this Account activation page.</p><p>When the user clicks the link to sign up for your app, they are redirected to your activation page, along with a JSON Web Token (JWT) which contains an <em>account id</em>. You must register their account along with this <em>account id</em> in your app, then approve the account on the <a href="https://cloud.google.com/marketplace/docs/partners/commerce-procurement-api/reference/rest/v1/providers.accounts/approve">Parner Procurement API</a>.</p><h3>Managing entitlements</h3><p>When users choose a pricing plan for your product, the Google Cloud Marketplace will send you this information through a PubSub <em>ENTITLEMENT_CREATION_REQUESTED</em> message containing an <em>entitlement id</em> and other info. You have to approve or reject the entitlement through a call to the <a href="https://cloud.google.com/marketplace/docs/partners/commerce-procurement-api/reference/rest/v1/providers.entitlements/approve">Partner Procurement API</a>.</p><p>After approving the entitlement, another PubSub <em>ENTITLEMENT_ACTIVE</em> message will indicate that the user plan is now active, and you can safely give him access to your product.</p><h3>Usage Reporting</h3><p>This step is only required if your pricing model is usage-based. If yes, then you’ll have to decide what you usage metrics are for your pricing model, then send billing usage data to the <a href="https://cloud.google.com/service-infrastructure/docs/service-control/reference/rest">Service Control API</a> so GCP can charge your client for their usage.</p><p>There’s extensive documentation to <a href="https://cloud.google.com/service-infrastructure/docs/reporting-billing-metrics">report billing metrics</a>, yet underneath all these details, it’s simply a call to the <a href="https://cloud.google.com/service-infrastructure/docs/service-control/reference/rest/v1/services/report">services.report methods</a> of the Service Control API.</p><p>While implementing this, we also understood that it’s very practical to have the usage reports broken by the hour of usage (and not the day), as GCP Billing reports will also be based on hourly usage.</p><h3>Google’s Marketplace validation process</h3><h3>Product details review</h3><p>The product details review is quite straight-forward. You submit a description of your product that’ll be used in the Marketplace and a Cloud Partner Engineer validates it.</p><p>Make sure the description is as complete as possible, including documentation of your product, a way to contact your support, etc.</p><h3>Pricing review</h3><p>In the pricing review, the Cloud Partner Engineer will go through each metric you defined and make sure it’s clear what exactly you are billing, so make sure you use explicit and precise metric names. Also a quick recap of what your product does and which actions are billed will go a long way to validate the pricing.</p><p>Also remember that it’s forbidden to sell professional services through the marketplace.</p><h3>Technical integration review</h3><p>The technical integration review is divided in two steps.</p><p>First, the usage-based billing testing. Here, we had to purchase our own application in the marketplace, and report usage billing metrics to Google hourly for four days. In the meantime, we had to configure a BigQuery export of our billing account, then filter it to only have the usage for our application in the marketplace, and verify that the export matches the reports we made to the Service Control API. If the exports match the reports on each usage metrics, showing the correct amount of usage for each day, you’ll pass this step.</p><p>Second, the solution review meeting. It was quite simple, we had to do a test purchase of our app again, while screen-sharing the process to the Cloud Partner Engineer assigned to us.</p><h3>Our experience</h3><p>Here is our advice for cruising through a Google Cloud Marketplace submission.</p><p><em>Submit manual validation early.<br></em>To ship a Google Cloud Marketplace integration fast, it’s crucial to understand that some steps will take time to be validated by the Google Cloud Partner team. The lengthier steps are the billing with custom metrics validation, together with the usage-based billing testing. The earlier you start those two steps, the better.</p><p><em>Build a Marketplace Integration sandbox.<br></em>We invested a bit of time to set up a development sandbox that emulated the marketplace functions: such as signing a JWT, accepting or rejecting account IDs and entitlement IDs, and reporting usage-based metrics. This enabled us to automate a lot of testing instead of doing it manually from the Producer Portal.</p><p><em>Prepare hourly usage reports<br></em>Usage-based billing is tough to debug, and the Billing export to BigQuery is broken into hours. So while it’s not specified anywhere in the official docs, we found that reporting usage hourly ended up very practical to pass the usage-based billing testing, as you can easily compare the usage report requests to its corresponding line in the BigQuery export.</p><h3>Conclusion</h3><p>We are proud to now offer our product through Google Cloud Marketplace. Access our AI management platform and only pay for what you use thanks to our Pay-As-You-Go billing option here — <a href="https://prevision.io/free-trial/">Try Prevision.io for free</a>. As an added bonus, get $400 in free credits when you sign up as a new GCP user and access Prevision.io.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba1413774e56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/what-does-it-take-to-offer-a-product-on-google-cloud-ba1413774e56">What does it take to offer a product on Google Cloud?</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Become a French cheese connoisseur with your smartphone]]></title>
            <link>https://medium.com/prevision-io/become-a-french-cheese-connoisseur-with-your-smartphone-84950efc3e49?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/84950efc3e49</guid>
            <category><![CDATA[cheese]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Thu, 16 Dec 2021 13:27:33 GMT</pubDate>
            <atom:updated>2021-12-16T13:27:33.450Z</atom:updated>
            <content:encoded><![CDATA[<p>Written by Arnold Zephir</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-hkpcTY_nvAxECgm.png" /></figure><p>Computer vision use cases are hot topics and a frequent one in the AI world. They often are impressive. Seeing AI naming or classifying an image truly shows how far the field has come in the last ten years.</p><p>That said, there are few real world use cases deployed. With the exception of the standard cat picture found everywhere, building your own Computer vision model from scratch is expensive and quite difficult without the skill and knowledge of a skilled data scientist.</p><p>Here at Prevision.io we tried to build a model trained to recognize french cheese, something that has never been done before, but with one major constraint. We limited our analysis to 5 days.</p><p>Did we succeed?</p><p>Yes. Let’s first start with some more general analysis of computer vision use cases.</p><p>Computer Vision is one of the areas that has benefited the most from progress in AI. It encompasses four main types of problem, ranging from the least amount of human intervention to the most :</p><ul><li>image generation: AI learns to generate images that look like a man made image</li><li>image clustering: AI is able to group different images by similarity without human intervention</li><li>image classification: AI learns to put a label on an image (a number, a tag, a sentiment)</li><li>Object detection and segmentation: inside an image, draw a bounding box or a detailed contour of an object</li></ul><p>Image generation and clustering share the same underlying techniques. Images are encoded with mathematical vectors such that vectors for similar images have a “small distance” between them and images can be rebuilt from its vector. The following is the DCGan Architecture and its variant, which is behind all the wow! demos seen on the Internet. Some examples are the famous or <a href="https://thispersondoesnotexist.com/">thispersonndoesnotexist </a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*iyfsnXP9AwkavpxPG2xSew.png" /></figure><p>As long as you have trained a DCGan, you can generate a new image by putting a new vector into it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*u8wg-we31gonwN8k9dZK7Q.png" /></figure><p>And common clustering techniques can be applied on the image’s vectors, as they are standard mathematical vectors upon which a distance can be built, representing similarities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*N-ryd4BzWtVXpVDDjH8Epg.png" /></figure><p><em>For each couple of images we can build a distance And from a distance matrix, we can build clusters</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*vlsyTiMpSt6eYmsiyrQ46g.png" /></figure><p>From Vectors and clusters, you may start to build classification:</p><ul><li>either in a semi-supervised way by putting manual labels on each cluster if they look good. But, be aware that your cluster may not be exactly your label (a typical example is a cluster built on image characteristics like lighting and colors or even the type of background)</li><li>put a class on your list of vectors. Thus, you can go back to a tabular use case where your classic model tries to find a relation between a list of float and a target class</li></ul><p><em>Of course you may go the traditional way with a </em><a href="https://en.wikipedia.org/wiki/Convolutional_neural_network"><em>CNN </em></a><em>Detection (left) and segmentation ( right ) (Convolutional Neural Network) where the target is your class but using a Gan brings the benefit of getting an embedded vector for each of your images that can be stored for later usage (typically charts of images similarity).</em></p><p>Another standard usage of Computer Vision is Object Detector and segmentation. Inside of an image, find the bounding box or contour of one or more objects, and label (classify) this object</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*ROsxwsdeKqLGuRFlyTDhfA.png" /></figure><p>Each of these use cases can obviously earn you some gain, typically by automating a human task ( <em>e.g. executing it faster or cheaper </em>) or by being better at it ( <em>for example detecting more of some object or sooner </em>)</p><p>Computing gain is a topic for another article but let’s see where expenses come from during a Computer Vision use case.</p><h3>Image Acquisition</h3><p>AI is good but still not as good as humans to learn something new.</p><p>For understanding an image, standard Computer vision still needs a lot of images (we are talking about tens of thousand images for a use case).</p><ul><li>You already have them for historical reasons. For example you work for an insurance company and have a lot of damaged car photos.</li><li>Scraping them on the web by using some API. Typically, Google charges $5 for 1000 satellite image taken from its api</li><li>Make your own database by recruiting humans and let them take photos of the object you want to tackle</li></ul><p>It sometimes depends on the problem. For example, detecting a pool, a blue rectangle in a forest, round green stuff, is easier than trying to classify a benign mole from a malignant one. But, it often comes with a cost.</p><p>There are 3 way to acquire images :</p><p><strong>The smartest way is obviously a mix of several methods. Start a model with the cheapest method, for example scrape a hundred images and then use humans to label them correctly or update the model with the obvious errors.</strong></p><p>Scraping images from the web is the cheapest. You can quickly access a few thousands of images for free. Be cautious, images from the web could be unrelated to your objective, or bring lighting quality that differ from your real use case (sometimes best, sometimes worst ).</p><p>As an alternative, build your own database by allowing others to take photos in true real condition. This avoids having too much difference between the train set and the production usage, as you can tell your photographer to take images in the same condition for future usage. This is very expensive. Expect to only receive one or two hundred image a day for each people working on acquiring images</p><p>You should not expect to have the perfect dataset at the first model. However, building a first cheap model allows your dev team or alpha tester to use it on real images and give feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*B4JGof8rS2MilLz-LUCdYQ.png" /></figure><p>The less expensive way to get a model good enough is to start with the higher complexity modeling (here an object detector) on a few images, then generate some predictions and get new images and corrections from the user to improve the initial model.</p><p>It will always be better to train the model on images sent from real usage rather than training on generic images, like the ones found on the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*h5HLd_AVxU4SpH0R2jMPSA.png" /></figure><p>Obviously, the best outcome is to already have images in your SI but we are dealing with the case you are starting a new modeling from scratch.</p><h3>Image Labeling</h3><p>In most computer-vision use cases, this could be the most expensive step. Depending on your use case, you can go from no labeling, and thus no cost, to very accurate labeling that is quite costly.</p><h3>Zero — labels: Image generation</h3><p>If you want to generate an image that looks like some other image from your dataset, you never need to label your images. Just throw the most pertinent image in a folder and run a DCGAN, or its variation, as the “A” in DCGAN implies — use an Adversarial neural network to decide if your generated image is good or not.</p><p>Yet, expect your training cost to be very high as good image generation with high definition often needs several days of training on racks of GPUs.</p><p>Even if you have no labeling costs, do not forget that this use case is the most expensive in compute cost for training</p><h3>Few labels: Clustering</h3><ul><li>grouped by similarity</li><li>moved away from each other the more dissimilar they are</li><li>assigned a “class” (a cluster number)</li></ul><p>Sometimes called semi-supervised. When clustering your image, you ask your algorithm to split your image “the best it can”.</p><p>By doing this the images will be :</p><p>The results will be that you have quite good output. All your images are pre-tagged and all you have to do is put labels on clusters instead of labeling each image.</p><ul><li>the orange cheese ( mimolette, langres, etc.)</li><li>the white cheeses ( goat cheeses in most cases )</li><li>the cheese on white plate ( aka a big round white circle is on the image )</li><li>the cheeses on black background</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*xlCT4qvQx4PHyg3gASShlA.png" /></figure><p><em>The clusters are perfectly valid. Image of cheese on a black background and image of orange cheese</em></p><p>I need to point out that unless you get lucky, this rarely works as the standard clustering algorithm will split the image along the characteristics and in most cases for images, it will be something like light conditions or background colors.</p><p>For example for cheezam, when we submitted a set of 1000 images, they were split into about 6 clusters that were :</p><p>Thus, in order to get a business related model, in most cases, you should apply some label to teach the machine what you expect (or which features of your image you want to put an emphasis on).</p><h3>Labeling: Image Classification</h3><ul><li>either by using scraper and labeling images while scraping them (for example, make an image search with the term “fromage mimolette” and put the result in the mimolette folder</li><li>or by using a tool that enabled labeling while acquiring images</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*THG4XvydJa8JnOVN0KNu2A.png" /></figure><p><em>The behind the scene Cheezam label tool: snap your cheeses with a smartphone and send them labeled. Images are then sent to storage</em></p><p>To train a model to recognize Fromage de Langre from Mimolette, you need some human in the loop that puts a “mimolette” label on mimolettes images. With high complexity classification like splitting Brie de Meaux from Camembert, this can take up to 3s an image, meaning a rate of about 1000 image/hour , or something like $25 for 1000 images ( $250 for 10000 images).</p><p>The other way is directly label images while acquiring them :</p><p>With this kind of tool, you can expect to acquire 250 images a day by person with its label putting the cost of acquiring + labeling to about $1 an image.</p><p>In order to kickstart a model, you probably need 200 images of each class (depending on separability of class ) so a typical multi classification with 5 classes needs 1000 images or $1,000 to get images labeled.</p><h3>Bounding box : Object Detector</h3><ul><li>there could be more than one object on each image</li><li>you need to draw one rectangle (bounding box) or more on each image, which is a very tedious task</li></ul><p>The next stage of computer vision is more difficult and costly. Object detection tries to number and locate an object on a photo by putting a bounding box around it.</p><p>It means that :</p><p>The benefit of a bounding box is that your model can tell you if there are several objects of different kinds on an image and count them. This avoids just classifying the images, which is exactly what we want to do with Cheezam.</p><p><em>Box bounding my mimolette</em></p><p>For building the Cheezam dataset, we managed to draw about 500 boxes a day using <a href="https://www.makesense.ai/">https://www.makesense.ai/ </a>. This is free, online and open source or for labeling <a href="https://blog.roboflow.com/labelimg/">https://blog.roboflow.com/labelimg/ </a>which requires some computer skill to set up.</p><p>This drawing cost adds to the image acquisition and labeling as when drawing your box you still must add a label. We found it very efficient to use only one generic label when drawing a bounding box (“cheese”) and splitting the process of putting the right label from the process of drawing the box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*m_5qhBN4DrBWyFUX635TRA.png" /></figure><h3>High complexity labeling: Object Segmentation</h3><p>Object segmentation is similar to Object Detection but instead of a bounding box, you draw a polygon that perfectly fits your object.</p><p><em>Detouring my valencay</em></p><p>The needs of human manual labor for this kind of task is huge, ten times the time for a “simple” object detector with bounding boxes.</p><p>We tried it on a small dataset of 200 images for Cheezam and the performances, counting the cheese and finding the right class, were not significantly improved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*2yrFSA1fZKkAL-bwdzMsjw.png" /></figure><p><em>Labeling cost</em></p><p>This kind of task happens when you need to compute the size or area of an object, or for some very specific use case which needs the exact shape of the object.</p><ul><li>image generation which is a very heavy process and which needs several hours of high-end GPUs</li><li>the other task, which takes around 10 hours for training a model on ~1000 images</li></ul><p>You should always compute your expected gain when starting this kind of project to evaluate the cost/benefit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*udKX8h3MICTMUdqNhLgAxg.png" /></figure><p>The training cost could be split into two parts:</p><p>The first costs several thousands of dollars.</p><p>The second costs about one hundred dollars.</p><p>Of course it greatly depends on the performance you expect so as a rule of thumb, count $100 for each 1000 images into your dataset, with iteration and optimization.</p><p>Cheezam was a dataset of 3000 images and we ran several trainings using a total of 110 hours of compute for a total $283 using <a href="https://console.cloud.google.com/marketplace/product/prevision-public/prevision.io">Prevision.io on GCP </a>.</p><p>Ok, so finally, what was the best way?</p><p>With trial and error, we successfully achieved a very efficient method that we applied on cheezam. Here’s what that looks like.</p><p><strong>Cost</strong>: 1 hour of work (scraping script)</p><p><em>The real ground breaking idea was to let the first model draw the bounding box because humans are too slow. Then to keep the human mind for applying the labels.</em></p><p><strong>Cost</strong>: 20 hours of computing</p><p><strong>Cost : </strong>2 days of (accumulated) work</p><p>The main issue of this method was that we asked co-workers not to take too many images at once, otherwise we would have needed a tool to pinpoint the cheese on each image, which was tedious with a smartphone.</p><p>Letting the model focus on bounding boxes while humans were in charge of putting labels while acquiring images was a good way to benefit from the best of both worlds.</p><p>If you are interested in trying the application. You can find Cheezam on its dedicated page <a href="https://cheezam.fr/">https://cheezam.fr </a>.</p><p>It is currently trained on a family of cheese instead of a specific name. Cheezam version 2.0 is in the pipeline and our team is working hard to get a lot of images of Cabecou and Rocamadour. Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84950efc3e49" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/become-a-french-cheese-connoisseur-with-your-smartphone-84950efc3e49">Become a French cheese connoisseur with your smartphone</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The next big thing: Synthetic Data for Machine Learning]]></title>
            <link>https://medium.com/prevision-io/the-next-big-thing-synthetic-data-for-machine-learning-1cd99c76c004?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/1cd99c76c004</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[image]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Wed, 15 Dec 2021 13:59:50 GMT</pubDate>
            <atom:updated>2021-12-15T13:59:26.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TWZhMb-PgPKcRVWI.png" /></figure><p>It’s often quoted, “Data is the new oil” and as a data scientist, I clearly know the value of good data to train our model. However, we often face some problems with data:</p><ul><li>Limited or no access because of confidentiality / governance requirements</li><li>Lack of sufficient quantity</li><li>It’s unbalanced and thus leads to biased models</li><li>… the list could go on and on.</li></ul><p>In this guide, we will explore the usage and performance of synthetic data generation. If you want to try it by yourself you can:</p><p>A common problem in the just about every industry (telecom, banking, healthcare, etc.) is confidentiality of data. We want to build models using sensitive data while avoiding confidential or proprietary information.</p><p>As an example, you may have a dataset in which each row is age, gender, size, weight, place of residence, etc. Even if data is anonymized by removing name or email, it’s often easy to find some particular person through their unique combination of features as in the following example:</p><p>A 30 year old man who lives near Cambridge, earns $30000 a month, whose wife is an engineer and subscribes to a sports magazine would be unique enough to know a lot about him, even without identity data. o no.</p><p>The second problem is lack of data or worse, a lack of data for some segment of the population that will then be disadvantaged by models.</p><p>For example, let’s say that we have a dataset with 10000 men and 1000 women and we are using RMSE as the metric for our training. Let’s say for the sake of this article that every prediction for men has the same error manError and women have a womanError constant error.</p><p>The Score of a model will be :</p><p>df[‘RMSE’] = ((10000*(df[‘manError’]**2) + 1000*(df[‘womanError’]**2))/(10000 + 1000))**.5</p><p>And the errors of each gender will be weighted as follows on the training metric:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*0J1RoUv4x7jz5PsAfnD6Dw.png" /></figure><p>We clearly see that making an error on the woman segment weighs much less on the optimized metric, as going from a 100 to 10000 woman error only goes from 100 to 3016 total error yet for men it goes from 100 to 9534.</p><p>A goal of machine learning models is to find the minimum of given metrics. A model trained on this data will advantage the men segment over the women segment. Of course, if we had the same number of men and women data, the error would be balanced and no bias toward a segment would occur. But adding more women data is not enough. We cannot just copy and paste more rows, as it will lack diversity.</p><p>What we should do instead is add more “realistic” women data by sampling from the underlying distribution of each woman feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hObxDpSLdf-mTHZw" /><figcaption><em>Weight distribution for some segment</em></figcaption></figure><p>Yet, if we just do that the naive way, this could happen :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*ZmgbxIMjZzbqc65GpyrDtg.png" /><figcaption>( Note: of course retired women may play american football, yet weighing 12kg and playing american football when you are 82 years old is quite odd )</figcaption></figure><p>There are two pitfalls when generating fake data :</p><ul><li>Assuming everything is a normal distribution and reducing a feature to its average and its deviation. In fact, most real world distribution is skewed</li><li>Not caring about <a href="https://en.wikipedia.org/wiki/Joint_probability_distribution">joint probability distribution.</a></li></ul><p>This is where a good synthetic Data Generator comes to the rescue.</p><p>Like every data science modelization, synthetic data generation needs some metric to measure performance of different algorithms. This metric should capture three expected outputs of synthetic data:</p><ul><li>Distribution should “look like” real data</li><li>The <a href="https://en.wikipedia.org/wiki/Joint_probability_distribution">joint probability distribution </a>should look like the real one too (of course, someone who weighs 12kg should not be 1.93m tall, this criteria is sometimes call likelihood fitness )</li><li>Model performance should stay constant</li></ul><h3>Statistical properties of Generated Synthetic Data</h3><p>The obvious way to generate synthetic data is to assume every feature follows a normal distribution, compute means and deviation, and generate data from gaussian distribution ( or discrete uniform distribution ) with the following statistics :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*8G106bMpQMUcolKr8X2r1Q.png" /></figure><p>In python</p><p>[…]</p><p>nrm = pd.DataFrame()</p><p>for feat in [“price”,“sqft_living”,“sqft_lot”,“yr_built”]:</p><p>stats=src.describe()[feat].astype(int)</p><p>mu, sigma = stats[‘mean’], stats[‘std’]</p><p>s = np.random.normal(mu, sigma, len(dst))</p><p>nrm[feat] = s.astype(int)</p><p><em># For discrete Features</em></p><p>for feat in [“bedrooms”,“bathrooms”]:</p><p>p_bath = (src.groupby(feat).count()/len(src))[“price”]</p><p>b=np.random.choice(p_bath.index, p=p_bath.values,size=len(dst))</p><p>nrm[feat] = b</p><p>This approach leads to poor feature and joint distribution.</p><p>In the following plot, we show the distribution of features of (blue) , distribution of a synthetic dataset generated with a CTGAN in quick mode (orange, only 100 epoch for fitting ) and distribution of data assuming each feature is independant and follows a normal distribution (see code above).</p><p>For price, we see that generated data perfectly captures the distribution. It’s so good that we barely see the orange curve. Gaussian random data has the same mean and deviation as the original data but clearly does not fit the true data distribution.</p><p>For yr_built, the synthetic data generator (orange) struggles to perfectly capture the true distribution (blue) but it looks better than the average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8h8ffGqU68d6Kful" /><figcaption><em>Price and bedrooms distribution</em></figcaption></figure><p>Below is the density chart for price, zoomed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JFdmKMFYOFzdDghl" /><figcaption><em>Zoom on price distribution</em></figcaption></figure><p>It’s more difficult to see joint distribution on this chart but we can draw a contour map to get a better view:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nr8fmf29ClgcTtO5" /><figcaption><em>Original true Data Joint distribution of price and year constructed</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XBKM4n2-PhN8Q3nz" /><figcaption><em>Generated Data Joint distribution of same features. We see density start to reach those of real data</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O7i33mTyriQusqFw" /><figcaption><em>Independent random variable distribution; jointplot density is totally wrong</em></figcaption></figure><p>The same conclusion goes for other features as well. Random independent data do not fit distribution and joint distribution as effectively as properly generated data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6_IqerMCXIOvKt4A" /><figcaption><em>Each feature has its own distribution and distribution relative to others</em></figcaption></figure><p>Price and bedrooms distribution</p><p>Below is the density chart for price, zoomed.</p><p>Zoom on price distribution</p><p>It’s more difficult to see joint distribution on this chart but we can draw a contour map to get a better view:</p><p>Original true Data Joint distribution of price and year constructed</p><p>Generated Data Joint distribution of same features. We see density start to reach those of real data</p><p>Independent random variable distribution; jointplot density is totally wrong</p><p>The same conclusion goes for other features as well. Random independent data do not fit distribution and joint distribution as effectively as properly generated data.</p><p>Each feature has its own distribution and distribution relative to others</p><p>Of course, better models than simple normal distribution do exist.</p><p>You can build kernel density estimation, gaussian mixture model or use other distribution than a normal. However,</p><p>But as you build more and more complex models, you soon will discover that you can use machine Learning to do the job instead of looking manually for the best distribution shape and parameters. And that is where you start to use Generative Adversarial Network.</p><p>Generating good Synthetic Data</p><p>Like the prior image, Tabular data benefits from the usage of Generative Adversarial Networks with a Discriminator. In order to avoid the issue shown above, we can train a neural network on data whose input would be for each sample vector of:</p><ul><li>discrete probability for each mode of categorical variable</li><li>probability of each mode of a a gaussian mixture model</li></ul><p>The network is then tasked to generate a “fake sample” output and a Discriminator then tries to guess if this output comes from the Generator or from the true data.</p><p>By doing this, the Generator will learn both distribution and joint distribution between each feature, converging to a generator that outputs a sample as true as the real one.</p><p>The current best implementation of this idea is that you can <a href="https://sdg.cloud.prevision.io/">use on our Marketplace</a></p><p>The best practical performance estimation of generated data is Machine Learning efficiency. We ran performance tests using our <a href="https://cloud.prevision.io/">AI Management Platform </a>with the optimal parameters (all models tried, hyper optimization and Blending of models) on 3 datasets with the same target :</p><ul><li>Original dataset</li><li>Synthetic dataset fit on 800 epochs</li><li>Random Dataset with no joint probability</li></ul><p>We expect that by using the AutoML feature, only data quality would change the performance.</p><p>Here are the results :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*vF68azLSBkJWNKrv" /><figcaption><em>Modelization on true Data</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/0*aJ0zgNFsIbAiJnRq" /><figcaption><em>Modelization on Synthetic Data</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*nkylshdJrIA72kQZ" /><figcaption><em>Modelisation on random independent data</em></figcaption></figure><p>We see that with CTGAN synthetic data, machine learning efficiency is preserved, which is not the case for independent data distribution.</p><p>Moreover, this efficiency is good even for segments of the original dataset, meaning you can generate synthetic samples for under-represented categories and keep their signal high enough to fix any bias in your dataset.</p><p>The CTGAN implementation is available on <a href="https://sdg.cloud.prevision.io/">our Marketplace </a>and you can test it with one of <a href="https://doc.prevision.io/en/latest/howto/useful_datas.html">our public standard Dataset</a></p><p>Remember that synthetic data is very useful when you need to enforce data privacy or unbias a dataset. Training a Synthetic Data Generator for a few hours to build a generator that generates synthetic data should become part of your data science toolkit.</p><p><em>Originally published at </em><a href="https://prevision.io/blog/the-next-big-thing-synthetic-data-for-machine-learning/"><em>https://prevision.io</em></a><em> on December 15, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cd99c76c004" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/the-next-big-thing-synthetic-data-for-machine-learning-1cd99c76c004">The next big thing: Synthetic Data for Machine Learning</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is experiment tracking essential in ML Part 3/3]]></title>
            <link>https://medium.com/prevision-io/why-is-experiment-tracking-essential-in-ml-part-3-3-87f3eb1881be?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/87f3eb1881be</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[experiment-tracking]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Thu, 09 Dec 2021 13:17:07 GMT</pubDate>
            <atom:updated>2021-12-09T13:17:07.569Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SBRg1s7xURSj6sAlZ0w5dw.jpeg" /></figure><p><em>by Mathurin Aché</em></p><p>This blog post series will be composed into 3 steps:</p><ul><li><a href="https://content.prevision.io/experiment-tracking-with-prevision-io-part-1-3">The first blog post will introduce the subject of the experiment tracking: What is experiment tracking? Why is experiment tracking essential? What is the most common way to organize data, pipeline and models in an experiment?</a></li><li><a href="http://content.prevision.io/experiment-tracking-with-prevision-io-part-2-3">The second blog post will deal with the subject of experimental tracking using the Prevision.io platform, without writing a line of code.</a></li><li>The third blog post will deal with the subject of experiment tracking by combining learning outside the platform and monitoring in the Prevision.io platform. It will conclude the present blog post series.</li></ul><p>In this post, we are going to answer this question:</p><ul><li>How can I benefit from the experiment tracking and other advantages included in the Prevision.io platform while continuing to build my experiment outside the platform and / or with third-party solutions?</li></ul><p>If you use another environment to train your models and you wish to benefit from the experiment tracking solutions offered by Prevision.io :</p><ol><li><strong>You load and prepare data in your environment, on Kaggle notebook or on Google Colab</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*jOmoMUg20d62PhBE" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*FB6YszYVghPJMm2A" /></figure><p><strong>2. You train one or many models in your environment, on Prevision.io notebooks, on Kaggle notebook or on Google Colab and export them in ONNX format.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/0*98Tuix5Us8w4dOc_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*KsZDn9GOBGQR-aWc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*LL_Wc7fW2ZuDc1H4" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/0*LZ_Z7zfCIzm3Xku_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*SSLdYux5N0w5-Eci" /></figure><p>For each model listed above, you will see a part which consists of converting the model from the scikit learn format to the <a href="http://onnx.ai/sklearn-onnx/">ONNX format</a> which is the format expected to be processed in the Prevision.io platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*vdFc6xmfvpnQCY38" /></figure><p><strong>3. You upload data models, configuration files using the user interface.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/0*YT9FQ07S4WmgmbfH" /></figure><p>Configuring your external model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/0*bnxi3wmLMFhL0XjF" /></figure><p>Process to import your external model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/0*q-u6XQmPtQAV3SYZ" /></figure><p>Configuring your external experiment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_V96NKAr2sLsFXhA" /></figure><p>For each external model, you need to set a name, a yaml with features configuration,</p><p>and a ONNX file containing the model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tidVW2u9CSOfl9Bk" /></figure><p>You can import as many models as you want</p><p>To go further, external model import uses the Standardized ONNX Format and most of the standard ML libraries have a module for export.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e6Ht5vvk9FlAVnJa" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cjhiW87HalmmhYSK" /></figure><p>After few minutes, you obtain a dashboard with all models</p><p>Now you can Evaluate your experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*Mj5n4Wt7YanfT-_A" /></figure><p>External model information</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-kv8KyYgJqJAhDCL" /></figure><p>External model feature importance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*6VWyMgCL6pnryQDT" /></figure><p>External model confusion matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*EV_fnrcyTUAE6Ep4" /></figure><p>External model metrics</p><p>Good news: Once imported, you still can benefit from the insightful analytics available for internally trained models.</p><p><strong>4. You upload data models and the related configuration files using the SDK (Python or R).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*lStzHa2elE2yvjkS" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*ZUfp1vOaltxF9IlL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*wslSJVDU2F4x6xuC" /></figure><p><strong>5. Once your imported model is deployed, you are able to use it periodically (every hour, every day, every month …).</strong></p><p>To proceed to Deployments, I refer you to the paragraph explaining how to deploy an experiment in article 2 of this series or in documentation here <a href="https://previsionio.readthedocs.io/fr/latest/studio/deployments/index.html">https://previsionio.readthedocs.io/fr/latest/studio/deployments/index.html</a></p><h3>Conclusion</h3><p>In this guide, we went through the whole experiment tracking process, while using Prevision.io.</p><p>As we have seen, it is essential for a data scientist to document the different iterations over all the data science project stages: from data ingestion, to feature engineering, to model selection, to hyperparameters tuning, while accessing the in depth visual analysis, until the model is deployed and in production.</p><p><em>Originally published at </em><a href="https://content.prevision.io/experiment-tracking-with-prevision-io-part-3-3"><em>https://content.prevision.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87f3eb1881be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/why-is-experiment-tracking-essential-in-ml-part-3-3-87f3eb1881be">Why is experiment tracking essential in ML Part 3/3</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is experiment tracking essential in ML Part 2/3]]></title>
            <link>https://medium.com/prevision-io/why-is-experiment-tracking-essential-in-ml-part-2-3-f38fc0049b6?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/f38fc0049b6</guid>
            <category><![CDATA[experiment-tracking]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops-platform]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 13:49:03 GMT</pubDate>
            <atom:updated>2021-12-08T13:48:42.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xetaEI1XRuSunrHgDJ2gLw.jpeg" /></figure><p><em>by Mathurin Aché</em></p><p>This blog post series will be composed into 3 steps:</p><ul><li>The first blog post will introduce the subject of the experiment tracking: What is experiment tracking? Why is experiment tracking essential? What is the most common way to organize data, pipeline and models in an experiment?</li><li>The second blog post will deal with the subject of experimental tracking using the Prevision.io platform, without writing a line of code.</li><li>The third blog post will deal with the subject of experiment tracking by combining learning outside the platform and monitoring in the Prevision.io platform. It will conclude the present blog post series.</li></ul><p>As we have seen previously, experimental tracking allows data scientists to identify factors that would affect the model’s performance, compare results and select the optimal version. It is very important to stay organized throughout the iterative learning process. This is also the case in the production phase, where you must follow the production version, possibly the challengers models, performance, and usage monitoring. In this post, we are going to answer this question:</p><ul><li>How is experiment tracking done with Prevision.io without a line of code ?</li></ul><h3>How is experiment tracking done with Prevision.io ?</h3><p>Prevision.io offers a platform in which you will be able to centralize, organize and visualize the results of all your experiments, throughout both training and production phase, whether you have used Prevision.io’s autoML for training or you have trained your own models outside the platform.</p><p>In what follows, we will distinguish 2 situations:</p><ul><li>if you are using Prevision.io to train your models,</li><li>if you use another environment to train your models and you wish to benefit from the experiment tracking solutions offered by Prevision.io. This part will be covered in the third blog.</li></ul><p>In Prevision.io, the first level of organization is the data science project. The project covers the entire life cycle, from design, to operational use, to periodic monitoring, while including any remodeling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3JXC4ueoPUW6wfEn" /><figcaption>View of one project in Prevision.io</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/154/0*fHMwp9I1l6Lz-Sw7" /></figure><p>Home allows you to return to the project summary.</p><ul><li>Data tab provides access to all the data sources used in the project, whether flat files, or files from data sources from third-party connections such as SQL database, Google Cloud Storage, Amazon S3.</li><li>Experiments tab allows you to access all the iterations relating to this project.</li><li>Pipelines tab allows you to access all of the data preparation and release flows that you have developed in your project.</li><li>Deployments tab allows you to access all the deployed models relating to this project.</li><li>Collaborators tab allows you to define the rights of the various stakeholders in the project, from the project administrator, to the contributor to the simple viewer.</li></ul><p>For each of the point mentioned <a href="https://content.prevision.io/experiment-tracking-with-prevision-io-part-1-3">above about experiment tracking</a>, here is what Prevision.io offers :</p><ul><li><strong>Dataset metadata (dataset name # rows, # columns, #size in Mo/Go),</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jKJ5iGJjNP9Zja4B" /><figcaption>General dataset view</figcaption></figure><p>This view is used to display all datasets used in the project either uploaded by the user or coming from a datasource.</p><p>The information available are: name of the dataset, number of rows and columns, volume, date of creation, person at the origin of the dataset, type of source, status, and available actions such as viewing, downloading , renaming, deleting or even exporting the dataset to another workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*38N0QjXtnhDQPqBI" /><figcaption>View of one version of a dataset</figcaption></figure><p>This interface enables to have additional information on a particular dataset such as the number of numerical and categorical variables, the correlation matrix, as well as the run experiments using this dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Noi_eoSWlFWlX4in" /><figcaption>View of features for one dataset</figcaption></figure><p>This screen allows you to have a deep understanding of the dataset providing detailed information by variable including the number of the missing values and its univariate representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ol_l3oGWAVpfIW9C" /><figcaption>View of linear feature information and distribution</figcaption></figure><p>This screen allows you to have some statistical information about linear features: minimum, maximum, mean, standard deviation. The feature distribution is also represented in the form of a histogram of the frequencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_o6LvUc9dRCm5xPc" /><figcaption>View of categorical feature information and distribution</figcaption></figure><p>This screen allows you to have some information about categorical : unique values and feature distribution represented by a pie chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RF0HRp0mWalI0v5T" /><figcaption>Preview of dataset</figcaption></figure><p>You can browse sample data from the Sample tab.</p><p><strong>Experiment’s settings: dataset you use, features engineering, features selection, list of algorithms you try, ensemble models, role of features (target, id, fold, weight, features selected and features dropped)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hzrUgmUz7MPEapss" /><figcaption>New experiment settings with Execution Graph Preview</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/0*yqPRA7Iy6fNUx8XC" /></figure><p>An experiment in Prevision.io starts by the experiment configuration interface in which you specify the train dataset, the possible validation dataset, a description for the experiment (do not neglect this point, the description allows you to note the key differentiating points, especially if you are in your n-th iterations), the metric you use and the variables used in the configuration: target (required), id (strongly recommended, need that this id be unique), (optional) weight if you want to specifically weight the rows, (optional) fold if you want to to set your own stratification for the cross validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/0*CaDrIrwoChk5SAoD" /><figcaption>Columns configuration</figcaption></figure><p>The columns configuration panel allows you to ignore some columns of your dataset. To do it, just deselect unwanted features. Please note that you can search by features name the columns you want to unselect and use the “select/unselect all” checkbox to apply your choice to the selection. In most cases, you should not drop any features. The Prevision Platform can handle up to several thousands of features and will keep those meaningful to build its models.</p><p>In which situations it may be interesting to uncheck several features ?</p><ul><li>Some features have too much importance in the model and you suspect that it is in fact a covector of the target, or that you have an important leakage. If you see that your simplest model (Simple LR) performs as well as a complex one and that some feature has a huge feature importance, you may drop it.</li><li>You want to get some results fast. Dropping features allows for faster training. If you suspect that some features bear low signal, drop it at the start of your project to iterate faster.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/0*tiYbTgb-4hgwi7Zp" /><figcaption>Models configuration</figcaption></figure><p>The Models configuration panel allows you to select/unselect algorithms you want Prevision.io to train on your data.</p><p>Simple models are models done with no complex optimization and using simple algorithms, a linear regression and Decision Tree with less than 5 splits. They allow us to check whether the problem is treatable with a very simple algorithm instead of fancy Machine Learning.</p><p>Moreover, simple model generates :</p><ul><li>A chart that explains model and is human readable</li><li>Python code to implement it</li><li>SQL Code to implement it</li></ul><p>You can unselect simple models but it is recommended to keep them when starting a project and watch how well they perform vs more complex models. If a simple Decision Tree performs as good as a Blend of XGBoost, or only marginally worse, do favor the simple model.</p><p>Normal models are models done with a fixed hyperparameters setting defined by default by Prevision.io from previous experiments. Normal models use the simplest preprocessings (label encoding for categorical variables, normalization for numeric fields, no use of textual data).</p><p>Advanced models are models done with tuning hyper parameters. The number of tests to optimize the hyper parameters depends on the performance option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/0*sSs18SJ_SuQPgnjL" /><figcaption>Performances configuration in Basics Panel</figcaption></figure><p>Performances : Quick for fast result, Advanced for best result. Be aware that advanced options could take 4 more times to train than a quick one and only improve marginally the overall performance. Choose wisely ;)</p><p>Blending models is a powerful technique to get the most performance yet it can be quite long to train. Blend uses a model over all the others to merge and combine them in order to get the best performance. Switch the option if you want to blend your models but be aware that the resulting train will last very long.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/0*dkCUD0krc_d5TCOX" /><figcaption>Feature engineering configuration</figcaption></figure><p>Feature engineering configuration</p><p>Four kinds of feature engineering are supported by the platform:</p><p>Date features: dates are detected and operations such as information extraction (day, month, year, day of the week, etc.) and differences (if at least 2 dates are present) are automatically performed.</p><p>Textual features:</p><ul><li>Statistical analysis using Term frequency–inverse document frequency (TF-IDF). Words are mapped to numerics generated using tf-idf metric. The platform has integrated fast algorithms making it possible to keep all uni-grams and bi-grams tf-idf encoding without having to apply dimension reduction.</li></ul><p>More information about TF-IDF on <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">https://en.wikipedia.org/wiki/Tf%E2%80%93idf</a> .</p><ul><li>Word embedding approach using Word2Vec/Glove. Words are projected in a dense vector space. Prevision trains a word2vec algorithm on the actual input corpus, to generate their corresponding vectors. More information about Word embedding on</li></ul><p><a href="https://en.wikipedia.org/wiki/Word_embedding">https://en.wikipedia.org/wiki/Word_embedding</a> .</p><ul><li>Sentence Embedding using Transformers approach. Prevision has integrated BERT-based transformers, as a pre-trained contextual model, that captures word relationships in a bidirectional way. BERT transformer makes it possible to generate more efficient vectors than word Embedding algorithms, it has a linguistic “representation” of its own. To make a text classification, we can use these vector representations as input to basic classifiers to make text classification. Bert (base/uncased) is used on English text and MultiLingual (base/cased) is used on French text. More information about Transformers on</li></ul><p><a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)">https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)</a> .</p><p>Categorical features:</p><ul><li>Frequency encoding: modalities are converted to their respective frequencies.</li><li>Target encoding: modalities are replaced by the average (TARGET, grouped by modality) for a regression and by the proportion of the modality for the target’s modalities in the context of a classification.</li></ul><p>Advanced features:</p><ul><li>Polynomial features: features based on products of existing features are created. This can greatly help linear models since they do not naturally take interactions into account but are less useful on tree based models.</li><li>PCA: features based main components of the PCA are added.</li><li>K-means: Cluster number coming from a K-means method are added as new features.</li><li>Row statistics: features based on row by row counts are added as new features (number of 0, number of missing values, …).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*jNmx6lBz8002eCU_" /><figcaption>Feature selection configuration</figcaption></figure><p>In this part of the screen you can choose to enable feature selection (off by default).</p><p>This operation is important when you have a high number of features (a couple hundreds) and can be critical when the number of features is above 1000 since the full Data Set won’t be able to hold in RAM.</p><p>You can choose to keep a percentage or a count of features and you can give a time budget to Provision.io to perform the search of optimal features given the TARGET and all other parameters. In this time, Prevision.io will subset the feature of the Data Set, and then start the classical process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/0*kTKHk2Snifv3v47k" /></figure><p>Once the configuration of your experiment is complete, you can start the training by clicking on the button at the bottom right of the screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NB3vMqR442K4h7LB" /></figure><ul><li><strong>During training, in real time, you can follow the progress of the modelling.</strong></li></ul><p>The progress of the various tiles making up your execution graph is displayed and changes as the tasks are completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VuaqV4iS5QUl-NGT" /><figcaption>Track training tasks</figcaption></figure><p>Track training tasks panel allows you to have a detailed view of the different parts of your execution graph, possibly with execution logs if the task does not complete as expected. You will find the information about when the task was started, when it finished, its duration and its status.</p><p><strong>When the experiment has trained at least 1 model, you can access the comparative page of the performances of the trained models and a dedicated page per model describing its characteristics.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HsIRrDiHBso1giUq" /><figcaption>Comparative page of the performances of the trained models</figcaption></figure><p>The models panel allows you to have a detailed view of all models trained, technology used, score (metrics you choose), training and prediction durations, time when it’s done.</p><p>In what follows, I present very briefly the analysis screens of a model (as we will see later, these analyzes are also available for models trained outside the platform).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KG78oYsuxk7Bf3Kw" /><figcaption>Model informations and hyperparameters</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N0cfov2l3lMFTbYV" /><figcaption>Feature importance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/0*MS2EYQ4sffYteXIj" /><figcaption>Metrics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*da7b0TQo3F7S49-0" /><figcaption>Confusion matrix</figcaption></figure><p><strong>When you have determined the model you want to deploy, Prevision.io makes it easy for you.</strong></p><p>In what follows, I will show you how to go from an experiment to a production model.</p><p>Deployments are the last step of a Machine Learning Project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*agjxJ-IrIY4z4xR4" /><figcaption>Deployment experiments User Interface</figcaption></figure><p>Deployment experiments panel is the space where you find all the models deployed in your project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zY0pfj1UaN9DQZVD" /><figcaption>How to Deploy a new experiment</figcaption></figure><p>To deploy an experiment, you need to define a name for your deployment, an experiment, a version, a model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lx8myxvbbs1cZRLp" /><figcaption>2 minutes later, your models are in production</figcaption></figure><p>Once an experiment is deployed :</p><ul><li>you can schedule batch prediction</li><li>you can monitor performance of one main model and one challenger</li><li>external users can use your model for unit prediction from an url</li><li>external application can call your model from a REST API</li></ul><p><strong>Once your model is deployed, Prevision.io facilitates its periodic use (hourly, daily, monthly…).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*BHIJdQk1aZXBz_D2" /><figcaption>You upload new data into Prevision.io</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y4oWTMrOYYwOk8mr" /></figure><p>You go to prediction page on your experiment and set your model and your dataset and click launch prediction</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zSOcV_xhJ-QOL-IZ" /><figcaption>You can download predictions</figcaption></figure><p>You can automate the periodic calculation of your forecasts using the Pipeline module and the Scheduler function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kqVoj-u_DcR1OqL6" /><figcaption>Pipeline and scheduled run Overview</figcaption></figure><p>If you want to know more about both the pipeline and the scheduler, I invite you to read this document :</p><p><a href="https://previsionio.readthedocs.io/fr/latest/studio/pipelines.html?highlight=scheduled#scheduled-runs">https://previsionio.readthedocs.io/fr/latest/studio/pipelines.html?highlight=scheduled#scheduled-runs</a></p><p>In the present blog post, I explained step by step how to train an experiment in the Prevision.io platform without writing a single line of code, how to analyze the different models trained,and how to deploy it.</p><p><a href="https://content.prevision.io/experiment-tracking-with-prevision-io-part-3-3">How can I benefit from the experiment tracking and other advantages included in the Prevision.io platform while continuing to build my experiment outside the platform and / or with third-party solutions?</a></p><p><em>Originally published at </em><a href="https://content.prevision.io/experiment-tracking-with-prevision-io-part-2-3"><em>https://content.prevision.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f38fc0049b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/why-is-experiment-tracking-essential-in-ml-part-2-3-f38fc0049b6">Why is experiment tracking essential in ML Part 2/3</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is experiment tracking essential in ML Part 1/3]]></title>
            <link>https://medium.com/prevision-io/why-is-experiment-tracking-essential-in-ml-part-1-3-32a3620f24ea?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/32a3620f24ea</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[experiment-tracking]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Tue, 07 Dec 2021 09:47:20 GMT</pubDate>
            <atom:updated>2021-12-07T09:47:19.990Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fo8L8AH-mEIXf1HLssZQ1Q.jpeg" /></figure><p>By Mathurin Ache</p><p>In this post, we are going to answer these questions:</p><ul><li>What is experiment tracking in machine learning?</li><li>Why is experiment tracking essential ?</li><li>How should you manage/organize your machine learning experiments?</li><li>What is the most common way to organize data, pipeline and models in an experiment?</li><li>What are the advantages of using Prevision.io as an experience tracking solution?</li></ul><h3>What is experiment tracking in machine learning?</h3><p>In machine learning, experiment tracking is the practice of keeping track of important information throughout the lifecycle of a data science project.</p><p>A typical ML model development process involves collecting, preparing training data, selecting a model and training the model with prepared data. A small change in the training data, model hyperparameters, model type, or code written to run the experiment can dramatically change the model performance.</p><p>Experiment tracking is the process of saving all experiment-related information that you care about for every experiment you run.</p><p>Data scientists typically run different versions of models by modifying hyperparameters. Therefore, obtaining the best performing model based on one or more performance metrics is an iterative process. Without tracking the experiments conducted during the ML model development process, it is not possible to compare and replicate the results of different iterations. We talk about data versioning to keep track of all the datasets tested, pipeline versioning to keep track of all steps you follow to build your train dataset and experiment versioning to keep track of your models’ training, performance, time to train and hyper parameters.</p><p>The following can be worth thinking about while experiment tracking:</p><p>⇒ These lists are non-exhaustive lists, since the experiment metadata to be prioritized may vary according to the project’s characteristics. For instance, in the case of a computer vision project, it would be more efficient to trace visual information like the training curve, some examples of well and badly classified images…</p><p>⇒ It is strongly recommended to set up alerts, and to monitor data, models, and uses, throughout the data science project life cycle. We will have the opportunity to deal with this subject in a future article.</p><h3>Why is experiment tracking essential ?</h3><p>Experimental tracking allows data scientists to identify factors that would affect a model’s performance, compare results and select the optimal version. It is very important to stay organized throughout the iterative learning process,l even if your models do not make it into production</p><p>“Save everything in one place and never lose your progress again.”</p><h3>What is the most common way to organize data, pipeline and models in an experiment?</h3><p>Several solutions are possible, depending on your habits, the need for collaboration, the need for real-time monitoring, graphical analysis, etc.</p><ul><li>Level 0, the most basic, includes using Excel or Google SpreadSheet as tools to trace the experiments as we go along</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*04t3CEX6fjVwguSd" /><figcaption>Extract from an excel file tracing the different Machine Learning experiments</figcaption></figure><ul><li>Level 1 consists of tracking the different versions of the code used under your Github account. This requires having another tool for managing versions of datasets used, which can be done via Google Drive, Google Cloud Storage, Amazon S3, etc.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*A3fs_6wUK3OJa5bj" /><figcaption>Extract from a versioned directory under Github.com</figcaption></figure><ul><li>Level 2 involves using dedicated solutions such as Weights &amp; Biases which offers a fully documented, graphic, interactive and collaborative environment to trace all the experiments you attempt. Since it’s multi-environmental, you can run your code locally or on a third party environment such as Google Colab or Kaggle.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xLCBLCynQz7mrZ2O" /><figcaption>Experiment Tracking with <a href="https://wandb.ai/site">Weights and Biases</a></figcaption></figure><ul><li>Level 3 embodies centralizing everything you do in a single environment from training to production : this is what Prevision.io offers.</li></ul><h3>What are the advantages of using Prevision.io as an experience tracking solution?</h3><p>From our point of view, here is a list of features covered by the tracking and management experiment in the Prevision.io platform.</p><p>→ See all you do in a central dashboard:</p><ul><li>Have your activity view of full experiments dashboard</li><li>Control your model building and experimentation</li><li>Record everything you care about for every ML job you run (in Prevision.io or out)</li><li>Trace on which dataset, parameters, and code every model was trained on</li><li>Organize all the metrics, charts, and any other ML metadata organized in a single place</li><li>Make your model training runs reproducible and comparable with almost no extra effort</li><li>Make an Auto Report for documentation of your experiment</li></ul><p>→ Be more productive:</p><ul><li>Don’t waste time looking for folders and spreadsheets with models or configs. Have everything easily accessible in one place</li><li>Make a significant acceleration of the model building process</li><li>Reduce context switching by having everything you need in a single dashboard</li><li>Find the information you need quickly in a dashboard that was built for ML model management</li><li>Debug and compare your models and experiments with no extra effort</li><li>Use code (Python/R SDK) or no code (User Interface)</li></ul><p>→ Focus on ML: (we manage the traceability)</p><ul><li>Propose dedicated modules for vision, text, and tabular data</li><li>Help your team get started with excellent examples, documentation, and a support team ready to help at any time</li><li>Know when your runs fail and react right away</li><li>Don’t re-run experiments because you forgot to track parameters. Make experiments reproducible and run them once</li></ul><p>→ Collaboration features and project management tools:</p><ul><li>Share your works with your team and stop duplicating expensive training runs</li><li>Cut unproductive meetings by sharing results, dashboards, or logs with a link</li></ul><p>→ Scalable solution:</p><ul><li>Use computational resources more efficiently</li></ul><p>→ Build reproducible, compliant, and traceable models:</p><ul><li>Make every ML job reproducible. Keep track of everything you need</li><li>Have everything backed up and accessible from anywhere even years after</li><li>Know who trained a model, on what dataset, code, and parameters. Do it for every model you build</li><li>Be compliant by keeping a record of everything that happens in your model development</li></ul><p>→ Focus on the whole lifecycle of the machine learning process:</p><ul><li>Make it possible to include external models that can be integrated with any ML library or language</li><li>Do not require additional software or systems like containers, Prevision.io does it for you</li></ul><p>The tracking experiment in Prevision.io will be explored concretely, in pictures, in future articles of this blog post series.</p><h3>Conclusion</h3><p>In this first blog post, I wanted to introduce you to the subject of experiment tracking through answering these simple questions:</p><ul><li>What is experiment tracking in machine learning?</li><li>Why is experiment tracking essential ?</li><li>How should you manage/organize your machine learning experiments?</li><li>What is the most common way to organize data, pipeline and models in an experiment?</li></ul><p>Thanks for reading.</p><p><em>Originally published at </em><a href="https://content.prevision.io/experiment-tracking-with-prevision-io-part-1-3"><em>https://content.prevision.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32a3620f24ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/why-is-experiment-tracking-essential-in-ml-part-1-3-32a3620f24ea">Why is experiment tracking essential in ML Part 1/3</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[THE HIDDEN COSTS OF BRINGING MODELS TO PRODUCTION]]></title>
            <link>https://medium.com/prevision-io/the-hidden-costs-of-bringing-models-to-production-efb85b399f3a?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/efb85b399f3a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Thu, 02 Dec 2021 13:51:52 GMT</pubDate>
            <atom:updated>2021-12-02T16:23:29.640Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5bcU-DqyeEWdPmev32_DA.jpeg" /></figure><h4>TABLE OF CONTENT</h4><ul><li><strong>Modeling is not the final horizon, brace yourselves for production</strong></li><li><strong>When changing anything changes everything (Experiment Tracking)</strong></li><li><strong>You better get prepared for data changes (Monitoring &amp; Alerting)</strong></li><li><strong>Models get old — keep an eye on them (Feedback Loop &amp; A/B Testing)</strong></li><li><strong>Data preparation and “the kitchen sink” (Pipeline &amp; Scheduler)</strong></li><li><strong>What makes a great machine learning platform for production</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVicjrGLvm-rLWxDnuoHgw.png" /></figure><p><strong>Modeling is not the final horizon, brace yourselves for production</strong></p><p>As data-science practitioners we eat machine learning for breakfast. On a day-to-day basis, we’re dealing with dozens of packages and libraries to get the most of the data we have. But do we really believe that consistently improving a model’s accuracy is our final horizon? Of course not! Getting into production and staying in production should be our mission.</p><p>But argh, is it hard if you’re not well prepared: Besides all of the particular ways a machine learning project can fail — because of crappy data or badly defined objectives (a subject that in itself could fill an entire encyclopedia) — in this white paper, we’re revealing hidden costs. That is, all of those unseen efforts you have to deal with along the machine learning journey to get your model into production. Whether you’re an OG (old guy) data scientist like myself, just getting started, or somewhere in between, we all can benefit from looking more closely at the icebergs that can bring your project to a screeching halt.</p><p><strong>When changing anything changes <em>everything</em> (Experiment Tracking)</strong></p><p>As you know, a machine learning model is not a piece of code you write explicitly. It’s a very unique piece of code derived from data. The piece of code — namely, the machine learning model — is conditioned on many variables with very different origins</p><p>This diversity, of course, stems from the training data (read more about this issue in the section on getting prepared for data changes below). But it also grows out of the training procedure itself that’s defined by the training algorithm with its own hyperparameters. Even though some of these procedures are stochastic by nature, it happens in the same way that your favorite random number generator can influence a final, produced machine learning model. To an experienced software developer, it might sound crazy to rely on a learning procedure that never outputs the same result twice. It’s sad but true: Consistently achieving the same results with machine learning procedures is a challenge for everybody in the field.</p><p>Where so many variables can change, the best option is to track every change in a systematic way. If you consider going and maintaining your machine learning model in production, you better rely on a very consistent experiment-tracking platform.</p><p>So you might think that Git-like version control software is the silver bullet to track all changes. You might think, how wrong could it be? The thing is, you can’t tell. How many times have you been in a situation where your notebook suddenly doesn’t behave the same way it did six months ago? Thinking about what might have changed, you’ll start to question what went wrong:</p><ul><li>Is it because some of your favorite packages have been upgraded?</li><li>Did you forget to fix the random state of any stochastic procedures?</li><li>Did you simply fail to save some of the required environment variables?</li></ul><p>There are so many ways to fail to reproduce a machine learning model in the wild.</p><p>The best advice here is to rely on a dedicated platform to track all these external changes. This is even more true when you consider that inevitable team turnover will at some point force you to maintain someone else’s machine learning models. So if you want to escape from a future full of painful model maintenance issues, use tools to ensure model reproducibility from the very start of your journey.</p><p><strong>You better get prepared for data changes (Monitoring &amp; Alerting)</strong></p><p>During the modeling phase of a machine learning project, we as data practitioners typically don’t pay much attention to data changes. Because during this very creative phase, the objective is to experiment with different ways to represent data. We try daring feature engineering techniques, timorous data aggregation, or bold data enrichment, consistently comparing results to see if any of this improves or not. Here we won’t talk about the ideal recipe to establish a consistent validation procedure (this too could get an encyclopedia volume of its own). Instead, we’ll focus on what could — and certainly will — happen once your model validation job is done and after you get it into production.</p><p>Here the million- or billion-dollar question is not: Will the data go wrong or not? Instead it’s: How do I react when data does go wrong? Because, without a doubt, data will go wrong, I assure you!</p><p>There could be so many paths to failure that we propose to categorize them in two different buckets depending on whether the problem is between the keyboard and the chair — or from the real world.</p><p>The first category of failure? Cumbersome <strong>data issues found in production</strong> — badly formatted data, out-of-range data, or simply missing data. What they often have in common is that the root cause stems from an external “improvement” to the IT system of the company. To illustrate, a classic and annoying example is when the IT guy next door decides to rename some data features. As a human being, you can probably accept a <em>passengerId</em> variable to be renamed into <em>passenged_id</em> as an improvement, and implicitly adapt to it. But your machine learning model won’t be so flexible and it might silently fall back to a missing data case. Another classic example from the trenches is an implicit change of units in the data. When your model was trained with a U.S. datetime format, and then used in production with a European one instead, then months turn into days and vice versa — guaranteed chaos. Unless your model fails silently, every time you encounter these kinds of issues, you should be able to put a finger on a particular decision someone made that cascaded to your fragile model in production.</p><p>The second category — <strong>data distribution changes from the real world, aka data drift</strong> — is a bit more interesting to cope with as a data science practitioner. (We elaborate on the concept of drift in the feedback-loop section below). Data drift is the natural evolution of feature distribution seen as a random variable. For example, the average age of a fixed population increases as the population gets older. If a model has been trained a long time ago with age as a feature on a younger population, it might have difficulties accurately predicting age on an older population. Drift is generally a good signal to consider triggering a model retraining procedure.</p><p>There’s nothing that can protect you from these data-dependency failures because of the very specific nature of machine learning models. But if you’re aware of them, you can monitor them and have alerting mechanisms in place. That way, as soon as issues happen in production, you can quickly implement mitigation or fallback techniques.</p><p><strong>Models get old — keep an eye on them (Feedback Loop &amp; A/B Testing)</strong></p><p>Most of the time, a supervised machine learning procedure is not an online learning procedure. Even if you’re very proud of a model’s accuracy at the time you release it into production, it’s very likely to get worse over time unless you refresh your model periodically.</p><p>So how do you detect things going wrong? The answer is simple — but implementation is complex. To mitigate the risk of model decay, you simply need to check your model’s performance on a recurrent basis. In data science jargon, you need to compare actuals (real-world ground truth) and predictions (model outputs). The tricky part is that in most cases, the actuals can only be known a long time after the prediction has been made.</p><p>For instance, let’s look at the extreme but common situation of a credit-scoring model that outputs the probability of a given loan to be repaid. Clearly in this situation, you’ll ultimately get the ground truth when the loan is either fully repaid — or not. And this might take place years after you’ve granted the loan. Even worse, you’ll only get the ground truth for the loan you’ve granted <em>so far</em>. In this situation, you need to establish A/B testing strategies. A/B testing can help ensure that some not-so-bad-but-not-so-good loans are tested in the wild — and that you have a persistent feedback loop to collect this information long after the initial prediction was made.</p><p>Feedback loop and A/B testing strategies are the way to go to mitigate prediction breakdown. A vivid example of this is <a href="https://www.jwatch.org/na53777/2021/09/07/epics-sepsis-model-not-ready-prime-time">the sepsis model from the electronic medical records vendor EPIC</a>, which was widely used by the health industry for two years. Before that, an independent survey detected the model was flawed in the real world — and that turned out to be true. This could have been monitored and remedied sooner if solid feedback loops have been put in place.</p><p><strong>Data preparation and “the kitchen sink” (Pipeline &amp; Scheduler)</strong></p><p>Before feeding a machine learning model with data, we all know that the data must be prepared. From data cleaning to data enrichment to data conversion, a lot of our daily effort resides in the data transformation that we need to put in place before a model can predict anything. In fact, a common joke among data science practitioners is that 90 percent of our job is about data preparation.</p><p>We won’t argue whether this is perfectly accurate or not, but what remains is the absolute need for a machine learning model to be shipped alongside with its own data transformation. Here’s why you should, very early in your modeling process, consider distinguishing what belongs to the model itself and what belongs to the data preparation.</p><p>Once you’ve separated the model part and the data preparation part, you’ll need to rely on a pipeline infrastructure to execute the data transformation ahead of model prediction in a consistent manner. In addition to a decent pipeline infrastructure, you’ll also need a scheduler for your peace of mind. No one wants to resurrect an ugly notebook every morning to get the daily predictions up and running when the model is in production.</p><p>That’s why successful data science practitioners design pipelines and use scheduling capabilities to get the job done in an automated way. So forget about the data preparation “kitchen sink” notebooks — and go for a clean and consistent pipeline to host your model and transformation together in production.</p><p><strong>What makes a great machine learning platform for production</strong></p><p>If you’ve successfully put a model into production, you know that the modeling is just the tip of the iceberg. Indeed, modeling is the most visible part of our job — the part your company cares about because it delivers real, tangible value.</p><p>But you know that to support your precious model in production, a lot of work has yet to be done. We detailed the need for experiment tracking, monitoring and alerting, feedback loop and A/B testing, and pipelines and scheduling.</p><p>And we have good news for you. At Prevision.io, we’ve done this journey many times for our customers, and we’re dedicated to building the most compelling AI management platform. Using our platform enables you to deploy and maintain models in production, so that you can focus on what really matters to data science practitioners: Collecting the data, defining the objective, building the model, and letting the platform service the model for you.</p><p>Do what you do best. Apply your talents to solving business problems while we take care of the technical grunt work in building, deploying, and monitoring your high-performing models. If you would like to try for yourself, please visit us at <a href="http://www.prevision.io/">www.prevision.io</a> to learn more.</p><p>Written by Nicolas Gaude, Co-founder at Prevision.io</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efb85b399f3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/the-hidden-costs-of-bringing-models-to-production-efb85b399f3a">THE HIDDEN COSTS OF BRINGING MODELS TO PRODUCTION</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How a cross-validation strategy can drastically impact model performance.]]></title>
            <link>https://medium.com/prevision-io/how-a-cross-validation-strategy-can-drastically-impact-model-performance-60b5c4ac3a58?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/60b5c4ac3a58</guid>
            <category><![CDATA[model]]></category>
            <category><![CDATA[crossvalidation]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cross-platform]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Wed, 01 Dec 2021 16:14:58 GMT</pubDate>
            <atom:updated>2021-12-01T16:14:58.151Z</atom:updated>
            <content:encoded><![CDATA[<p>Written By Arnold Zephir</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NnRsID2pV5clkKVMqVA9tA.jpeg" /></figure><p>There is a common trope about data science that beginner data scientists often outperform more experienced ones and get models with 99,9% accuracy, compared to experienced data scientists, who rarely get 80%.</p><p>Yet, as the term implies, data science is all about data and science. And science means having a good validation protocol.</p><h3>Want high performance? Do not validate your model.</h3><p>Learning is a process that involves:</p><ol><li>Gathering data</li><li>Making a model or hypothesis that explains the experimental data</li><li>Checking that hypothesis on never-seen data</li></ol><p>(See this excellent<a href="https://www.scottaaronson.com/democritus/lec15.html">, if very technical article, about learning by Scott Aaronson.</a>)</p><p>Like human learning, machine learning should follow this protocol. Most scientists are fully aware of how important the third part of the process is, because building models that work only on the data you got is not very useful. But for those just starting out in data science, it’s a common mistake to neglect it.</p><p>Often, ignoring this step results in models that get very high performances while experimenting, but then collapse when deployed and used on real, never beforeseen data. This comes from two main phenomena:</p><ul><li>Data overfitting</li><li>Data leakage</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HQIo1tKwaw5SW_ty" /><figcaption><em>A poorly done modelization. Model scored 88,976 RMSE on training data, but a 346,703 RMSE on validation data.</em></figcaption></figure><p>As machine learning is very powerful, data overfitting can, and will, happen a lot. With thousands of parameters, you will always succeed in building a model that perfectly fits your experimental data, yet predicts few — or no — new data.</p><p>Another issue, data leakage, is when your model starts to memorize the relationship between samples and targets instead of understanding them. It’s common with data that’s split into groups, like categories of goods or people. The model might memorize the group’s most frequent target instead of the relationship between its feature and the target.</p><p>When confronted with a new category of goods, the model will then be clueless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*xFaf3RnE3__g7k4o" /><figcaption><em>A 100% accurate model, in green, and a good model, in black.</em></figcaption></figure><h3>Validation technique: Know about the future performance</h3><p>Validation techniques are an important part of a solid strategy in order to keep models and hyperparameters performing their best on future data.</p><p>It’s a set of techniques that helps guarantee that, in addition to getting a good model initially, the model will stay good when exposed to new data. In other words, the wonderful performance you got in the lab won’t collapse when your company agrees to use your model worldwide for piloting all its vital processes.</p><h3>Common techniques to prevent performance collapse</h3><p>Luckily, the validation problem has been widely studied and we, as data scientists, benefit from several methods to get model performance as good in production as it was during experimentation. All of these methods rely on a common principle — put data aside when training, and evaluate models on the data put aside.</p><p>Yet, the art is all about knowing which data to put aside.</p><h3>Level 0: Train test split</h3><p>The first technique is a very basic one, but it works: Get our data, split it in two sets, usually 80% and 20% of the size of the original dataset, respectively called the trainset and holdout. Then, train on one set, and measure performance on the other one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZbGHnWCsFO1Pag8X" /></figure><p>Even if it’s not foolproof, this technique often prevents too much overfitting and is the least time-consuming. Split your data, make one train pass,and validate! Just be sure to use a sample without replacement when splitting your dataset, so none of the sample data in the trainset is memorized and then automatically predicted during validation (except if you want to cheat).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/0*_T7LsIXh4C2CQiTJ" /><figcaption><em>Use two datasets: One for training, one for validation.</em></figcaption></figure><h3>Level 0.5: Train test split with good normalization</h3><p>Before moving on to the next technique, let’s talk about a common error that’s often underestimated.</p><p>Data scientists frequently normalize, or apply some kind of transformation, to data before training. When doing that, always fit your transformation on the trainset, not the whole dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*3pcG5vapkuJ42kBh" /><figcaption>Do</figcaption></figure><p>If you apply some kind of transformation, like normalization, to the whole dataset, some information will leak between the holdout step to the training step. You will get a little boost of performance from it, but that boost will disappear when the model goes into production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jbjB1lRKhVpqMXXx" /><figcaption><em>Don’t</em></figcaption></figure><h3>Level 1: Folds</h3><p>In the very basic technique of splitting a dataset, we did not address the topic of variance, or stability, of a model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BcPbePZHKeq8Hupr" /><figcaption><em>Variance represented as error bar on each model performance.</em></figcaption></figure><p>Variance is a way to quantify how much your performance will change if you change the split of your model, even if you preserve the underlying distribution across each trial.</p><p>It is a very good indicator of generalization capability, as your performance should not change that much if you randomly sample training data from your original dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*TNdxDPm9OnP5Chmv" /><figcaption><em>Which one would you put to real use? RF-1 is a little bit better, but if you take its standard deviation into consideration, represented as an error bar, expect to be less confident about its performance on future data.</em></figcaption></figure><p>If the performance changes significantly when you change your training data, it means that the way you built your model is very unstable and, in most cases, it will be very unpredictable when put in production.</p><p>It’s like reaching a high equilibrium on top of a very narrow mountain. You get to be the best, but any minor change in the data could throw everything off and plummet you to the bottom.</p><p>That’s why when building models, you should adopt a K-fold validation strategy:</p><ol><li>Choose an integer k (generally k = 3 or 5 works well).</li><li>Split your dataset into k parts.</li><li>Choose one part.</li><li>Train on the remaining (k-1) parts.</li><li>Validate on the part you chose in step 3.</li><li>Start again from step 3, until you round around all of your k parts.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*VT-h50Y9_mOpNFzc" /></figure><p>By splitting data in a round robin manner you got two results:</p><ul><li>Average performance</li><li>Variance of this performance</li></ul><p>There is no definitive way to solve the tradeoff of performance vs. variance, but as a data scientist, you must be aware of the consequences of selecting a high-variance model — and be able to explain that to your business coworkers.</p><p>In the Prevision.io platform, k-fold cross-validation is always done each time a model is trained, and variance is displayed on the model list of each version, so you can choose which one fits best with your team. (Or, more precisely, Prevision.io’s platform uses a stratified k-fold strategy — check it out in the next section.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pwZ5NcZY54mfsMKu" /><figcaption><em>Don’t feel that good about this one…</em></figcaption></figure><h3>Level 2: Stratify</h3><p>Even when folding, another problem would arise when the data is very unbalanced.</p><p>Let’s say that you are training a classifier on a dataset whose target rate is 1%, and you use a 5-fold strategy. On a dataset of 100,000, each fold will be 20,000 samples, and the training set on each round will be 80,000 samples. As the target rate is 1%, we expect to get 800 positives in the training set and 200 in the validation set.</p><p>Yet, if you just sample randomly, the distribution of your target rate will look this way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*uMgteIyBeS5XHJHC" /></figure><p>In approximately 10% of your folding round, the target rate difference between the trainset and the validation set will be more than 10%!</p><p>In the worst case, for example, your trainset could get 720 out of 1,000 of the existing positives,that is a 0.9% rate, while the validation set would get 280 positives, a 1,4% rate.</p><p>This Problem only increases with lower target rates. With a target rate of 0.1%, you can see a 10x factor between the target rate of the trainset and the validation set when splitting your fold with a basic random algorithm. One of the most common use cases where this problem arises is in building a fraud classifier, as it’s very common that fraud target rates are low.</p><p>Cross-validating data with this method is a bad practice as the performance and variance you gain are built on poor foundations. Expecting consistent and reliable performance is not possible.</p><p>In order to avoid this, you need to use a stratified k-fold cross-validation strategy, which is a way of splitting the data so that the trainset and validation set always have the same target rate. Even on a balanced dataset, the impact of a stratified k-fold is clearly visible compared to a simple k-fold split:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*ZiW8InCSaVcbn-14" /><figcaption><em>k-fold</em></figcaption></figure><p>To a stratified one where target rate is well distributed across folds :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*3bUts7ALyD6JVbuJ" /><figcaption><em>stratified k-fold</em></figcaption></figure><h3>Level 3: Custom Folds</h3><p>The last common technique, and a very important one, solves the problems that arise on two types of datasets:</p><ol><li>Dataset with time-indexed data</li><li>Dataset with columns that represent some kind of group, like goods or people</li></ol><p>With this kind of data, machine learning models sometimes get some information from future leakage, or group leakage, and instead of deducing general rules about the relationship between the features and target, it memorizes the average target value of a group or a period of time.</p><p>Let’s say, for example, that you want to build a model that predicts the prices that customers are willing to pay for fruits based on some of their features (e.g. weight, amount of sugar, origin, or color).</p><p>One of your columns probably is the category of fruit (apple, berries, etc.).</p><p>If you build a model that uses the category of fruit as a feature, you will probably get some good results. BUT, if a new kind of fruit comes in, your model will probably fail because instead of really learning what determined the price of a fruit, it instead memorized your existing catalog. So sometimes you need to build custom folds in order to know how your model will behave when a new item from an unknown category comes in. Custom folds depend entirely on your use case, and should be built manually with regards to the problem you want to solve in the context of your application .</p><p>Temporal data presents the same kind of trap. If you use just a random fold across your dataset, you’re going to leak some data from the future that you cannot expect to benefit from when applying the model in real usage.</p><p>For example, if you build a trainset with some data from each year over a 10-year span, your model will probably learn some target statistics from each year. For example, if in 2020 the average price of a fruit is 2,3€, you cannot use that for a year that hasn’t happened yet, e.g. 2023.</p><p>What you really want are general rules that stay true in the future, e.g. a price increase of 3% each year since 2000, or something like that.</p><p>Even if using stratified k-fold, you should not sample random data across the whole dataset, but instead build a new feature, often called “fold” that respects the unknown in your future data.</p><p>In the case of a fruit category, you could attribute an integer to each category of fruit and split your data regarding this number. In other words:</p><ul><li>Train model on banana and apples, validate it on citrus</li><li>Train model on citrus and apples, validate on banana</li><li>Train model on banana and citrus, validate on apple</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*Af5vj4-HGEqx2Pvf" /></figure><p>The performance will probably decline from building folds this way, but you could expect it to stay stable if your grocery gets a new kind of fruit.</p><p>For temporal data, in addition to folding over a component of the date (year, month, week), you should train on past data and validate on future data. This strategy is called temporal blocks folding and should always be used when dealing with temporal data. It might look like:</p><ul><li>Training on 2010 and 2011, validating on 2012</li><li>Training on 2010, 2011 and 2012, validating on 2013</li><li>Training on 2010, 2011, 2012, 2013 through 2019, validating on 2020</li></ul><p>These cross-validation strategies, both group folding and temporal block folding, are quite complex and you really need to understand your data well to implement them. Because building custom folds always leads to a dip in performance, you want to be sure that doing so serves a model stability purpose.</p><p>In the Prevision.io platform, you could define one of your columns as a custom fold and the cross-validation will be done along that fold. When you use the “timeseries” training type, the platform will automatically use a temporal block to compute its cross-validation score, so you don’t need to define blocks by yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*Ja0vZw9qsVGa1mXo" /></figure><h4>The impact of a cross-validation strategy</h4><p>Let’s see how a cross-validation strategy may impact your performance.</p><p>We built a dataset of sales, with two categories (product and date/time), ranging over 3 years, and decided to measure the performance by holding out sales from 2012 and training models on sales from 2010 and 2011.</p><p>The quality of our models will be measured as the RMSE on holdout, and we used 6 different folding strategies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/0*u_aCAKWyvsZsBpRP" /></figure><ul><li>No folds: We don’t input any folds, so Prevision.io will just use a random stratified k-fold along the trainset</li><li>Random folds: We use a 7-fold, built randomly</li><li>Trimester folds: Trimesters (4) are used as fold</li><li>All month folds: Months (12) are used as fold</li><li>Triplet month folds: The month number modulo 3 is used as fold (e.g, we train on January and February, and validate on March)</li><li>Store folds: We use the store number as fold</li></ul><h3>Results</h3><p>Here is a summary of the model’s performance for each strategy. Remember, the lower the RMSE, the better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*xc8-0XJ8lSfw35s5" /></figure><ul><li>We see that there are 3 main types of input from our strategy:</li><li>Strategies with no smart folds get very good models (no folds and random rolds) in the trainset, as their RMSE are the best. But when applied on holdout, the performance totally collapses. You should never use models like that.</li><li>Some models (rimester folds and ll month folds) get the worst score, but remain more stable, and we get a similar performance in holdout. Yet the variance in cross-validation, and the fact that the holdout score is lower than the trainset score, must trigger attention. These models should probably be tested on more data to validate their stability or lack of stability.</li><li>The triplet month fold strategy gets some good performance on the trainset and a very similar one on the holdout. The variance of cross-validation is still high, but could probably be lowered with more powerful models. Of the three, this fold strategy is probably the strongest, and more powerful modelisation algorithm could now be used to get better performance</li></ul><p>Note that folding on the store columns get some interesting results, but this is probably due to the lack of temporal folds, which artificially decrease the RMSE and improve their performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60b5c4ac3a58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/how-a-cross-validation-strategy-can-drastically-impact-model-performance-60b5c4ac3a58">How a cross-validation strategy can drastically impact model performance.</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Simple steps to use Machine Learning for Image Classification]]></title>
            <link>https://medium.com/prevision-io/5-simple-steps-to-use-machine-learning-for-image-classification-1cc35ae49e65?source=rss----72b644e841a2---4</link>
            <guid isPermaLink="false">https://medium.com/p/1cc35ae49e65</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[image]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Prevision.io]]></dc:creator>
            <pubDate>Tue, 30 Nov 2021 16:53:53 GMT</pubDate>
            <atom:updated>2021-11-30T16:53:52.808Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Zeineb Ghrib</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55NrHduXQtYxgJF1xAQqOg.jpeg" /></figure><p>In this tutorial, you will learn how to use the Prevision.io UI interface to build a powerful image classifier without an advanced technical background. The machine Pipeline process is fully automated thanks to the Prevision.io platform.</p><h4>The dataset</h4><p>The dataset used for this example is composed of images of different components of a train:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/0*5bfv4C_nvMs14d9s" /></figure><p>Each image is associated with a label from 4 classes:</p><ul><li>Faces</li><li>Pantographs</li><li>Rails</li><li>Roofing</li></ul><h4>How an Image Classification experiment works</h4><p>Image classification is a supervised learning problem: each image is labelled with its corresponding name, and then the classifier is trained to recognize them using the labeled pictures.</p><p>In order to launch a Prevision AutoML experiment to the an image dataset, you need to follow these steps:</p><h4>Step 0 — Sign up for a free trial of Prevision.io</h4><p>Visit <a href="http://www.prevision.io/">www.prevision.io</a> &gt; Click Free Trial</p><h4>Step 1 — Create a new project or use an existing one</h4><p>In order to launch a new experiment, create a project or select an existing one so that you can gather experiments by type or by shared users.</p><p>To create a new project click on “New Project” on the corner right of the interface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*bhR5OW3RVWm7FapT" /></figure><p>Then you add the name of the project and optionally a description as shown below then click on “Create” button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/0*AMdSZ0kIvnNFjsQv" /></figure><h4>Step 2 — Upload an Image Folder</h4><p>An “Image Folder” is a ZIP containing images that will be used to train models. To upload the folder, all you have to do is to select “Image folders” tab on the head of the dataset section, then drag and drop into the image box (see figure below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*2cgRYh7K2JOrKiyp" /></figure><h4>Step 3 — Upload a tabular dataset</h4><p>In order to launch the experiment, you also need to provide a csv that would carry out the mapping between the images (available in the image folder) and their corresponding labels. It should contain, at least:</p><ul><li>the filename: relative path to images within the image folder</li><li>the target : the corresponding label</li></ul><p>Since Prevision.io also supports exogenous features for images experiments, this tabular dataset may also contain features that can be used as predictors to the models (temperature, nebulosity, time stamp of the picture, camera brand…)</p><p>To upload the dataset, you pick the “Datasets” tab, click on “Import dataset” button, then pick the “Import file” option:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/0*w3Hr5RSVYkNk-bmH" /></figure><h4>Step 4 — Launch an experiment with Prevision.io</h4><p>As we deal with a multi-labelled dataset, it is a multi-classification training type: to launch the experiment go to:</p><ol><li>Experiments Section</li><li>Click blue button ‘New experiment’</li><li>Select the ‘AutoML’ mode</li><li>Data type: ‘Images’</li><li>Training type: ‘Multi-Classification’</li><li>Choose a name to the experiment</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*1Ge6bWLhMbINQI7j" /></figure><p>Then launch the experiment by clicking on “Create Experiment”. Before launching the experiment, some settings are required:</p><ol><li>Select the pre-imported dataset and the Image folder</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/0*BNFQga7rkG9Sr263" /></figure><p>2. Then chose the training options:</p><ul><li>The metric that will be used to train the models</li><li>The training profile (Quick, Normal or Advanced)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/0*maErI3p2ROK_vl5P" /></figure><p>3. Then configure the target and the images path columns by choosing the corresponding features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/0*qFHZJDrXQPsSwOPh" /></figure><p>4. To configure the other experiment’s parameters (models, feature engineering ops…), you can browse the other tabs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/0*GRVQhxp3Y1LKVLIR" /></figure><p>Note that this step is optional: the platform provides reasonable default settings for all the training types, making it easy to quickly create baseline experiments.</p><p>Once the experiment is launched, the platform performs several computations on the image dataset to extract images “embeddings” that will be used as input instead of the initial images. These embeddings are numerical vectors that carry efficient representation of the initial image input.</p><p>The image embedding operation is performed for two reasons:</p><ul><li>Transform the images into numerical vectors that are more ‘appreciated’ by the models: in fact numeric input is less sophisticated than images pixels</li><li>Reduce the dimensionality of the data input</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*26TYhLbNMCsedIkn" /></figure><p>Once the experiment process finished, you can visualize the new generated embedding-features (even if it’s hard to find a nice name for them 🙃):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/0*tXvjItmuJBYrzQJ2" /></figure><h4>Step 5 — Prediction</h4><p>To make a Prediction on new images: you have to:</p><p>1. Upload the ZIP of images you want to predict creating a new Prevision.io image folder (step 2)</p><p>2. Upload the corresponding csv ( as explained on step 3)</p><p>3. Go to the prediction tab, choose your model and launch the predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*PiYmMKTNzM-dfxoE" /></figure><p>In step 4 (see figure above) after launching the prediction, a line will appear within the ‘User Generated Predictions’ tab’. Once the status is done you can download the prediction file (step 5).</p><h3>Conclusion</h3><p>In this post you learned how to use Prevision.io, to create an image classification experiment, in a future post I will show you how to use the platform to create an ‘object-detection’ experiment. If you have any issues or need help, please contact me <a href="https://content.prevision.io/contact-form">here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cc35ae49e65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prevision-io/5-simple-steps-to-use-machine-learning-for-image-classification-1cc35ae49e65">5 Simple steps to use Machine Learning for Image Classification</a> was originally published in <a href="https://medium.com/prevision-io">Prevision.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>