<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Himanshu Chandra on Medium]]></title>
        <description><![CDATA[Stories by Himanshu Chandra on Medium]]></description>
        <link>https://medium.com/@himanshuhc?source=rss-88960b2eb891------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qfoFHdA_85DdBH_GTE-tOA.jpeg</url>
            <title>Stories by Himanshu Chandra on Medium</title>
            <link>https://medium.com/@himanshuhc?source=rss-88960b2eb891------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:12:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@himanshuhc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How To Acquire Clients For AI/ML Projects By Using Probability]]></title>
            <link>https://medium.datadriveninvestor.com/how-to-acquire-clients-for-ai-ml-projects-by-using-probability-e92ca0f3ba68?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/e92ca0f3ba68</guid>
            <category><![CDATA[binomial-distribution]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-strategy]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Thu, 30 Jul 2020 03:52:42 GMT</pubDate>
            <atom:updated>2020-07-30T20:03:21.308Z</atom:updated>
            <content:encoded><![CDATA[<p>Beating competing bids via binomial theorem &amp; statistics</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rm2M-qjnWL01Nxzs" /><figcaption>Photo by <a href="https://unsplash.com/@icons8?utm_source=medium&amp;utm_medium=referral">Icons8 Team</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Quite often, we machine learning practitioners get swept away in the rush of applying different models to solve a certain problem. Extensive use of statistical tests takes either a back seat or comes into play only when presenting end results to the client.</p><p>While certain metrics like ROC, PR curves &amp; MCC help you fine tune your results at the fag end of your project (<em>read</em>: <a href="https://towardsdatascience.com/how-not-to-use-roc-precision-recall-curves-mcc-matthews-correlation-coefficient-f68a33108f8b">How NOT to use ROC, Precision-Recall curves &amp; MCC</a>), certain other metrics might help you win new projects for your organization if used at the start itself.</p><p><a href="https://towardsdatascience.com/how-not-to-use-roc-precision-recall-curves-mcc-matthews-correlation-coefficient-f68a33108f8b">How NOT to use ROC, Precision-Recall curves &amp; MCC (Matthews Correlation Coefficient)</a></p><p>Today, I present a case study of how we typically bag new projects and how metrics like p-values, confidence intervals and concepts like binomial series help us gain customers’ trust.</p><h3>The Case In Point</h3><p>This particular project was one of <a href="https://en.wikipedia.org/wiki/Computer_vision">computer vision</a> for the manufacturing industry. The client produces medium-sized metallic parts which are visually inspected for cracks, dents, rust and few other categories of defects, at the end of the production line. An image based automated solution was required to replace the current manual process — since their daily production volume was on the rise — currently at around 1,500 parts per day.</p><p>Bids from multiple software vendors was invited, and we were one of them. As often happens, the client had a great record and expertise in the manufacturing domain, but not quite so much in the AI/ML field. The typical approach they planned to take was to check past projects, company size, commercial quotation and try to quantify that into a general sense of confidence for a particular service provider.</p><h3>The Gamble</h3><p>We were not particularly happy about this approach, being a growing, but a small company still. We were however, confident of our skills and hence I proposed this to the client:</p><blockquote>Why don’t we test our initial models, one month from now, on 1,000 parts and amongst all the vendors, see whose model performs the best?</blockquote><p>This would become a PoC (Proof of Concept) and also help us estimate accurately for the longer engagement. By the time I proposed this, they had already narrowed the list down to us and another competitor. They were, however onboard with the PoC idea, since it seemed more quantifiable than their current methods and asked us and the competitor to get started on the initial classification model.</p><h3>The Catch</h3><p>How do you confidently say someting is better than the other based on just one classification run over 1,000 samples? After all, this model would be run on 350,000+ parts per year. We were questioned:</p><blockquote>Is such a ‘sample run’ reliable enough for comparison?</blockquote><p>Also, when we say ‘better’, what metric are we considering here? Is it accuracy, precision, recall, specificity, AUC or something else?</p><p>The second question is easier to answer, so let’s start with that. Our client was very clear in the fact that they track <strong>precision</strong>. So nothing we could discuss further there. <em>For a quick review of the confusion matrix and the associated metrics, read -</em></p><p><a href="https://towardsdatascience.com/the-two-variations-of-confusion-matrix-get-confused-never-again-8d4fb00df308">The Two Variations of Confusion Matrix — Get Confused, Never Again</a></p><p>To anwer the first question, we first set an expectation, a confidence level — 95% in this case. We told them that whatever the result of comparing the two models may be, we would be 95% confident that it is not erroneous. We could easily have chosen 90% or 99% or any other number, but they were fine with 95% at the PoC stage.</p><p>Now to compare classification models, there are a few statistical methods; <a href="https://machinelearningmastery.com/mcnemars-test-for-machine-learning/">McNemar’s paired test</a> is one which is commonly used. It could work here, but we wanted<em> </em>something more intuitive which could be trusted by the client too, when explained in day-to-day terms.</p><h3>Binomial Series To The Rescue</h3><p>Imagine this: Suppose you flipped a coin and got a head. You flipped again and got a head again. Then a tail. Then a tail...</p><p>After 20 flips, you saw that you had 7 heads and 13 tails. If I asked you to be 95% confident in your statement, would you say that the coin is a fair one, or is it biased towards tails? After all, one knows from experience that more often than not, you do not get an exact 50–50 split of heads-tails in 20 flips. That does not mean the coin is biased.</p><p>Alternatively put, how many tails would you want out of 20 flips, before saying that the coin seems to be favouring tails?</p><blockquote>Or that the probability of tail performing better than head is significantly higher?</blockquote><p>The answer to the above question is this:</p><figure><img alt="Tails required to prove a tail-biased coin" src="https://cdn-images-1.medium.com/max/1024/1*oVKqQV_8cjArbBCwCu0NrA.png" /><figcaption>Tails required to prove a tail-biased coin (Image copyright : Author)</figcaption></figure><p>The above is derived by applying the <a href="https://stats.stackexchange.com/questions/21581/how-to-assess-whether-a-coin-tossed-900-times-and-comes-up-heads-490-times-is-bi">Binomial Formula</a>. You can also find this out quickly using python:</p><pre>from scipy.stats import binom</pre><pre>N, p, alpha = 20, 0.5, 0.05</pre><pre>distribution = binom(N, p)</pre><pre>heads = distribution.ppf(alpha)</pre><pre>if distribution.cdf(heads)&gt;alpha:</pre><pre>   heads = heads -1</pre><pre>print(f&quot;At {100*(1-alpha)}% confidence level, minimum tails required out of 20 flips for coin to be tail-biased: {format(N-heads,&#39;.0f&#39;)}&quot;)</pre><p>The output for N=20 (twenty flips or trials) is:</p><pre>At 95.0% confidence level, minimum tails required out of 20 flips for coin to be tail-biased: 15</pre><p>This is pretty much the same question we want answered for our PoC, where ‘tail’ is our model and ‘head’ is the competitor’s model. If we start out by assuming that both models are basicially similar, none better than the other, then both models should have a 50% probability of scoring higher precision after testing on multiple datasets.</p><blockquote>But if we test it on 20 datasets, and see that model1 has a higher precision on 15 occasions, then we can conclude with 95% confidence that model1 is better than model2, precision-wise.</blockquote><p>So that’s what we decided to ask for. We requested the client to split the 1,000 unique samples/parts into <strong>20 groups of 50 parts each</strong> such that each group has a fair &amp; similar representation of the defects found in production and we ran the two models on these 20 groups, noting down how many times our model ‘won’.</p><p><a href="https://www.datadriveninvestor.com/2020/06/24/disclosure-and-resolution-program-wont-prevent-physicians-from-practicing-defensive-medicine/">Artificial Intelligence, Deep Learning, and Medical Practice | DataDrivenInvestor</a></p><p>Our model showed a higher precision 16 out of 20 times. Looking back, we could even have gone for 99% confidence, and won! But then, hindsight’s always 20/20.</p><blockquote>You can change alpha to 0.01 in the above code to see the output as 16.</blockquote><h3>Learnings</h3><p>It was much more reliable for the client to see that we won 16 out of 20 times with smaller datasets, instead of just witnessing a single win with a bigger sample size.</p><p>Knowing that we could deliver a better model was not enough, showcasing a rough version to the client as quickly as possible in the most intuitive way, was.</p><blockquote>Interested in knowing how similar metrics help us gain customer confidence and ensure we serve their business needs better? Read:</blockquote><p><a href="https://towardsdatascience.com/a-common-mans-guide-to-mae-and-rmse-d5efcd238221">A common man’s guide to MAE and RMSE</a></p><h4>References:</h4><ul><li><a href="http://www.jmlr.org/papers/volume7/demsar06a/demsar06a.pdf">http://www.jmlr.org/papers/volume7/demsar06a/demsar06a.pdf</a></li><li><a href="https://www.researchgate.net/post/How_would_you_determine_whether_a_classifier_is_significantly_better_than_random_guessing">https://www.researchgate.net/post/How_would_you_determine_whether_a_classifier_is_significantly_better_than_random_guessing</a></li><li><a href="https://machinelearningmastery.com/mcnemars-test-for-machine-learning/">https://machinelearningmastery.com/mcnemars-test-for-machine-learning/</a></li><li><a href="https://stats.stackexchange.com/questions/21581/how-to-assess-whether-a-coin-tossed-900-times-and-comes-up-heads-490-times-is-bi">https://stats.stackexchange.com/questions/21581/how-to-assess-whether-a-coin-tossed-900-times-and-comes-up-heads-490-times-is-bi</a></li></ul><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me through my website, <a href="http://iamjustastudent.com/about">I am Just a Student</a> or on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a> or <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a>.</p><p>See you around &amp; happy learning!</p><p><a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">Himanshu Chandra - Business Head - AI, ML &amp;amp; QA - Ideas To Impacts | LinkedIn</a></p><p><strong>Gain Access to Expert View — </strong><a href="https://datadriveninvestor.com/ddi-intel"><strong>Subscribe to DDI Intel</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e92ca0f3ba68" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/how-to-acquire-clients-for-ai-ml-projects-by-using-probability-e92ca0f3ba68">How To Acquire Clients For AI/ML Projects By Using Probability</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How NOT to use ROC, Precision-Recall curves & MCC (Matthews Correlation Coefficient)]]></title>
            <link>https://medium.com/data-science/how-not-to-use-roc-precision-recall-curves-mcc-matthews-correlation-coefficient-f68a33108f8b?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/f68a33108f8b</guid>
            <category><![CDATA[recall]]></category>
            <category><![CDATA[roc]]></category>
            <category><![CDATA[specificity]]></category>
            <category><![CDATA[precision]]></category>
            <category><![CDATA[sensitivity]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Sat, 11 Jul 2020 15:20:53 GMT</pubDate>
            <atom:updated>2020-07-11T18:00:45.390Z</atom:updated>
            <content:encoded><![CDATA[<h4>Common mistakes we make with these metrics &amp; how to choose the one best for your business case</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5d7SZ2e4p6hZGOEB" /><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There are quite a few things about being a data scientist that I love. Developing a holistic sense of the business you are creating a solution for, is one of them. Having that real-world context to the formulae we use makes it all the more fun. That’s why when a junior colleague in my team showed me the comparison of various models via the Precision-Recall curve, and I enquired if it was the best metric for our case, his reaction was, “I can display this via ROC curves; maybe that would be better?”</p><p>As every experienced data scientist will tell you, there’s no one right or wrong. The business dictates the best approach. So, let me pose the same problem to you, and let’s walk through what could be a good approach.</p><blockquote>All images are owned by the author unless otherwise stated.</blockquote><h3>Problem Statement</h3><p>We are in the process of implementing an automatic quality control system for a manufacturing plant which produces gear parts. It’s a computer vision problem. As an initial step, we have picked up one gear part and trying to classify it into ‘good’ or ‘bad’ based on factors like presence of scratches, rust, incorrect dimensions etc. The type of defect found is unimportant right now, just classifying it as good or bad is required.</p><p>Also, the penalty for shipping a bad part to end customer is huge, so we should be very particular in rightly classifying the bad parts.</p><p>Let’s assume the actual and predicted (from our model) results give us the following <a href="https://towardsdatascience.com/the-two-variations-of-confusion-matrix-get-confused-never-again-8d4fb00df308">confusion matrix</a>, where we consider a <strong>good part as <em>Positive</em></strong> and a <strong>bad part as <em>Negative</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*RqzMeF6Vnfgz6aA28Yh4mw.png" /></figure><p>If you want to brush up on the various metrics associated with the confusion matrix like Precision, Recall, Specificity or prefer the ‘other’ representation where predictions serve as columns, I would recommend reading <a href="https://towardsdatascience.com/the-two-variations-of-confusion-matrix-get-confused-never-again-8d4fb00df308"><em>The Two Variations of Confusion Matrix - Get Confused, Never Again</em></a> for a greater sense of clarity.</p><p>Of course, the above matrix is based on a particular probability threshold (0.5 in this case) and the ROC curve will be constructed by considering various thresholds, but you get a decent idea of the general class distribution.</p><p>Which metric should you use? -</p><ol><li>ROC curve</li><li>Precision-Recall (PR) curve</li><li>Something else?</li></ol><h3>Thought Process</h3><p>First of all, let’s eliminate ROC curve as it is not best suited for imbalanced class problems. Here’s a great <a href="https://www.youtube.com/watch?v=4jRBRDbJemM">video</a> to get one’s foundation right for this.</p><p>On to <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html">Precision-Recall</a> curve now - this is what my colleague had chosen. It has Precision as y-axis and Recall as x-axis. If I use the above confusion matrix at various thresholds to plot this curve, do you see any problem in the approach?</p><blockquote>The Precision-Recall curve is supposed to draw our attention to the P<strong>ositive class</strong>.</blockquote><p>The positive and negative class ought to be swapped if we want to concentrate on the ‘bad’ class. This is akin to a medical use-case where detection of cancer is equivalent to the test coming back ‘<strong>Positive</strong>’. If only few people/parts are going to have cancer/defects then we make it the positive class and then use the PR curve.</p><p>Another non-traditional way to do this, if you do not want to swap class labels, is to map the curve with y-axis as <strong>TN/(TN+FN)</strong> and x-axis as Specificity or <strong>TN/(TN+FP)</strong>. I could go into the maths to explain these new axes, but a very intuitive way to remember this is:</p><p><strong>Case 1</strong> (Traditional PR curve): With TP as the ‘pivot’, pivot divided by row sum is the y-axis (Precision), while pivot divided by column sum is the x-axis (Recall).</p><figure><img alt="“Traditional” PR curve" src="https://cdn-images-1.medium.com/max/533/1*aaC8IIMAqou_c8QZkcQM2Q.gif" /><figcaption>“Traditional” PR curve</figcaption></figure><p><strong>Case 2 </strong>(Non-traditional curve): Since you wish not to swap labels, we make TN as the pivot. Once again, exactly as above, pivot divided by row sum is the y-axis, while pivot divided by column sum is the x-axis (Specificity).</p><figure><img alt="“Non-traditional” curve" src="https://cdn-images-1.medium.com/max/533/1*UzuKwWs0AGkYF5DJ_3jGIQ.gif" /><figcaption>“Non-traditional” curve</figcaption></figure><p>An ideal PR curve has the ‘best’ value at (1,1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*OVyzJzdPEuvWgRpsUASmPg.png" /></figure><h3>Some other Metric?</h3><p>This article would be incomplete without a mention of Matthews Correlation Coefficient or MCC.</p><p>MCC is suitable for cases where you want to gauge performance of your model in predicting all the classes, with no particular focus on any specific one. Here’s a <a href="https://lettier.github.io/posts/2016-08-05-matthews-correlation-coefficient.html">great blog</a> to read on the topic.</p><p>Imagine this — pretty soon, our problem statement will expand and the client will ask that all the different defects be classified separately, instead of just ‘good’ or ‘bad’. So it could be ‘scratches’, ‘wrong dimension’, ‘rusted’, ‘good’ — so on and so forth.</p><p>Once we shortlist a few models, it would be great to have a single metric to compare their overall performance over different classes. MCC just does that.</p><p>Hope this gave you a glimpse into the different scenarios where different metrics are valuable. It’s always a good idea to understand the mathematics behind them all, as only then one is able to tweak them as per one’s particular needs — like we tweaked the PR curve into a non-traditional one.</p><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me through my website, <a href="http://iamjustastudent.com/about">I am Just a Student</a> or on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a> or <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a>.</p><p>See you around &amp; happy learning!</p><p><a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">Himanshu Chandra - Business Head - AI, ML &amp;amp; QA - Ideas To Impacts | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f68a33108f8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-not-to-use-roc-precision-recall-curves-mcc-matthews-correlation-coefficient-f68a33108f8b">How NOT to use ROC, Precision-Recall curves &amp; MCC (Matthews Correlation Coefficient)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An intuitive approach to DTW — Dynamic Time Warping]]></title>
            <link>https://medium.com/data-science/an-intuitive-approach-to-dtw-dynamic-time-warping-f660ccb77ff4?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/f660ccb77ff4</guid>
            <category><![CDATA[dynamic-programming]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[recursion]]></category>
            <category><![CDATA[dynamic-time-warping]]></category>
            <category><![CDATA[dtw]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 18:57:51 GMT</pubDate>
            <atom:updated>2020-06-29T05:43:18.869Z</atom:updated>
            <content:encoded><![CDATA[<h3>An intuitive approach to DTW — Dynamic Time Warping</h3><h4>How to visualize the algorithm, so that you can customize it on your own.</h4><figure><img alt="Warp" src="https://cdn-images-1.medium.com/max/1024/1*yOGITbltunSw9W8hcJ-ToQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@nxvision?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Nigel Tadyanehondo</a> on <a href="https://medium.com/photos/i9gR_dz_xzU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Since you are here, I assume you already know the reason why we use <strong>D</strong>ynamic <strong>T</strong>ime <strong>W</strong>arping, or DTW in time-series data. Simply put, it’s used to align or match two similar patterns.</p><h3>A brief overview</h3><p>One of the reasons DTW was initially developed was for speech recognition. Your mother may speak slowly one day, and hurriedly on another; even have a bit of cold and sore throat on some days, but you still can recognize her voice. Can machines do the same? Can they somehow match the highs and lows, the peaks and troughs, the frequencies of her voice, no manner how she speaks and tell us it is her voice indeed?</p><p>There are several awesome real life situations were DTW just rocks — if you do not know them already, I recommend getting to know them — they are really fun!</p><blockquote>You can read more about DTW <a href="https://en.wikipedia.org/wiki/Dynamic_time_warping">here</a> and about it’s many applications <a href="http://seninp.github.io/assets/pubs/senin_dtw_litreview_2008.pdf">here</a> (refer the <em>Introduction</em> section)</blockquote><figure><img alt="Two signals being aligned" src="https://cdn-images-1.medium.com/max/775/1*lPecfOMHaASWUcFXjaB4jQ.png" /><figcaption>Two signals being aligned — “Image by author”</figcaption></figure><h3>Take me there, quick!</h3><p>I love this algorithm! In essence, it is one of those short and elegant algos which pack a punch. It is a classic example of finding the shortest path using the <a href="https://en.wikipedia.org/wiki/Dynamic_programming"><em>Dynamic Programming</em></a> approach.</p><p>I have had colleagues running away from understanding the inner workings of the algo as it involves <a href="https://en.wikipedia.org/wiki/Recursion_(computer_science)"><em>recursion</em></a><em>. </em>This eventually stops them from understanding the nuances of the approach and from learning how to tweak it as per their requirements.</p><p>Let’s visualize the logic behind the algorithm in a non-programmatic way so that we can write it from scratch.</p><blockquote>To understand recursion, one must first understand recursion.</blockquote><blockquote>— Stephen Hawking</blockquote><p>Imagine you are standing at the blue square and wish to go to the red square. All the numbers that you see in the cells along the path, correspond to the toll amount you have to pay at each step (arbitrarily decided, for now). If I were to ask you, “<em>Tell me the least amount one needs to spend to reach from blue to red</em>”, how would you do it?</p><figure><img alt="Find the ‘cheapest’ route" src="https://cdn-images-1.medium.com/max/401/1*jXVEZO662GINnPOaDOgDxQ.png" /><figcaption>Find the ‘cheapest’ route — “Image by author”</figcaption></figure><p>In fact, I’ll make it simpler by saying that you are supposed to move only in the ‘forward’ direction. The black arrows show the only 3 ‘allowed’ directions (right, down, right-down diagonal) and red arrows show ‘restricted’ directions, which are everything apart from the 3 allowed ones (up, left, other diagonals, etc.). This is similar to what DTW assumes before solving the problem.</p><figure><img alt="Allowed directions" src="https://cdn-images-1.medium.com/max/395/1*EDbUq61O55WxcIpIXLYr9w.gif" /><figcaption>Allowed directions — “Image by author”</figcaption></figure><p>Another way to say the above fact is that, “You can arrive at any square only from one of the 3 adjacent squares”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*_D5M5XxFthueepO4coSGPg.gif" /><figcaption>Another way to look at it — “Image by author”</figcaption></figure><p>A ‘brute-force’ way would be to try all the paths possible from the blue to the red square and choose the cheapest one. However, dynamic programming and recursion give us a better, smarter way.</p><h3>The lazy boss</h3><p>I like to solve all recursion problems by thinking as ‘the lazy boss’ who has uncountable minions at his disposal to do the job. If someone asked me to solve the above problem, I would simply say, “Minions A, B and C, tell me the least cost to reach the 3 squares surrounding the red square, and then I’ll calculate the answer in a minute. Till then, don’t bother me.”</p><blockquote>This is the trick recursion uses — give the <strong><em>exact same kind of work</em></strong> you have been asked to do, to your underlings, after ‘scaling down’ the work a bit. And make sure to keep something simple for yourself, so that no one can say you did nothing!</blockquote><p>Suppose by some magic (which you are not interested in knowing about), the minions bring you the answer, marked in green cells:</p><figure><img alt="Answers of Minions A, B and C" src="https://cdn-images-1.medium.com/max/393/1*vAc5YbC4dLPGQ_0eIXhNwg.png" /><figcaption>Answers of Minions A, B and C — “Image by author”</figcaption></figure><p>All I need to do now is find the <strong>minimum</strong> of <strong>(10, 7, 7)</strong>, which is 7 and then add current square’s cost, which is 2. This gives me the answer as <strong>9</strong>. Done!</p><p>9 is the minimum cost of travelling from blue to red. But wait, how did the minions come up with the green values? They of course, emulate you and act as lazy bosses too!</p><p>Each minion gets hold of 3 junior minions (the damned hierarchy in the office, I tell you!) and tells them to bring them least cost values for their 3 neighbourhood/adjacent squares. Here’s an animation to better explain this:</p><figure><img alt="Minions at work" src="https://cdn-images-1.medium.com/max/747/1*yXV5QgCRsi3v7HYu_v6z0g.gif" /><figcaption>Minions at work — “Image by author”</figcaption></figure><p>This keeps going on and on, with each junior minion ‘magically’ getting its answer. But for how long? Surely it ends sometime? Yes, it does. <strong>When there is no way to subdivide the work further.</strong> When a minion gets the task to calculate the cost at the blue square, he can’t delegate it to anyone else, as there is no neighbourhood to go to. He just needs to know that the cost there is zero.</p><p>That’s it. You can translate this into a recursive python function in just a few lines of code:</p><pre>def cheapest(cost, i, j):<br> if (i==0) &amp; (j==0):<br>     return 0  ## can&#39;t subdivide the work anymore</pre><pre> if (i&lt;0) | (j&lt;0):<br>     return np.inf</pre><pre> ## current square&#39;s cost + minimum of the 3 neigbours<br> return cost[i][j] + min(cheapest(cost, i-1, j-1), <br>                         cheapest(cost, i-1, j), <br>                         cheapest(cost, i, j-1))</pre><p>Here, cost is a 2-D array, our initial matrix:</p><pre>cost = [<br> [0,2,5,1,1],<br> [5,3,4,2,2],<br> [1,1,6,1,3],<br> [1,3,2,1,2]<br>]</pre><p>Notice how I have returned np.inf (infinity) when either i or j is less than zero. This is just a way to ignore the neighbour which doesn’t exist. For example, when you are in a top-row square, the only neighbour you need to consider is the left one, as there’s no square above the current square. Similar logic goes for squares in the first column, where there’s no neighbour to the left.</p><p>And you can call the recursive function by simply calling:</p><pre>output_cost = cheapest(cost, 3, 4)<br>print(output_cost)</pre><p>The output it will give is 9.</p><p>If you call it for all the squares, you can finally create a cheapest cost matrix as output:</p><pre>output_cost_matrix = []   ## used to store all outputs<br>for i in range(4):<br>   for j in range(5):<br>      output_cost_matrix.append(cheapest(cost, i, j))</pre><pre>## reshape the output for better display<br>output_cost_matrix= (np.array(output_cost_matrix)).reshape(-1,5)<br>print(output_cost_matrix)</pre><p>Output:</p><pre>[[ 0,  2,  7,  8,  9],<br> [ 5,  3,  6,  8, 10],<br> [ 6,  4,  9,  7, 10],<br> [ 7,  7,  6,  7,  9]]</pre><p>So, what would be the cheapest <strong>path</strong> from the blue square to the red square?</p><p>We just need to track the <em>minimum-neighbour</em> around each square. Here’s the code if you want to print the path too. This is certainly not optimized, but is simple to understand:</p><pre>def trace_path(output_cost_matrix, i, j):<br> path = [(i, j)]<br> while ((i&gt;0) | (j&gt;0)):<br>   neighbours = [output_cost_matrix[i-1, j-1], <br>                 output_cost_matrix[i-1, j], <br>                 output_cost_matrix[i, j-1]]<br>   ## see which neighbour is the smallest<br>   path_min = np.argmin(neighbours)</pre><pre>   ## store the position of the smallest neighbour<br>   if path_min==0:<br>      i=i-1<br>      j=j-1<br>   elif path_min==1:<br>      i=i-1<br>   else:<br>      j=j-1<br>   path.append((i, j))</pre><pre> return path[::-1] ## return after reversing the list</pre><p>Let’s see the path returned from blue to red square by calling:</p><pre>trace_path(output_cost_matrix, 3, 4)</pre><p>Output:</p><pre>[(0, 0), (0, 1), (1, 2), (2, 3), (3, 4)]</pre><p>Representing this graphically:</p><figure><img alt="Traced path" src="https://cdn-images-1.medium.com/max/393/1*gm7eMWs5d1u53Z_DTZK1Vw.gif" /><figcaption>Traced path — “Image by author”</figcaption></figure><p>Or tracing it over our original matrix:</p><figure><img alt="Cheapest path" src="https://cdn-images-1.medium.com/max/393/1*mQ910QZS3ZD7nJZCSah1vg.gif" /><figcaption>Cheapest path — “Image by author”</figcaption></figure><h3>Cool! But where’s DTW in all of this?</h3><p>Quite right! How does DTW picture into all of this?</p><blockquote>Well, DTW is nothing but simply matching one point in time-series pattern1 with the closest point in pattern2.</blockquote><p>Let’s look at the mapping of two signals again. The one in cyan is pattern1 and the one in orange is pattern2, with the red lines trying to find a corresponding point on pattern1 for every point on pattern2:</p><figure><img alt="Pattern mapping" src="https://cdn-images-1.medium.com/max/697/1*EU7BJnoXrjxAeuWszbhEBw.png" /><figcaption>Pattern mapping — “Image by author”</figcaption></figure><p>So essentially, we are finding the shortest or <em>cheapest</em> path between a pattern2-point and pattern1-point. But what is the cost between any two points. Do we put it randomly like we did in our example for each square?</p><p>You have various choices you can pick from here, as per your use-case, but the most common one is the <a href="https://en.wikipedia.org/wiki/Euclidean_distance">euclidean distance</a> between those points.</p><p>You arrange all pattern1-points as column axis of the matrix and all pattern2-points as row axis. Then fill each square with the euclidean distance:</p><figure><img alt="Patterns arraged as a matrix" src="https://cdn-images-1.medium.com/max/647/1*6us6pc7YPECym-oNJ4oKmA.png" /><figcaption>Patterns arraged as a matrix — “Image by author”</figcaption></figure><p>And then it’s the recursive algorithm we just went through. An output path like [(0,0), (0,1), (1,1), (2,2), (3,3), (4,3)] would mean that point 0 in pattern1 should be matched up with points 0 and 1 in pattern2. Points 1, 2, 3 and 4 in pattern1 should be matched with points 1, 2, 3 and 3 in pattern2.</p><p>But what does that actually achieve? What does such shortest-distance matching mean?</p><blockquote>I like to imagine it as a solution where you are trying to match pattern2 with pattern1 in a manner that ensures least stretching (or shrinking) of pattern2; greedily matching it to the closest neighbour.</blockquote><p>And there you have it. A simple DTW algorithm implemented from scratch.</p><p>I have not mentioned several nuances and variations of the DTW process, for example a <a href="https://en.wikipedia.org/wiki/Dynamic_time_warping#Implementation">windowed DTW</a> where we add a locality constraint. I hope this article has sparked your interest in knowing more about it. Do let me know if you discover something cool about it!</p><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a>, <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a> or through my website: <a href="http://iamjustastudent.com/about">I am Just a Student</a>.</p><p>See you around &amp; happy learning!</p><p><a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">Himanshu Chandra - Business Head - AI, ML &amp;amp; QA - Ideas To Impacts | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f660ccb77ff4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-intuitive-approach-to-dtw-dynamic-time-warping-f660ccb77ff4">An intuitive approach to DTW — Dynamic Time Warping</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Two Variations of Confusion Matrix — Get Confused, Never Again]]></title>
            <link>https://medium.com/data-science/the-two-variations-of-confusion-matrix-get-confused-never-again-8d4fb00df308?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d4fb00df308</guid>
            <category><![CDATA[specificity]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sensitivity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Fri, 12 Jun 2020 18:14:14 GMT</pubDate>
            <atom:updated>2020-06-12T18:27:05.529Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Two Variations of Confusion Matrix</h3><h4>Memory hacks to remember its two variations and associated formulae</h4><figure><img alt="Confusion Matrix" src="https://cdn-images-1.medium.com/max/1024/1*ev5EKJ4HftM9v7caJEet5w.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/HOerwin56-2108907/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=2667890">HOerwin56</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=2667890">Pixabay</a></figcaption></figure><p>You may already know that there are two schools of thought (<a href="https://en.wikipedia.org/wiki/Confusion_matrix">wiki</a> vs <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html?highlight=confusion">python</a>) on how to display a confusion matrix.</p><p>For every beginner, there comes a time when one sees the ‘other’ variation and tilts one’s head to make sense of it. The literature supports one layout while the code seems to spew out the other.</p><blockquote>To understand the basics of Confusion Matrix, you can have a look <a href="https://en.wikipedia.org/wiki/Confusion_matrix">here</a>.</blockquote><p>Here are the variations:</p><figure><img alt="Actual Class as Columns" src="https://cdn-images-1.medium.com/max/698/1*r4Sj_rRPPvtfJdA-iOkQpA.png" /><figcaption><strong>Perspective 1</strong>: <a href="https://en.wikipedia.org/wiki/Confusion_matrix">Actual (True) Class as Columns</a></figcaption></figure><p>as compared to:</p><figure><img alt="Perspective 2: Predicted Class as Columns" src="https://cdn-images-1.medium.com/max/568/1*ZNUeqoHg9S-2m3-oGYdpCQ.png" /><figcaption><strong>Perspective 2</strong>: <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html?highlight=confusion">Predicted Class as Columns</a></figcaption></figure><p>It’s not a case of simply interchanging the rows and columns, but also the fact that <strong>True Positives </strong>which were at top-left corner are now replaced by <strong>True Negatives</strong>.</p><blockquote>If you are using built-in scikit-learn functions like <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html">confusion_matrix</a>, you need to be aware of the input and output layout, else all your calculations of Sensitivity (Recall), Specificity, Precision, F1 score are going to be wrong.</blockquote><h3>Show me some proof!</h3><p>Okay, let’s have an actual example to figure this out, through code.</p><figure><img alt="Our dataset" src="https://cdn-images-1.medium.com/max/335/1*1afjQ4jtrn3N17P58RO3dw.png" /><figcaption>Our dataset</figcaption></figure><p>This is our dataset of medical results of 17 people. They are either unwell (Positive test result) or healthy (Negative test result). The first column shows their actual state while the second column is the one predicted by our algorithm. The code used to generate this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31e8e99fdfb5528299a159b4031b2423/href">https://medium.com/media/31e8e99fdfb5528299a159b4031b2423/href</a></iframe><h3>Perspective 1:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*Bic68Z2ZyXolnsVByW4CuA.png" /></figure><p>If like me, you have studied most of statistics via books and publications rich in examples from the medical field, the Actuals (or the <a href="https://commons.wikimedia.org/wiki/File:Preventive_Medicine_Statistics_Sensitivity_TPR,_Specificity_TNR,_PPV,_NPV,_FDR,_FOR,_ACCuracy,_Likelihood_Ratio,_Diagnostic_Odds_Ratio_2_Final.png"><em>Gold Standard</em></a>) will be on the top as columns and <strong>True Positive</strong> in the top-left cell. Our dataset will look like this:</p><figure><img alt="Perspective 1: Actuals on Top" src="https://cdn-images-1.medium.com/max/518/1*hJ5TVvE1TwrRA1v7qoAvcQ.png" /><figcaption>Perspective 1: Actuals on Top</figcaption></figure><p>Here’s the python code to generate this:</p><pre>cm = confusion_matrix(data.Predicted, data.Actual, labels=[‘<strong>Positive</strong>’, ‘Negative’])</pre><pre>print(cm)</pre><p>Note how we pass data.Predicted first and data.Actual later. Also the order of labels is Positive first.</p><p>Output:</p><figure><img alt="Confusion Matrix as Code Output" src="https://cdn-images-1.medium.com/max/126/1*wwk3wZPfRSKF_r66CDhgRg.png" /></figure><p>My memory hack to remember the order of arguments:</p><blockquote>I just read it the way one reads English — <strong><em>left to right</em></strong>. And I’m <strong><em>positive</em></strong> this is the layout one ought to use.</blockquote><figure><img alt="Left to Right" src="https://cdn-images-1.medium.com/max/465/1*0kwAB1uk5YuZiUo23daaMg.gif" /><figcaption>Left to Right</figcaption></figure><p>To calculate all the metrics, one can use sklearn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html"><em>classification_report</em></a> function.</p><blockquote>I love this function because the order of Actual and Predicted arguments remains the same, no matter what your perspective is. All you need to specify differently is the order of labels.</blockquote><pre>report = classification_report(data.Actual, data.Predicted, labels=[‘<strong>Positive</strong>’, ‘Negative’])</pre><pre>print(report)</pre><p>It’s always <em>Actual</em> first, <em>Predicted</em> later — alphabetically arranged, elegantly. The output it generates is:</p><figure><img alt="Classification Report" src="https://cdn-images-1.medium.com/max/757/1*Av7-fLDqHpcvYlCXFKaZ7A.png" /><figcaption>Classification Report</figcaption></figure><h3>Perspective 2:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*dxK0RIzWB-yJkvgXmsErPw.png" /></figure><p>Those who have learnt it through programming documentation and tutorials hate the above representation. It’s always the other way round!</p><figure><img alt="Predictions on top" src="https://cdn-images-1.medium.com/max/811/1*3JOsI64k-sCtjWy9DN1reg.png" /><figcaption>Predictions on top</figcaption></figure><blockquote>The position of <strong>False Positives</strong> and <strong>False Negatives</strong> remain the same in a 2x2 confusion matrix, whichever perspective you choose.</blockquote><p>The memory hack remains the same, read from left to right but this time give more importance to <em>Negatives</em> by mentioning them first in the labels argument.</p><pre>cm2 = confusion_matrix(data.Actual, data.Predicted, labels=[‘<strong>Negative</strong>’, ‘Positive’])</pre><pre>print(cm2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/116/1*9jFmkWwbqjtraSZKsGE3pg.png" /></figure><p>And the good old <em>classification_report</em> remains the same, the only change being in the order of labels:</p><pre>report2 = classification_report(data.Actual, data.Predicted, labels=[‘<strong>Negative</strong>’, ‘Positive’])</pre><pre>print(report2)</pre><figure><img alt="Classification Report — 2" src="https://cdn-images-1.medium.com/max/788/1*_A0K26HhxHklltYRMrXdpA.png" /><figcaption>Classification Report — 2</figcaption></figure><p>There you are, both the results perfectly in sync now!</p><h3>Additional Tip — Never Forget the Formulae:</h3><p>I often had to look up my notes to remember what <em>Precision</em> meant, or what was the formula for <em>Sensitivity</em> (also known as <em>Recall</em>). So, here’s a memory hack I use to remember them.</p><p>First, the actual formulae for them, so that you can relate easily. I explain here on the basis of <strong>Perspective 1</strong>, but you can change it very easily for perspective 2 if you wish to.</p><p>I like to arrange them in an alphabetical order first, so that I don’t mix them up — 1. <strong>P</strong>recision 2. <strong>R</strong>ecall 3. <strong>S</strong>pecificity</p><p>Precision = TP/(TP+FP)</p><p>Recall = TP/(TP+FN)</p><p>Specificity = TN/(TN+FP)</p><p>My memory hack to remember them:</p><figure><img alt="Formulae" src="https://cdn-images-1.medium.com/max/654/1*1PGx39qCZ2pN2KRftuEYzQ.gif" /></figure><p>Precision is the first cell divided by the row sum: 2/6 = 0.33</p><p>Recall is the first cell divided by the column sum: 2/7 = 0.29</p><p>Specificity is the last cell divided by the column sum: 6/10 = 0.60</p><p>That’s it! Made my life a little bit easier, hope it helps you too. Have you got any tips and tricks which work for you? Do let me know in the comments section or drop me a line separately.</p><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a>, <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a> or through my website: <a href="http://iamjustastudent.com/about">I am Just a Student</a>.</p><p>See you around &amp; happy learning!</p><p><a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">Himanshu Chandra - Business Head - AI, ML &amp;amp; QA - Ideas To Impacts | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d4fb00df308" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-two-variations-of-confusion-matrix-get-confused-never-again-8d4fb00df308">The Two Variations of Confusion Matrix — Get Confused, Never Again</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prepare Dinner & Save the Day by Calculating Confidence Interval of Non-Parametric Statistical…]]></title>
            <link>https://medium.com/data-science/prepare-dinner-save-the-day-by-calculating-confidence-interval-of-non-parametric-statistical-29d031d079d0?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/29d031d079d0</guid>
            <category><![CDATA[statistical-test]]></category>
            <category><![CDATA[confidence-interval]]></category>
            <category><![CDATA[wilcoxon-signed-rank]]></category>
            <category><![CDATA[mann-whitney]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Sat, 30 May 2020 13:59:19 GMT</pubDate>
            <atom:updated>2020-05-30T16:06:43.102Z</atom:updated>
            <content:encoded><![CDATA[<h3>Make Dinner &amp; Save the Day by Calculating Confidence Interval of Non-Parametric Statistical Tests</h3><h4>Complete python code with worked examples for <a href="https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test">Mann-Whitney U test</a> and <a href="https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test">Wilcoxon signed-rank test</a>.</h4><figure><img alt="A feather" src="https://cdn-images-1.medium.com/max/1024/0*YW6TDkKmpRtmYb9F" /><figcaption>Photo by <a href="https://unsplash.com/@daiga_ellaby?utm_source=medium&amp;utm_medium=referral">Daiga Ellaby</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>“I think I need to visit the doctor”, said sleepy-eyed Janhvi.</p><p>“Why?”, I asked, still half-asleep, enjoying the lazy morning vibes.</p><p>“Feeling quite tired since the last few days, and my appetite’s gone. I better get diagnosed”, she answered, while pulling my blanket away and half-kicking me out of bed. That was my cue to get ready and drive her to the doctor — there went my lovely morning!</p><p>“O! But you look stunning in that white dress. I’m sure you’ll be okay in a couple of hours” — buttering always helps, I thought. The expression in her eyes while looking at me could have given the <em>“dull”</em> emoji a run for its money, so I cut my losses and got up. Time to go and visit our trusted doctor.</p><h3>What Shalt Thou Choose — Love or Money?</h3><p>“Your blood plasma count has dipped a bit. Working under stress and not eating right, are you Janhvi?”, said “Doctor Uncle” scoldingly.</p><p>“I can write you the prescription for your usual medicines, should be fine in a couple of weeks. There is however, this new brand which seems to be giving better results in the patients I have tried them on, in the past 6 months. I think you should try it. Results are evident in a week.”</p><p>“Same price?”, I enquired, albeit a bit too eagerly, I must confess.</p><p>“I thought asking about any new side effects would have been the first question. But yeah, it costs a bit more. Which one then?”, Doctor uncle quipped dryly without shifting his eyes from Janhvi.</p><p>The costly one it was then. My mind raced for the next 10 minutes to come up with a lengthy excuse to save face. In the car, driving back home, the data scientist in me kicked in. “You know, statisticians have a horde of tests to compare two categories or treatments. They just don’t go about saying that one is better than the other based on their gut feeling. With so few patients, it’s difficult to generalize findings of difference between approaches.”</p><p>Janhvi: “Mmm…hmmm”</p><p>Me: “I imagine in this case, one could very well document the plasma count for every patient undergoing treatment under our doctor, divide it into two groups based on the treatment they underwent and then use statistical tests to check if there’s really any difference.”</p><p>Janhvi: “Mmm…hmmm”</p><p>Me: “It’s your blood plasma count we are measuring, isn’t it? I think that does not follow a normal distribution. A Mann-Whitney <em>U</em> test should give us the answer.”</p><p>Janhvi: “Say what?! My blood plasma is not normal? Is that what you are saying?”</p><p>Me: “No, no! I said it does not follow a normal <em>distribution</em>. See that far-away mountain? Imagine that, just in 2D. A peak in the middle with gradual tapering to the left and right. Most of the things we measure in our universe tend to follow a <a href="https://www.mathsisfun.com/data/standard-normal-distribution.html">normal distribution </a>— like height, blood pressure, test scores or IQ. Without going into the Maths, if I were to count all the people, or at least many of them, on this planet in the range of say, 6 inches to 20 foot, and jot it down in the following manner in a table— count of shortest on the left-most side → count of 1 inch taller to its right → count of 1 inch taller to its right → so on till count of tallest as the right-most entry, then I will see an intersting pattern, observed in so many things we experience around us.”</p><figure><img alt="Height &amp; Count" src="https://cdn-images-1.medium.com/max/424/1*GYciih6OtUo6NvnJ6gPIgA.png" /><figcaption>Height &amp; Count</figcaption></figure><p>Janhvi: “Which is?” Her interest grew.</p><p>Me: “A mountain, a <em>Bell curve</em>. If I go to our garden and start stacking bricks to create a new column for each entry I made in my table (each new column erected just to the right of the previous one), with the height of the brick column equal to the count I entered in my table, then you will have a mountain-shaped structure.”</p><figure><img alt="Normal distribution" src="https://cdn-images-1.medium.com/max/514/1*_qnHrw8fEDS7VOJzJxJY2Q.png" /><figcaption>Normal distribution</figcaption></figure><p>Janhvi: “Interesting. Don’t do that in our garden.”</p><p>Me: “Ok.”</p><blockquote>“I think that’s one of the reasons why we are able to empathize with others in this world. Most of us go through the same things, at some point or another.</blockquote><blockquote>We share a lot.”</blockquote><blockquote>She said softly while looking away into the horizon.</blockquote><p>I smiled. I had never thought of it that way. I guess she was right.</p><p>“And plasma counts don’t follow this normal distribution?”. She came back to this world as quickly as she had floated away from it.</p><p>Me: “No. Not all things do, and plasma counts across people do not, I think. But we can check that later with the doctor.”</p><p>Janhvi: “So why does that matter?”</p><p>Me: “To put it simply, the type of statistical test to be used depends on the type of data you deal with. In our case, we compare 2 categories of treatments based on the output data, plasma count which is continuous data but non-normal. <a href="https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test">Mann-Whitney <em>U</em> test</a> is used for such cases.”</p><p>Janhvi: “And you will explain this to the doctor, ask him to do this comparison test and will then be able to tell for sure if the new treatment’s better than the old one?”</p><p>Me: “Well…… I will be able to say it with some level of confidence. I would be able to make a statement which sounds something like — We can be 95% confident that the difference in the average plasma counts between the treatments lies in the range between <em>x</em> and <em>y</em>. Since this range does not look too big, there’s a good chance both the treatments have pretty much the same effect on patients. Or alternatively, that they are likely to be different.”</p><p>“So much work for <strong><em>that</em></strong> statement!”, she exclaimed.</p><p>“Anything for you dear! I would never let you take a new medicine without being reasonably confident about it myself”, I said, hoping to convey a glint of pride and care in my voice. “You may not realize it right now, but such tests are critical to how studies are conducted in the pharmaceutical or financial sectors, among others.”</p><p>Janhvi: “Yeah, right. And I suppose just by luck you had a chance to study about these tests in the past week and have found them quite interesting?”</p><p>Me: “Yes! Finding out the confidence intervals for some of these tests, called non-parametric tests, is not easy and no standard programming packages exist, at least not in python. So I did some digging around and created custom functions for those. I plan to write an article on it soon. How did you know?”</p><p>Janhvi: “I know you too well my dearest. We’re almost home. Tell you what, I feel like I need to sleep till late evening and I’ll be famished when I get up. How about you finish off your article in peace in the next few hours and then cook me a wonderful dinner, so that I totally forget about the ‘<em>price</em>’ question you asked Doctor uncle?”</p><p>“Sounds fair”, I said, grinning sheepishly.</p><h3>Getting Our Hands Dirty with Code</h3><p>So here it is, the step-by-step code.</p><p>The methodology behind the code has been referenced from the British Medical Journal, volume 296, 21 May 1988: “<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2545906/pdf/bmj00286-0037.pdf">Calculating confidence intervals for some non-parametric analyses</a>” by M J Campbell &amp; M J Gardner.</p><blockquote>You can download my complete code either from <a href="https://colab.research.google.com/drive/1E_l4-Z6132DGZoVRqpecn6I3Ny62Nksj">Colab</a> or <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/Non-parametric%20Confidence%20Interval">GitHub</a>.</blockquote><h4>For Unpaired Samples</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac338c5e0633cfa06c2d454ccc5cea26/href">https://medium.com/media/ac338c5e0633cfa06c2d454ccc5cea26/href</a></iframe><p>Let’s go through the example listed in the paper itself, so that we can compare results:</p><figure><img alt="Unpaired dataset" src="https://cdn-images-1.medium.com/max/846/1*-wrpLdkZSf8ROxlEFaImEA.png" /><figcaption>Unpaired dataset</figcaption></figure><p>n =Group 1 or Sample 1 size = 10</p><p>m =Group 2 or Sample 2 size = 10</p><p>Sort both the Samples’ values and out them as column headers and row indices respectively. We need to calculate the difference of each observation in Sample 1 with every observation in Sample 2, giving us <strong><em>n</em></strong><em> </em>x<em> </em><strong><em>m</em></strong> differences. Here’s how it looks:</p><figure><img alt="n x m differences" src="https://cdn-images-1.medium.com/max/783/1*ubXac678i-3v1PZlMCBFDg.png" /><figcaption>n x m differences</figcaption></figure><p>In our python function, <strong><em>non_param_unpaired_CI( )</em></strong>, this is achieved by the following line of code:</p><figure><img alt="n x m differences in python" src="https://cdn-images-1.medium.com/max/912/1*qn0VEy2YxbhdlH2Kyh4iVA.png" /><figcaption>n x m differences in python</figcaption></figure><p>The estimate of the difference in population medians or means is now given by the median of these differences. From the 100 differences in the table the 50<em>th</em> smallest difference is -6 g/l and the 51<em>st</em> is -5 g/l so the median difference is estimated as (-6+(-5))/2= -5.5 g/l. This can be calculated in python by :</p><figure><img alt="median of the differences" src="https://cdn-images-1.medium.com/max/1024/1*4TLPLw5sFbSLf4MBG6Mhmg.png" /><figcaption>median of the differences</figcaption></figure><p>We need the K<em>th</em> smallest and K<em>th </em>largest median difference value to find the interval range. K is given as:</p><figure><img alt="K for unpaired samples" src="https://cdn-images-1.medium.com/max/916/1*oHi95QAmzANxbk9MRhEsMg.png" /><figcaption>K for unpaired samples</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*G4n9ql6ScrEOHze6iAVpJw.png" /></figure><p>Here, this term is simply the <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html">percentile</a> at a given confidence level. If our confidence level is 0.95, then alpha would be (1–0.95) = 0.05. We can use this in python to calculate <strong>N</strong> as :</p><figure><img alt="N calculated" src="https://cdn-images-1.medium.com/max/806/1*vL4Shm5KvyLNqyI60-c5Dg.png" /><figcaption>N calculated</figcaption></figure><p>and hence <strong>K</strong> as :</p><figure><img alt="K calculated" src="https://cdn-images-1.medium.com/max/907/1*-haBrt9UF0rYeNwCoSaDgw.png" /><figcaption>K calculated</figcaption></figure><p>This comes out to be <strong>24</strong>. The 24<em>th</em> smallest difference is -10 g/l and the 24<em>th</em> largest is + 1 g/l. Hence the 95% confidence interval for the difference in population medians is from <strong>-10 g/l to +1 g/l</strong>.</p><h4>For Paired Samples</h4><p>This can be used in case of <a href="https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test">Wilcoxon signed-rank test</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/074305398b3fce46dfa5c634609e59ab/href">https://medium.com/media/074305398b3fce46dfa5c634609e59ab/href</a></iframe><figure><img alt="Worked example — paired samples" src="https://cdn-images-1.medium.com/max/784/1*naxpZyiafzo_7RDZvE1ZCg.png" /><figcaption>Worked example — paired samples</figcaption></figure><p>Notice that the very first step here is to calculate the differences in observations of each sample (<em>After-before</em>).</p><p>n = size of each Sample = 11</p><p>Let’s now calculate our <strong>N</strong> and <strong>K</strong>, with confidence level 0.95 and alpha = (1–0.95) = 0.05 :</p><figure><img alt="N calculated" src="https://cdn-images-1.medium.com/max/868/1*5MopOLbFf_WYDDOYz3JbtA.png" /><figcaption>N calculated</figcaption></figure><figure><img alt="K formula" src="https://cdn-images-1.medium.com/max/962/1*Qyhfl67X2Cxxg_rk47TX_Q.png" /><figcaption>K formula</figcaption></figure><figure><img alt="K calculated" src="https://cdn-images-1.medium.com/max/905/1*Mefhd74nvRVvO8iwTE8LTA.png" /><figcaption>K calculated</figcaption></figure><p>K comes out to be <strong>11</strong>.</p><p>We’ll now create a <strong><em>n</em></strong> x <strong><em>n</em></strong> table with all the average of differences. Since the column headers and row indices will be the same, we only need the n(n+1)/2 values :</p><figure><img alt="Average of differences" src="https://cdn-images-1.medium.com/max/878/1*uDo7o51VwYMf2LOpB3Eo_w.png" /><figcaption>Average of differences</figcaption></figure><p>In python :</p><figure><img alt="Average of differences in python" src="https://cdn-images-1.medium.com/max/1024/1*6VO44EOEp21fnPlY4Memlg.png" /><figcaption>Average of differences in python</figcaption></figure><p>The 11th smallest and the 11th largest averages are <strong>11.9</strong> and <strong>25.1</strong>, which are therefore, our intervals.</p><figure><img alt="Dinner" src="https://cdn-images-1.medium.com/max/1024/0*toDQe5TB4eEd6rAp" /><figcaption>Photo by <a href="https://unsplash.com/@shotbylana?utm_source=medium&amp;utm_medium=referral">Lana Abie</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Phew! Just in time! Hope this helps someone out there. Do let me know if it does. Now off I go, got to make a spectacular dinner :)</p><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a>, <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a> or through my website: <a href="http://iamjustastudent.com/about">I am Just a Student</a>.</p><p>See you around &amp; happy learning!</p><p><a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">Himanshu Chandra - Business Head - AI, ML &amp;amp; QA - Ideas To Impacts | LinkedIn</a></p><p><em>Code download links</em>: <a href="https://colab.research.google.com/drive/1E_l4-Z6132DGZoVRqpecn6I3Ny62Nksj">Colab</a>, <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/Non-parametric%20Confidence%20Interval">GitHub</a>.</p><p><em>Credits &amp; references</em>:</p><p><a href="https://www.amazon.in/Statistics-Square-One-Michael-Campbell/dp/1405191007">“Statistics at Square One” by M J Campbell &amp; T D V Swinscow</a></p><p><a href="https://machinelearningmastery.com/nonparametric-statistical-significance-tests-in-python/">Jason Brownlee’s blog — Machine Learning Mastery</a></p><p><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2545906/pdf/bmj00286-0037.pdf">“Calculating confidence intervals for some non-parametric analyses” by M J Campbell &amp; M J Gardner</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29d031d079d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/prepare-dinner-save-the-day-by-calculating-confidence-interval-of-non-parametric-statistical-29d031d079d0">Prepare Dinner &amp; Save the Day by Calculating Confidence Interval of Non-Parametric Statistical…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A common man’s guide to MAE and RMSE]]></title>
            <link>https://medium.com/data-science/a-common-mans-guide-to-mae-and-rmse-d5efcd238221?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5efcd238221</guid>
            <category><![CDATA[mape]]></category>
            <category><![CDATA[aem]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rmse]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Sat, 16 May 2020 17:02:37 GMT</pubDate>
            <atom:updated>2020-05-16T17:21:35.788Z</atom:updated>
            <content:encoded><![CDATA[<p>Understand in simple words — What do they mean in the real world and when to choose which?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XuGNkiKIWCNKa3Kkk42ADQ.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/geralt-9301/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=1594742">Gerd Altmann</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=1594742">Pixabay</a></figcaption></figure><p>A businessman, a prospective client of mine, asked me yesterday when I showed him my forecast models, “How accurate do you think these will turn out to be?”. I was ready for the question. “<strong>Very</strong>”, I said with quite some confidence. “It has an extremely low MAE”.</p><p>Hiding his smile at my naive answer, and looking at me the way an elderly looks at a teenager yet to understand the ways of the world, he asked again. “I am not very good at understanding advance financial and mathematical terms, could you explain it to me in some simple way, so that I understand how much risk am I taking here, believing your forecasts?”.</p><blockquote>Common sense isn’t that common and simplicity isn’t that simple to attain.</blockquote><p>So I politely asked for another day’s time and took his leave.</p><h3>Mistakes maketh the man</h3><p>We understand very clearly in our lives that the same magnitude of mistake carries different weights in different situations. Being late by 10 minutes might not be bad when you are attending a casual get-together dinner with friends, but is a life-altering catastrophic event when it is being hosted by your wife’s parents.</p><p>You juuust might be able to get away by saying that you are quite punctual, most of the times. You are basically asking to be considered on your average performance.</p><blockquote>That pretty much, is Average Error or <strong>Mean Error</strong> for you.</blockquote><p>We should think this in the context of my client. He is the owner of a family-run restaurant — not too big, started a few months back. His ask was to predict the daily customer footfall so that he is able to plan for correct amount of raw food materials to be bought from the wholesale market. Being a new business, he is equally afraid of incurring losses due to excessive food being wasted at the end of the day, as well as having to ask customers to go back because he has nothing more to cook for the day.</p><p>Let’s visualize this on a weekly scale. Suppose these are the actual number of customers per day:</p><figure><img alt="Daily Customer Footfall" src="https://cdn-images-1.medium.com/max/333/1*BoNgM3ap-obGsDUSp8KouQ.png" /><figcaption>Daily Customer Footfall</figcaption></figure><p>I create the models, and forecast something like this:</p><figure><img alt="Daily Predictions" src="https://cdn-images-1.medium.com/max/487/1*018aa1Th3SxJVroukWYjvQ.png" /><figcaption>Daily Predictions</figcaption></figure><p>I then calculate the error or mistakes committed by the model:</p><figure><img alt="Prediction Errors" src="https://cdn-images-1.medium.com/max/636/1*6D30OhKdgpjpbypl1ZMxzA.png" /><figcaption>Prediction Errors</figcaption></figure><p>Total Error = 0. It’s very easy to see the fallacy in this logic. My mistakes in positive direction get cancelled by those in the negative direction.</p><p>I say, a mistake is a mistake. Mathematically, that means taking the absolute value of the mistake.</p><figure><img alt="Mean Absolute Error (MAE)" src="https://cdn-images-1.medium.com/max/643/1*UMU2kH9Bw8Fj5IrdcVXh-Q.png" /><figcaption>Mean Absolute Error (MAE)</figcaption></figure><blockquote>We just understood what <strong>MAE </strong>(Mean Absolute Error) is.</blockquote><p>It says that on an average, I will mis-predict a day’s customer count by 6.285. Whether I will predict more or less, is not obvious by knowing the MAE. My client might be wasting food of 6.285 people each day or turning away 6.285 customers each day — on an average.</p><p><em>For the sake of understanding, let’s assume that my model makes mistakes only in the positive direction, that is, I might make a mistake of saying, “There would be 25 customers today”, when there actually may be only 20. But never will there be a case where I under-predict the number to be only 15. This basically means food might get wasted, but all customers are entertained.</em></p><h3>One’s a Thief, Another’s a Robber</h3><p>So does that mean two approaches (models) having same MAE are equally good (or bad)?</p><p>Imagine a competitor of mine, whom I will refer to as BloodyC Pvt. Ltd., is also trying to sell its approach to this gentleman who owns the restaurant. Our predictions look something like this:</p><figure><img alt="Two predictions, same MAE" src="https://cdn-images-1.medium.com/max/983/1*i6Vcx113RPfG5AaW1w2K-Q.png" /><figcaption>Two predictions, same MAE</figcaption></figure><p>I make small mistakes every day, while BloodyC is accurate on most days but commits bigger mistakes on some days. Which is better?</p><p>The answer can not be derived by looking at numbers alone. It’s about the business, its vision, its morals and the people behind it. So I will ask you to imagine a situation, not far removed from reality, where my model comes out better.</p><blockquote>(Data) Science is never about numbers and logic. It’s always about emotions and lives.</blockquote><p>In many countries, shops and homes, when there’s excess food left, we try to give it to those who need it more than others, or feed pets &amp; stray animals if it’s not harmful to them. Everyone has limits, of course. So if there’s food left only for 2–3 people it’s easier to immediately find someone needy but a bulk is often thrown out into the garbage bin if the quantity’s more. Not every shop owner connects to a homeless shelter or similar service.</p><p>My prospective client, the restaurant owner, behaves similarly. It’s also an emotional pain for many to see so much food go to waste. Although it’s the same financially to him, he would much rather prefer my model, which wastes little each day as compared to BloodyC’s which on a couple of days, wastes a lot.</p><p>Find the situation too abstract or hypothetical? Here’s a very real one:</p><h3>$crooge McDuck</h3><p>Professional traders often prefer investing/trading in the stock market via derivatives — futures &amp; options. In many strategies, bets are made on the price range within which a stock or derivative will stay for the next <em>n</em> number of days.</p><p>Does my model now feel a safer bet than BloodyC’s? Even when I go wrong, I go wrong by a small margin. A trader will still make money in that limited range, even if his bet was not accurate. If the price had swayed away a lot from the prediction on which the trader had trusted, it means losing a huge amount of money in <strong>one</strong> bad investment — trusting BloodyC would have cost him dearly.</p><p>If this makes you feel that my model is much more suitable for your business case, then which metric do you use to express this while comparing the two models? MAE certainly is unable to penalize one more than the other.</p><h3>Enter, RMSE</h3><p>RMSE is square <strong>R</strong>oot of <strong>M</strong>ean <strong>S</strong>quared <strong>E</strong>rror. So if you square each mistake made in the prediction, and add them up, then divide by 7 (total number of predictions made), you get MSE. If you want RMSE, just do an additional square root. (Phew, wasn’t that a mouthful!)</p><p>Let’s see how RMSE looks for our predictions:</p><figure><img alt="RMSE" src="https://cdn-images-1.medium.com/max/1024/1*oDleW-pT-jtsl_wH3JyfBA.png" /><figcaption>RMSE</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*27sdEr2TlDJXzU5P4RiVsA.gif" /><figcaption>Yoohoooo!</figcaption></figure><p>Bigger mistakes, bigger penalty — that’s one of the features of RMSE. My prediction got a lower RMSE (<em>2.138</em>) value than BloodyC’s (<em>3.505</em>), which means it is better when evaluated with this objective.</p><h3>Pray to RMSE!</h3><p>Does this mean we should always use RMSE to check a model’s accuracy?Absolutely not!</p><p>If instead of rice, chicken and Indian curry, my prospective client sold pickles which comfortably last for years, allowing one day’s excess to easily be sold the next day, then he would more likely be benefited by focusing on MAE or any other similar metric.</p><p>There are several other nuances of RMSE, MAE, MAPE (Mean Absolute Percentage Error) and one should certainly consider reading about those and diving deeper into the maths behind it. Few articles I found interesting were <a href="https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d">this</a> and <a href="https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/">this</a> (jump to <em>Section 11: Accuracy Metrics for Time Series Forecast</em>), for those who wish to explore further.</p><h3>Parting Thoughts</h3><p>I hope my prospective client would now be able to understand how much, and what type of risk he is taking by trusting my forecasts. His questions certainly made me revisit what fits his use case better.</p><p>As for the not-so-punctual husband, he can only hope not to be judged on RMSE when he goofs-up big in the near future!</p><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a>, <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a> or through my website: <a href="http://iamjustastudent.com/about">I am Just a Student</a>.</p><p>See you around &amp; happy learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5efcd238221" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-common-mans-guide-to-mae-and-rmse-d5efcd238221">A common man’s guide to MAE and RMSE</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pipelines & Custom Transformers in scikit-learn: The step-by-step guide (with Python code)]]></title>
            <link>https://medium.com/data-science/pipelines-custom-transformers-in-scikit-learn-the-step-by-step-guide-with-python-code-4a7d9b068156?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a7d9b068156</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Wed, 06 May 2020 19:26:25 GMT</pubDate>
            <atom:updated>2020-06-27T19:16:58.660Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand the basics and workings of scikit-learn pipelines from the ground up, so that you can build your own.</h4><p>This article will cover:</p><ol><li>Why another tutorial on Pipelines?</li><li>Creating a Custom Transformer from scratch, to include in the Pipeline.</li><li>Modifying and parameterizing Transformers.</li><li>Custom target transformation via <em>TransformedTargetRegressor</em>.</li><li>Chaining everything together in a single Pipeline.</li><li>Link to <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/PipelineFoundation">download</a> the complete code from GitHub.</li></ol><blockquote>There’s a <a href="https://youtu.be/mOYJCR0IDk8">video walkthrough</a> of the code at the end for those who prefer the format. I personally like written tutorials, but I’ve had requests for video versions too in the past, so there it is.</blockquote><figure><img alt="Pipeline pic" src="https://cdn-images-1.medium.com/max/1024/1*Cl2jZ1tgwrdgWFjtRXFcDA.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/Robzor-840419/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=681175">Robson Machado</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;amp;utm_medium=referral&amp;amp;utm_campaign=image&amp;amp;utm_content=681175">Pixabay</a></figcaption></figure><h3><strong>Why another tutorial on Pipelines?</strong></h3><p>Since you are here, there’s a very good chance you already know Pipelines make your life easy by pre-processing the data. I heard that too and tried to implement one in my code.</p><blockquote>A shout-out to the few great tutorials I could find on the topic! I recommend you certainly browse through them, before or after the current article :</blockquote><p>i. <a href="https://towardsdatascience.com/custom-transformers-and-ml-data-pipelines-with-python-20ea2a7adb65"><em>https://towardsdatascience.com/custom-transformers-and-ml-data-pipelines-with-python-20ea2a7adb65</em></a><br>ii. <a href="https://machinelearningmastery.com/how-to-transform-target-variables-for-regression-with-scikit-learn/"><em>https://machinelearningmastery.com/how-to-transform-target-variables-for-regression-with-scikit-learn</em></a><br>iii. <a href="http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html"><em>http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html</em></a></p><p>It was all good while following the tutorials and using standard imputing, scaling, power-transforms, etc. But then I wanted to write specific logic to be applied to the data and wasn’t very sure what was being called where?</p><p>I tried to look for a lucid explanation on when are the constructor, fit(), transform() functions, actually being called, but couldn’t get a simple example. So I decided to step through the code bit by bit and present my understanding for anyone who wants to understand this from scratch.</p><p>Let’s get started then!</p><h3>Creating a Custom Transformer from scratch, to include in the Pipeline</h3><figure><img alt="Create dataframe" src="https://cdn-images-1.medium.com/max/725/1*8knrEH1zFaVvhtw1MA64pg.png" /><figcaption>Create DataFrame</figcaption></figure><p>To understand the examples better, we’ll create a dataset that will help us explore the code better.</p><p>The code above creates data which follows the equation y = X1 + 2 * sqrt(X2). This makes sure a simple Linear Regression model is not able to fit it perfectly.</p><p>Let’s see what prediction results are thrown at us:</p><figure><img alt="LinearRegression predictions on raw data" src="https://cdn-images-1.medium.com/max/694/1*P05EEG-koQ2QEJ1_MfB_ug.png" /><figcaption>LinearRegression predictions on raw data</figcaption></figure><p>A perfect prediction would be 14 and 17. The predictions are not bad, but can we do some calculations on the input features to make this better?</p><figure><img alt="Predictions after input feature manipulation" src="https://cdn-images-1.medium.com/max/724/1*M2s09ZVCUalfzwLSxthiYg.png" /><figcaption>Predictions after input feature manipulation</figcaption></figure><p>The input manipulations cause it to fit a perfect linear trend (y=X1+X2 now), and hence the perfect predictions. Now, this is just an example, but suppose for a dataset, your analysis said such input transformation would be good, how do you do that in a safe manner via Pipelines.</p><p>Let’s see a basic LinearRegression() model fitted by using a Pipeline.</p><figure><img alt="LinearRegression() with Pipeline" src="https://cdn-images-1.medium.com/max/899/1*UabbjB0Kf-7Z6T_mAzDnug.png" /><figcaption>LinearRegression() with Pipeline</figcaption></figure><p>As expected, we get the same predictions as our first attempt. The syntax at this point of time is quite simple —</p><ol><li>We declare a pipe1 variable using Pipeline class with array of steps inside it. The name of the step (in this case linear_model) could be anything unique of your choice. It is followed by an actual Transformer or Estimator (in this case, our LinearRegression() model).</li><li>Like any other model, it is fitted on the training data, but using the pipe1 variable.</li><li>Use pipe1 to predict on test set as you would do in any other model.</li></ol><p>To perform the input calculations/transformations, we’ll design a custom transformer.</p><figure><img alt="Custom Input Transformer" src="https://cdn-images-1.medium.com/max/752/1*jOT9Z8uZue8Ociwd1ejfrg.png" /><figcaption>Custom Input Transformer</figcaption></figure><p>We create a class and name it ExperimentalTransformer. All transformers we design will inherit from BaseEstimator and TransformerMixin classes as they give us pre-existing methods for free. You can read more about them in the article links I provided above.</p><p>There are 3 methods to take care of here:</p><ol><li>__init__ : This is the constructor. Called when pipeline is initialized.</li><li>fit() : Called when we fit the pipeline.</li><li>transform() : Called when we use fit or transform on the pipeline.</li></ol><p>For the moment, let’s just put print() messages in __init__ &amp; fit(), and write our calculations in transform(). As you see above, we return the modified values there. All the input features will be passed into X when fit() or transform() is called.</p><p>Let’s put this into a pipeline to see the order in which these functions are called.</p><figure><img alt="ExperimentalTransformer in Pipeline" src="https://cdn-images-1.medium.com/max/753/1*ezBGILQtOVC6DNZoECztkQ.png" /><figcaption>ExperimentalTransformer in Pipeline</figcaption></figure><p>You can see in the code comments above, one can also use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.make_pipeline.html">make_pipeline()</a> syntax, which is shorter, to create pipelines.</p><p>Now the output:</p><figure><img alt="Output with ExperimentalTransformer" src="https://cdn-images-1.medium.com/max/704/1*y-0kRdxTQkJ1IfuxWuybUQ.png" /><figcaption>Output with ExperimentalTransformer</figcaption></figure><p>3 important things to note:</p><p>a. __init__ was called the moment we initialized the pipe2 variable.</p><p>b. <strong>Both</strong> fit() and transform() of our ExperimentalTransformer were called when we fitted the pipeline on training data. This makes sense as that is how model fitting works. You would need to transform input features while trying to predict train_y.</p><p>c. transform() is called, as expected, when we call predict(test_X) — the input test features need to be square-rooted and doubled too before making predictions.</p><p>The result — perfect predictions!</p><h3>Modifying and parameterizing Transformers</h3><p>But..</p><p>We’ve assumed in the transform() function of our ExperimentalTransformer that the column name is X2. Let’s not do so and pass the column name via the constructor, __init__().</p><p>Here’s our ExperimentalTransformer_2:</p><figure><img alt="Passing arguments to the constructor" src="https://cdn-images-1.medium.com/max/711/1*Hk05A38Nv_E5jDP3bjRfSQ.png" /><figcaption>Passing arguments to the constructor</figcaption></figure><p>Take care to keep the parameter name exactly the same in the function argument as well as the class’ variable (feature_name or whichever name you choose). Changing that will cause problems later, when we also try to transform the target feature (y). It causes a double-call to __init__ for some reason.</p><p>I also added an additional_param with a default value, just to mix things up. It’s not really needed for anything in our case, and acts as an optional argument.</p><p>Create the new pipeline now:</p><figure><img alt="Calling with new Pipeline" src="https://cdn-images-1.medium.com/max/769/1*Nfx5VWTqU130x00kJvtkwg.png" /><figcaption>Calling with new Pipeline</figcaption></figure><p>Output is as expected:</p><figure><img alt="New output, same as before" src="https://cdn-images-1.medium.com/max/765/1*mDiG777k3ejzBao0q_xfSw.png" /><figcaption>New output, same as before</figcaption></figure><h3>Custom target transformation via TransformedTargetRegressor</h3><p>What about a situation when some pre and post processing needs to be done?</p><p>Consider a slightly modified data set:</p><figure><img alt="y squared in dataset" src="https://cdn-images-1.medium.com/max/712/1*f28krk9Kfp_PaiG5n_df4g.png" /><figcaption>y squared in dataset</figcaption></figure><p>Everything’s the same, but now y has been squared. To make this fit into a simple linear model, we will need to square-root y before fitting our model and also later, square any predictions made by the model.</p><p>We can use scikit-learn’s TransformedTargetRegressor to instruct our pipeline to perform some calculation and inverse-calculation on the target variable. Let’s first write those two functions:</p><figure><img alt="Transform and inverse-transform functions" src="https://cdn-images-1.medium.com/max/710/1*tCiPr9HUEo_snJp00Wb-cg.png" /><figcaption>Transform and inverse-transform functions</figcaption></figure><p>One square-roots y and the other squares it back.</p><p>Calling via pipeline:</p><figure><img alt="TransformedTargetRegressor call" src="https://cdn-images-1.medium.com/max/768/1*JhmhLlY2cHfyyXMiFBL2Tg.png" /><figcaption>TransformedTargetRegressor call</figcaption></figure><p>The TransformedTargetRegressor class takes regressor, func and inverse_func arguments which connects our pipeline to these new functions.</p><p>Note how we fit the model now, not the pipeline.</p><p>The output shows up something interesting and unexpected though:</p><figure><img alt="First output with TargetRegressor" src="https://cdn-images-1.medium.com/max/760/1*iZ78j7Qe3GEFJwNGcdsYhQ.png" /><figcaption>First output with TargetRegressor</figcaption></figure><p>The results are fine, but can you see how our target_transform() and inverse_target_transform() methods have been called multiple times when fit() was called? That is going to become an overhead in big projects and complex pipelines. The change needed to handle this is simply to set check_inverse param of TransformedTargetRegressor to False. We’ll do that in the next step along with looking at another way to handle target transformation — by using transformer param inside TransformedTargetRegressor instead of func and inverse_func.</p><p>We can pass an in-built transformer or our custom transformer instead of the two functions we designed. The custom transformer will look almost identical to the one we designed earlier for our pipeline, but will have an additional inverse_transform function inside it. Here’s the implementation:</p><figure><img alt="CustomTargetTransformer" src="https://cdn-images-1.medium.com/max/717/1*Rs6zdQkYagtKQrZC6c_YtQ.png" /><figcaption>CustomTargetTransformer</figcaption></figure><p>That’s it, just use it in our TransformedTargetRegressor call now:</p><figure><img alt="Call with transformer param" src="https://cdn-images-1.medium.com/max/775/1*-IhQDrIiHTeaSWLaEFFZ4A.png" /><figcaption>Call with transformer param</figcaption></figure><p>The output looks fixed now:</p><figure><img alt="Output with no repeated calls" src="https://cdn-images-1.medium.com/max/755/1*d1e1uiO5jJROnqc7cjqmRg.png" /><figcaption>Output with no repeated calls</figcaption></figure><p>One last thing to do here. We’ll make use of <a href="https://scikit-learn.org/stable/modules/compose.html?highlight=transformedtargetregressor#pipeline-chaining-estimators">caching</a> to preserve computations and also see how to get or set parameters of our pipeline from outside (this would be needed later if you want to apply GridSearch on top of this).</p><figure><img alt="get_params()" src="https://cdn-images-1.medium.com/max/846/1*dlrPaRLK2sdqoTDLjbXIPg.png" /><figcaption>get_params()</figcaption></figure><p>Notice how each parameter of each component of the pipeline can be accessed by using it’s name followed by a double underscore __.</p><p>We’ll tie it all together and even try to set a parameter from outside — the column name X2 we have been passing to the constructor.</p><figure><img alt="Tying it all together" src="https://cdn-images-1.medium.com/max/674/1*7sVjs_VekJlp98D0snfoyQ.png" /><figcaption>Tying it all together</figcaption></figure><blockquote>Complete Code: <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/PipelineFoundation">https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/PipelineFoundation</a></blockquote><blockquote>Code walkthrough: <a href="https://youtu.be/mOYJCR0IDk8">https://youtu.be/mOYJCR0IDk8</a></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmOYJCR0IDk8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmOYJCR0IDk8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmOYJCR0IDk8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd85921b5c239f364776cf2f9db74f94/href">https://medium.com/media/fd85921b5c239f364776cf2f9db74f94/href</a></iframe><h3>What Next?</h3><p>Any practical pipeline implementation would rarely be complete without using either a <strong>FeatureUnion</strong> or a <strong>ColumnTransformer</strong>. The very first reference link I provided above walks you through FeatureUnions. References I found extremely helpful for ColumnTransformers vs FeatureUnions are:</p><p>i. <a href="https://scikit-learn.org/stable/modules/compose.html#featureunion-composite-feature-spaces"><em>https://scikit-learn.org/stable/modules/compose.html#featureunion-composite-feature-spaces</em></a> <br>ii. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer"><em>https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer</em></a></p><p>Also, you will eventually use GridSearch on your model. Using it with pipelines is explained here: <a href="https://scikit-learn.org/stable/auto_examples/compose/plot_feature_union.html?highlight=pipeline"><em>https://scikit-learn.org/stable/auto_examples/compose/plot_feature_union.html?highlight=pipeline</em></a></p><p>Using these concepts should be easy enough, now that you have a good grasp of the foundations of pipeline creation.</p><p>Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me on <a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">LinkedIn</a>, <a href="https://www.youtube.com/channel/UChNybQwpqrX9SOc1D4K_x5Q">YouTube</a>, <a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent">GitHub</a> or through my website: <a href="http://iamjustastudent.com/about">I am Just a Student</a>.</p><p>See you around &amp; happy learning!</p><p><a href="https://www.linkedin.com/in/himanshu-chandra-33512811/">Himanshu Chandra - Business Head - AI, ML &amp;amp; QA - Ideas To Impacts | LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a7d9b068156" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pipelines-custom-transformers-in-scikit-learn-the-step-by-step-guide-with-python-code-4a7d9b068156">Pipelines &amp; Custom Transformers in scikit-learn: The step-by-step guide (with Python code)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine tune SARIMA hyperparams using Parallel processing with joblib (Step by Step Python code)]]></title>
            <link>https://medium.com/analytics-vidhya/fine-tune-sarima-hyperparams-using-parallel-processing-with-joblib-step-by-step-python-code-2037fec1659?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/2037fec1659</guid>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[joblib]]></category>
            <category><![CDATA[sarima]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Sun, 19 Apr 2020 17:22:50 GMT</pubDate>
            <atom:updated>2020-05-08T10:22:19.580Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>While working with most machine learning or statistical models, there comes a time when you need to fine tune the various parameters of your algorithm to get better accuracy in results. With so many combinations to test, it’s a time taking process. Using parallel processing and multiple threads drastically reduces this time. Here’s a step by step guide on how to do this in python.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F15192172&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fswaying-to-the-beat-snakes-dancing-ten-in-the-bed-gif-15192172&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fimages%2Fb70642da65fcce4b26cc3285e7363255%2Ftenor.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/d40bc8363d93b15c8ca7887550588f57/href">https://medium.com/media/d40bc8363d93b15c8ca7887550588f57/href</a></iframe><h3>Let’s get the basics right:</h3><p>Let’s look at a simple piece of code which has a for loop in it:</p><pre>import numpy as np</pre><pre>def square_a_number(a):<br>    return (a*a)</pre><pre>for i in np.arange(1,11):<br>    print(square_a_number(i))</pre><p>The code simply prints square of each number from 1 to 10. This, however is calculated sequentially. So. output would be:</p><pre>1<br>4<br>9<br>16<br>25<br>36<br>49<br>64<br>81<br>100</pre><p>I would now want that each loop is handled by a thread separately, so that all 10 values are calculated at once. I need a multi-core GPU for this, where each core handles one thread/process. Since I do not have one with me, I rent it online. I googled for the cheapest services, and got one named <em>vast.ai. </em>You can use whichever suits you.</p><h3>Renting a multi-core GPU:</h3><p>I went to <a href="https://vast.ai/console">vast.ai</a> and rented a 48-core machine.</p><figure><img alt="Select a configuration with 48 cores" src="https://cdn-images-1.medium.com/max/1024/1*RmR2Us3_qIH8oH0dLlcoZQ.png" /></figure><p>Connect to the machine when instance is ready.</p><figure><img alt="Connect to the rented machine" src="https://cdn-images-1.medium.com/max/1024/1*dwbyoklyY2VtndxnG5GreQ.png" /></figure><p>A Jupyter notebook would be launched, where you can write your code.</p><p>Right, on to the paralleled code for the above loop now:</p><pre>!pip install joblib</pre><p>The joblib library allows parallel processing in python.</p><pre>from multiprocessing import cpu_count<br>from joblib import Parallel<br>from joblib import delayed</pre><pre>executor = Parallel(n_jobs=cpu_count(), backend=&#39;multiprocessing&#39;) <br>tasks = (delayed(square_a_number)(i) for i in np.arange(1,11))<br>result = executor(tasks)<br>result</pre><p>Output:</p><pre>[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]</pre><ol><li>We first create an <em>executor </em>which will have <em>n_jobs </em>threads. In this particular case, we have set it to <em>cpu_count()</em> which would be 48 for this GPU - that is the number of cores it has.</li><li>Then, using the <em>delayed</em> keyword, we call our function. Instead of a traditional for loop syntax, we write it in the same line as shown above.</li><li>The <em>executor </em>runs the tasks and returns a list.</li></ol><blockquote>Notice how the output is now not printed as before, but rather is a list.</blockquote><p>That’s all for the basics. Let’s crack on with the SARIMA hyperparam tuning now.</p><h3>SARIMA Tuning:</h3><p>We want to try multiple conbinations of (p,d,q) and (P,D,Q,m). If you want to read abour ARIMA, SARIMA or other time-series forecasting models, you can do so <a href="https://otexts.com/fpp2/seasonal-arima.html#">here</a>.</p><p>We need to have multiple nested loops, something like:</p><pre>for p in p_vals:<br>  for d in d_vals:<br>    for q in q_vals:<br>      for P in P_vals:<br>        for D in D_vals:<br>          for Q in Q_vals:<br>            call_sarima_model_evaluation_function(p,d,q,P,D,Q)</pre><p>That is exactly what we do after converting it into paralleled mode. The main two functions look like these:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/375dc2081f797054680a7a66ad33abd2/href">https://medium.com/media/375dc2081f797054680a7a66ad33abd2/href</a></iframe><p>As you can see, we’ve called the evaluate_sarima_model using the delayed keyword.</p><p>We call this by:</p><pre><em># specify the range of values we want ot try for the different hyperprameters</em><br>p_values = np.arange(0, 2)<br>d_values = np.arange(1, 2)<br>q_values = np.arange(1, 4)<br>P_values = np.arange(0, 2)<br>D_values = np.arange(1, 2)<br>Q_values = np.arange(0, 3)<br>m_values = np.arange(7, 8)<br><br><em># total combinations being tried: 2*1*3*2*1*3*1 = 36</em><br><br>scor=evaluate_models_parallely(train, valid, p_values, d_values, q_values, P_values, D_values, Q_values, m_values)<br><br>scores=[]<br><strong>for</strong> tup_list <strong>in</strong> scor:<br>    <strong>for</strong> tup <strong>in</strong> tup_list:<br>        scores.append(tup)<br><br><em># sort the results on basis of RMSE scores (ascending)</em><br>scores.sort(key=<strong>lambda</strong> x: x[0])<br><br>print(&#39;<strong>\n</strong>Top 5 SARIMA params with minimum RMSEs:<strong>\n</strong>&#39;)<br><strong>for</strong> x <strong>in</strong> scores[:5]:<br>  print(f&#39;RMSE=<strong>{x[0]}</strong>  order=<strong>{x[1]}</strong>  seasonal_order=<strong>{x[2]}\n</strong>&#39;)<br><br>print(&quot;DONE!&quot;)</pre><blockquote>And here’s the link to the complete code, which you are free to use and customize:</blockquote><blockquote><a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/Demo-Parallel_Grid_Search_SARIMAX">Parallel Grid Search for SARIMA</a></blockquote><h3>Output:</h3><p>The output in my case, where 36 models were evaluated simultaneously, looks something like this:</p><figure><img alt="result of SARIMA tuning" src="https://cdn-images-1.medium.com/max/1024/1*Vw0_bc-GpfxDEG7rIvELWA.png" /></figure><h3>Destroy the GPU:</h3><p>Once you are done, do remember to destroy the rented GPU instance, you’re being charged every second!</p><figure><img alt="destroying the GPU instance" src="https://cdn-images-1.medium.com/max/1024/1*QYOZJ7an4X-3hFSucLHSkQ.png" /></figure><p>Give me a shout out if you need any help with the above, and I would be happy to help!</p><p>Ciao!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F15267145&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fsnake-notebook-gif-15267145&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fimages%2F6dac1c7efb1e6a01df5c26175d928b51%2Ftenor.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c9c21198b5b2053e388b27e46b2159b/href">https://medium.com/media/1c9c21198b5b2053e388b27e46b2159b/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2037fec1659" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/fine-tune-sarima-hyperparams-using-parallel-processing-with-joblib-step-by-step-python-code-2037fec1659">Fine tune SARIMA hyperparams using Parallel processing with joblib (Step by Step Python code)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Fun Project (Pose Detector) With Google’s Teachable Machine]]></title>
            <link>https://medium.com/analytics-vidhya/a-fun-project-pose-detector-with-googles-teachable-machine-6c7c8d650be1?source=rss-88960b2eb891------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c7c8d650be1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[teachable-machine]]></category>
            <category><![CDATA[pose-estimation]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[posenet]]></category>
            <dc:creator><![CDATA[Himanshu Chandra]]></dc:creator>
            <pubDate>Mon, 06 Apr 2020 15:35:34 GMT</pubDate>
            <atom:updated>2020-05-08T10:29:18.953Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s 2 PM… Done with a delicious lunch… Now working on your laptop… Droopy eyes… Goodbye world, Hellooo sweet dreams………. Ring any bells? ;)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F8805562&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fsleeping-gif-8805562&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fimages%2F28d98a17ae2260a0dc2085a09dcb5837%2Ftenor.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/040b3770a185c5c7451e4f004af20c46/href">https://medium.com/media/040b3770a185c5c7451e4f004af20c46/href</a></iframe><h3>What are we making?</h3><p>I prefer knowing in advance what I’m going to get at the end of an exercise, a blog or a training video, and I prefer it being fun, specially for beginners’ courses. So here’s what we are going to be building — <em>A real-time detector using your computer’s webcam which plays an alarm when you get sleepy &amp; tilt your head beyond certain degrees on either side.</em></p><blockquote><em>Right, so you may already have heard about </em><a href="https://teachablemachine.withgoogle.com/"><em>Google’s Teachable Machine</em></a><em>. In Google’s own words</em> —”<em> </em>Train a computer to recognize your own images, sounds, &amp; poses”. <em>It’s genuinely a fun tool to play around with, and with some coding expertize, you can extend it to create loads of fun apps.</em></blockquote><p>So let’s dive right in —</p><h3>Step 1</h3><p>Go to <a href="https://teachablemachine.withgoogle.com/train">https://teachablemachine.withgoogle.com/train</a> and start a ‘Pose Project’</p><h3>Step 2</h3><p>Add a class. Then, change the class names to ‘Neutral’, ‘Sleepy_Left’ and ‘Sleepy_Right’. You can have as many classes as you like. For example, you could have a class to handle a case when there’s no one in front of the camera. We’ll stick with three for now.</p><figure><img alt="Create placeholders for three classes" src="https://cdn-images-1.medium.com/max/1024/1*qzm69SXOaFvcOQMERoq6Dw.gif" /></figure><h3>Step 3</h3><p>Under ‘Neutral’ class, click on ‘Webcam’. I am working on Chrome browser, and you may be prompted to allow access to camera/microphone. Allow it if you see the prompt. Wait for your video to appear on screen.</p><p>Sit straight, and ‘Hold to Record’ for about 5 seconds — around 30–50 pose samples would be populated by Teachable Machine.</p><figure><img alt="Training the model for a straight pose, to be treated as neutral" src="https://cdn-images-1.medium.com/max/1024/1*-Je4YwMyybl4HToc3BlkZA.gif" /></figure><p>Do the same for the remaining two classes, tilting your head to the left for one class and tilting to the right for another.</p><figure><img alt="Training the model for left and right head tilts" src="https://cdn-images-1.medium.com/max/1024/1*xLfydNL7sHAMZNUclVIlwg.gif" /></figure><h3>Step 4</h3><p>Click on ‘Train Model’. We’ll not bother with any of the advanced settings for this project. It might take a couple of minutes to train the model.</p><p>You should be able to see the preview on the right when the model training is done. You can check by tilting your head if the model is able to classify your poses correctly now.</p><figure><img alt="Confirm that the model’s working fine via the preview pane" src="https://cdn-images-1.medium.com/max/1024/1*c5jXDCE0Qr7vrejk0Eyxcg.gif" /></figure><h3>Step 5</h3><p>Click ‘Export Model’.</p><p>— First, click on the ‘Download’ radio button. Copy the code generated and save it on your computer into a file named say, <em>index.html</em></p><p>— Second, click on ‘Upload (shareable link) radio button and then select ‘Upload my model’. Give it some time to upload.</p><p>Copy the link under ‘Your shareable link’ and keep it aside for use later.</p><p>You can exit the browser now, we’ll work on <em>index.html </em>next.</p><figure><img alt="Steps to export the generated model code into HTML format" src="https://cdn-images-1.medium.com/max/1024/1*HX_8pZe-ptfKy9XHbu0Yqg.gif" /></figure><h3>Step 6</h3><p>We’ll now make edits to <em>index.html . </em>The file is a mix of HTML and JavaScript code.</p><p>Change the value of <strong>const URL</strong> to the shareable link you copied in the last step. In my case, it looks something like this —</p><pre>// the link to your model provided by Teachable Machine export panel<br>const URL = &quot;<a href="https://teachablemachine.withgoogle.com/models/L71vjtyQV/">https://teachablemachine.withgoogle.com/models/L71vjtyQV/</a>&quot;;</pre><p>This single line of code change is enough to test it on your local system. Just open <em>index.html</em> in a browser and you should be able to see it work. The video below would appear flipped to you (like a selfie camera), but will work fine when you try it out yourself.</p><figure><img alt="Demo in browser showing pose detection on head tilt" src="https://cdn-images-1.medium.com/max/479/1*gpsHN0PUGsnOPCdNQ8LiGg.gif" /></figure><p>You must be able to see how the probability of each class, shown below the video canvas, changes as you make a movement. The dots you see is the model’s way of showing the body’s keypoints.</p><h3>Step 7</h3><p>Let’s put together the last bit now!</p><p><strong>1</strong>. First, download any alarm sound in the form of an mp3 file, which you would like to be played. Save it in the same folder where you have your <em>index.html</em></p><p>In my case, the name of the sound file is <em>wakeupalarm.mp3</em></p><p><strong>2</strong>. Insert a piece of code just after the <em>label-container</em> div so that it looks like this (<em>code marked as bold is new code</em>):</p><pre>&lt;div id=&quot;label-container&quot;&gt;&lt;/div&gt;</pre><pre><strong>&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;<br>&lt;audio id=&quot;wakeUpAudio&quot;&gt;<br>  &lt;source src=&quot;wakeupalarm.mp3&quot; type=&quot;audio/mpeg&quot;&gt;<br>  Your browser does not support HTML5 audio.<br>&lt;/audio&gt;</strong></pre><p>The above code creates a simple container for embedding our audio file into the HTML page.</p><p><strong>3</strong>. Just above the function <em>init(), </em>add this:</p><pre><strong>// functions to play and pause audio file</strong></pre><pre><strong>var aud = document.getElementById(&quot;wakeUpAudio&quot;);</strong></pre><pre><strong>function playAud() { <br>   aud.play(); <br> }</strong></pre><pre><strong>function pauseAud() { <br>   aud.pause(); <br> }</strong></pre><p>We just created functions to play and pause our audio file. We’ll c all these functions later.</p><p><strong>4</strong>. And finally, update the <em>predict()</em> function (<em>code marked as bold is new code</em>):</p><pre>async function predict() {<br>// Prediction #1: run input through posenet<br>// estimatePose can take in an image, video or canvas html element</pre><pre> const { pose, posenetOutput } = await model.estimatePose(webcam.canvas);</pre><pre>// Prediction 2: run input through teachable machine classification model</pre><pre> const prediction = await model.predict(posenetOutput);</pre><pre> for (let i = 0; i &lt; maxPredictions; i++) {<br>   const classPrediction = prediction[i].className + &quot;: &quot; +        prediction[i].probability.toFixed(2);<br>   labelContainer.childNodes[i].innerHTML = classPrediction;</pre><pre> <strong>// don&#39;t play audio when head&#39;s neutral with probability &gt;= 75%<br> if (prediction[0].probability.toFixed(2) &gt;= 0.75)<br>     pauseAud();<br> else<br>     playAud();</strong><br> }</pre><pre>// finally draw the poses<br> drawPose(pose);<br>}</pre><p>This needs a bit of explaining.</p><p>The <em>maxPredictions</em> variable present in above code equates to 3 in our case, as we have 3 classes — Neutral, Sleepy_Left and Sleepy_Right.</p><p>We have an array variable <em>prediction</em> which is used to access these 3 classes. So,</p><p>prediction[0] : Neutral</p><p>prediction[1] : Sleepy_Left</p><p>prediction[2] : Sleepy_Right</p><p>Our new piece of code simply states that if the probability of the pose being that of Neutral class is equal to or greater than 75%, pause the audio, else play it. The figure of 75% is something I kept to manage certain degree of inaccuracy, but you can play around with that for your particular case.</p><h3><strong>That’s it!</strong></h3><blockquote><em>You can download the full </em>index.html<em> file from </em><a href="https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/SleepyOne">https://github.com/HCGrit/MachineLearning-iamJustAStudent/tree/master/SleepyOne</a>.</blockquote><blockquote><em>You will need to add the sound file by yourself.</em></blockquote><h3>Cool Facts</h3><ol><li>This works for not only you, but anyone sitting in front of the webcam! Since the model works by detecting key body parts/joints, it can be used by anyone whose keypoints are detectable by the model.</li><li>The actual algorithm working behind the scenes is known as <strong>PoseNet</strong>. You can read more about it, and it’s various use cases here — <a href="https://www.tensorflow.org/lite/models/pose_estimation/overview">https://www.tensorflow.org/lite/models/pose_estimation/overview</a>.</li><li>You can also train an audio model and create an app to implement an Alexa or Siri like functionality.</li></ol><h3>Parting Thoughts</h3><p>I had created this project primarily for my colleagues at my workplace. Most of them had enquired about how to get started with Artificial Intelligence &amp; Machine Learning and I wanted to introduce them to the field with something fun and interesting, before they move on to the heavy stuff.</p><p>Some of them extended this to make gesture-controlled apps to launch Outlook, Excel and do various daily tasks without touching the keyboard.</p><p>I hope this sparks interest in those reading this. If you happen to create any interesting projects around this, please do let me know through your comments, I would love to explore those!</p><p>See you soon with something new!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F12942559&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Frobot-blender-test-robot-bender-robot-bye-gif-12942559&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fimages%2F5161d5afc2322c8bd7cd653cf857147b%2Ftenor.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/fab54608fa450ce69d121057dbde7854/href">https://medium.com/media/fab54608fa450ce69d121057dbde7854/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c7c8d650be1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-fun-project-pose-detector-with-googles-teachable-machine-6c7c8d650be1">A Fun Project (Pose Detector) With Google’s Teachable Machine</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>