<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[convergeML - Medium]]></title>
        <description><![CDATA[converging to all things ML - Medium]]></description>
        <link>https://medium.com/convergeml?source=rss----42cc29e4dc68---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>convergeML - Medium</title>
            <link>https://medium.com/convergeml?source=rss----42cc29e4dc68---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:49:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/convergeml" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stack Overflow and Open source]]></title>
            <link>https://medium.com/convergeml/stack-overflow-and-open-source-5ddcd38213f?source=rss----42cc29e4dc68---4</link>
            <guid isPermaLink="false">https://medium.com/p/5ddcd38213f</guid>
            <category><![CDATA[stackoverflow]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[stackoverflow-survey]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Shoaibkhanz]]></dc:creator>
            <pubDate>Mon, 23 Sep 2019 06:23:49 GMT</pubDate>
            <atom:updated>2019-09-23T06:54:52.623Z</atom:updated>
            <content:encoded><![CDATA[<h4>Stack Overflow 2019 survey analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-qn8SnUg86F1N8-LaHY5A.png" /></figure><p>Stack overflow is where everyone goes to get help. <strong>It is driven by the community of developers to help the community back</strong>.They conduct a survey every year and record all the information about the community.</p><p><strong>In 2019, </strong>they did it again and this post summarises what I found answering few questions. There are many questions that you may ask out of a survey. Stack overflow had their own questions and they have published their insights <a href="https://insights.stackoverflow.com/survey/2019">here</a> and I encourage you to read it. I have my own set of 3 questions centred around the open source world and Stack overflow also from here on I am going to state Stack overflow as <strong>(SO).</strong></p><p><strong>Question 1.</strong> Who are the users of Stack overflow in <strong>2019</strong> ?</p><p><strong>Question 2.</strong> How soon can you start contributing to <strong>open source</strong> ?</p><p><strong>Question 3.</strong> Will contributing to open source help you <strong>earn more</strong> ?</p><h4><strong>Where is the data and notebook ?</strong></h4><p>The data used for this analysis can be found <a href="https://insights.stackoverflow.com/survey">here </a>. Is there a notebook to follow along the analysis ? certainly <a href="https://github.com/shoaibkhanz/so_open-source_survey_analysis"><strong>yes</strong></a>…</p><h4><strong>What does the data contain ?</strong></h4><p>Stack overflow is proud of their community and they say the following about their survey.</p><blockquote><strong><em>‘</em></strong><em> with nearly</em><strong><em> 90,000 responses</em></strong><em> fielded from over </em><strong><em>170 countries</em></strong><em> and dependent territories, our 2019 Annual Developer Survey examines all aspects of the developer experience from career satisfaction and job search to education and opinions on open source software.</em><strong><em>’</em></strong></blockquote><h3>Diving into the analysis</h3><h4><strong>Question 1. Who are the users of Stack overflow in 2019 ?</strong></h4><p>Let us start by looking at some interesting statistics about this data, I believe this should help us to understand and appreciate the diversity of users that come to SO to seek and give help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*X5yHrigyAjmPhZ2mdDfKEQ.png" /></figure><p>Beware, the dataset contains some spurious information, such as age it looks like infants are also visiting SO for filling the survey, thus for age I have <strong>excluded</strong> the users below and at the <strong>age of 5</strong>.</p><blockquote><strong><em>AGE</em></strong></blockquote><p>Overall, out of the ~<strong>80,000 users</strong>, the average user age is <strong>30</strong> years using SO, with maximum age as high as <strong>99 </strong>(<em>not sure if this is for real :)</em>).</p><p>The message is<strong> “its never too late to start coding!!”</strong></p><blockquote><strong><em>GENDER</em></strong></blockquote><p>Male are the majority in the respondents. However, female users have increased over the years. With efforts like <a href="https://rladies.org/"><strong>RLadies</strong></a> and <a href="https://djangogirls.org/"><strong>DjangoGirls</strong></a>, the future looks bright.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nQ1slv_8D0ZTvNAXPWOkw.png" /></figure><blockquote><strong><em>Education</em></strong></blockquote><p><strong>45%</strong> are bachelors, <strong>23%</strong> are masters and only <strong>3%</strong> are PHDs, thus combined together ~70% of users have education ≥= Bachelors, whilst others have some sort of professional education.</p><blockquote><strong><em>Professional Coder</em></strong></blockquote><p>The average years of experience as professional coder is<strong> 8 years</strong>, while the median <strong>6 years</strong>, look above for the distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qtYcOImp_caN4Bq009BFVQ.png" /></figure><blockquote><strong><em>Data Scientist or not</em></strong></blockquote><p>Only <strong>7% of data scientist</strong> responded to the survey, while 93% percent were other developers and engineers.</p><blockquote><strong><em>Salaries</em></strong></blockquote><p>Salaries are highly skewed to the right as there are really high salaries to some individuals. The average salary overall is ~127,000 USD, while median salaries are around 60,000 USD. We will revisit salaries and look at them in detail, while answering the third question.</p><blockquote><strong><em>Country</em></strong></blockquote><p>Here is the visualisation of the countries who responded to the survey. The top respondents came from either <strong>USA , Canada, UK, Germany or India.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQWhrEn-qm6PhOJgHxKTwg.png" /></figure><blockquote><strong><em>Programming Languages</em></strong></blockquote><p><strong>Javascript and HTML/CSS</strong> which are the building blocks of any web technology top the list. <strong>SQL and Python</strong> which are heavily used in data science/analytics quickly follows. Java comes after Python and surprisingly <strong>R</strong> is around 18th in position. Look here for <a href="https://spectrum.ieee.org/static/interactive-the-top-programming-languages-2019">ranking</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CtcrI9lX_QNl42IunHZIdQ.png" /></figure><blockquote><strong><em>First lines of code</em></strong></blockquote><p>The average age by which an individual has written their first line of code is <strong>15 years</strong>. have a look at the distribution there are people writing their first lines of code even past their 60s. It’s incredible to see such motivation and contribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w4ElKDAwa-1LQiKEaeYnhw.png" /></figure><blockquote><strong><em>Coding Experience</em></strong></blockquote><p>The users have coding experience ranging from as little as just <strong>1 year</strong> to around <strong>50 years</strong> with an <strong>average of 12</strong> years of experience.</p><blockquote><strong><em>Visit Duration(SO)</em></strong></blockquote><p><strong>64% </strong>of users spend at least 30 minutes on SO</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1OvNTdUQFqvO8jg8PqHtHg.png" /></figure><blockquote><strong><em>Visit Frequency(SO)</em></strong></blockquote><p><strong>61% </strong>of people visit SO daily, while around a percent had never visited SO before they filled this survey i.e. around 800 people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2p7G5M5rzbazkPxvnv5sqg.png" /></figure><h4>Question 2. How soon one can start contributing to open source ?</h4><p>To answer this question we need to ask 2 more questions.</p><ul><li>What is the age group of people who contribute to open source ?</li><li>How much time open source contributors devote to coding ?</li></ul><p>Before we go asking the question, let’s just see the usual statistics. We see that around <strong>36%</strong> of users <strong>never contributed</strong> to open source</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*45_K8oPKxaBbGnMfeMNpnw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHgXMBh-A_JQ70kSg_LGcg.png" /></figure><blockquote><strong><em>Age + Open source</em></strong></blockquote><p>I will encode the above categories(OpenSourcer) into simple ones and call them <strong>highly_engaged</strong>,<strong> slightly_engaged</strong>,<strong> almost_not_engaged </strong>and<strong> not_engaged. </strong>The percentage (<strong>12%</strong>) of highly engage is same across the years 20–50, I am not going to read too much into this but It would be interesting to investigate if users who are highly engaged in open source in their 20s continue to be highly engaged in their 50s and beyond.</p><p>Thus the early you start contributing to open source the better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCs-RhMdEjKsDsFgD8ursw.png" /></figure><h4>Question 3. If one contributes to open source will one earn more ?</h4><p>Let us as the most simple question whats the difference in salaries between an opensourcer vs non-opensourcer. There is a difference of around <strong>9000 USD</strong>, so it looks like if I contribute I end up earning more ? but open source doesn&#39;t pay directly!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*6xFNIkUKOPuM8xXfVckCHw.png" /></figure><blockquote><strong><em>Salaries + Open source + Region</em></strong></blockquote><p>Interestingly, median salaries in Africa doesn’t vary much, while thats not true for all the other regions. The figures(USD) also highlight the difference in salaries across regions. It looks like if you are contributing to open source you are earning better at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*BYGpNq6d9ILmdYUICp6XIw.png" /></figure><blockquote><strong><em>Salaries + Open source + Country</em></strong></blockquote><p>I would encourage you to go to the notebook link to really play around with these collab notebooks to visually interact with the below maps. There places in Africa, south America, Middle east and very few parts in Europe where respondents have never contributed to open source.</p><p>You can also see the size of the dots, the largest is USA where the median salary is around 120,000(USD) if you contribute to open source otherwise its 100,000(USD).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7QZ-D8DZC9vMDuigqRVGw.png" /><figcaption>OPEN SOURCE = YES</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kaq8BwDY8JoDtaLCC4FWWQ.png" /><figcaption>OPEN SOURCE = NO</figcaption></figure><h4><strong>Summary</strong></h4><p>Stack overflow is a great community where people come from different age groups, countries, education etc. Age is not the barrier to start coding.</p><p>We also saw in this analysis that although open source is practiced with more experienced individuals the difference is not very large but the returns are. The Salaries are higher if you contribute to open source. The big driver for this salary difference are the skills you will gain by contributing to the open source. So while I prepare myself to become a contributor so should you.</p><p>Happy Coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ddcd38213f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/convergeml/stack-overflow-and-open-source-5ddcd38213f">Stack Overflow and Open source</a> was originally published in <a href="https://medium.com/convergeml">convergeML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Simple ML-Framework of Cookie Jars]]></title>
            <link>https://medium.com/convergeml/a-simple-ml-framework-of-cookie-jars-56ee3d686051?source=rss----42cc29e4dc68---4</link>
            <guid isPermaLink="false">https://medium.com/p/56ee3d686051</guid>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shoaibkhanz]]></dc:creator>
            <pubDate>Mon, 11 Feb 2019 20:27:41 GMT</pubDate>
            <atom:updated>2019-02-11T20:32:22.336Z</atom:updated>
            <content:encoded><![CDATA[<p>In our every day lives we make important decisions for our well-being and it’s necessary that we make the right ones more often than wrong. Over a period of time, we learn from our mistakes and we avoid repeating them.</p><blockquote>We are constantly approaching an expert state as we learn to solve particular kinds of problems.</blockquote><p>In this post, we will read through and understand a framework that will help you grasp the idea of Machine Learning. It will also help you recognize the concepts as you experience them in the real world.</p><p>If you didn’t know Machines can also become experts if they are trained well, we will explore how they can become efficient at certain tasks and sometimes exceed human capabilities.</p><p>In the past, we would have never imagined driverless cars. Most of us always thought of <em>flying cars</em> but never driverless, our imagination failed us, developments in AI has enabled us to create such products and so the possibilities are limitless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F22Mh7b7Y85OscnOPwO2kw.png" /><figcaption>driver-less car(left) — flying car(right)</figcaption></figure><blockquote>So, If you want to create something useful for the benefit of the society, you can use Machine Learning to optimise and make it even better.</blockquote><p><em>Let’s dive into the </em><strong><em>Cookie Jar Framework</em></strong><em> and take a glimpse at 6 Jars.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*JNWB_lsLPMpYEQP0eHpArw.png" /></figure><h3>1| Data</h3><p>Data is essential to make any decision, machines use them too. <strong><em>But is more data always better?</em> </strong>The answer is counter-intuitive. In the simplistic sense, the more data we have the better but we need to be careful and must try to understand <strong><em>how that data was produced. </em></strong>We thus avoid biases as we model a solution controlling for the factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*Er1GgyYHYmpukMHCD6CU7w.png" /></figure><h3>2| Tasks</h3><p>Imagine we are a data scientist at <strong>NETFLIX </strong>and our job is to build models that classify movies into <em>action, romance, adventure, comedy </em>etc, perhaps we will use the scenes in the movies and take cues from the title of the movies to classify them into their respective genre. i.e. we will perform a <strong><em>task of classification</em></strong> using some statistical technique.</p><p>So to summarise, even if we are from <strong><em>GAFA(google, amazon, facebook, apple)</em> </strong>we need to strategically think and prioritise tasks or problems that we will invest our time to solve. <strong><em>It could be classifying movies or fake videos, or perhaps trying to predict sales of iPhones or a toothbrush.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*kVOylM4hEH6dsRq2nc6hOA.png" /></figure><h3>3| Model</h3><p>Let’s continue our job as a data scientist at NETFLIX. We definitely have a lot of data and we are sure that business’s priority is to <strong><em>predict viewers favorite movies</em></strong> and then recommend them similar ones.</p><p><strong>How are we going to approach the task?</strong></p><p>We may build a simple model or a complex one that looks at the distribution of movies that viewers have watched in the past. But most movies do not have just action or comedy it is a mix and perhaps some viewers like Tom Cruise more than any other actor or perhaps they prefer movies made in 1990s and 2000s and so this relationship can get complex very quickly.</p><p>Anyways since we build these models subconsciously i.e our likes and dislikes are rooted in our subconscious, machines can have a hard time predicting them, but we will see how machines can approximate them really well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*WJu_hAqYhzHhosN3_O36Xw.png" /></figure><h3>4| Loss</h3><p>We will use our previous example of predicting favourite movies of viewers. In the 3rd jar we built a model but we couldn&#39;t decide if the simple model is better than a complex one or vice versa. To solve this particular conundrum we look at the errors that our model makes. If we make fewer errors than most models then we would consider that model to be a good one.</p><blockquote><em>_________________ERRORS == LOSS_________________</em></blockquote><p><em>Error is our loss and loss is our Errors</em>, we mean the same thing when we call either of them. We are now 4 jars deep and so we are almost there to understand the ML trickery!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*6XQ12R25ZsL0q-xAczcFCg.png" /></figure><h3><strong>5| Learning Algorithm</strong></h3><p>As humans, we learn as we make mistakes and so when a model makes a mistake we want the model to update itself and avoid the errors. <em>All algorithms do the same. </em>They learn from mistakes and they optimise for the solution. <strong><em>Thus a learning algorithm consists of a loss function(error function) which it optimises, correcting its predictions as it learns more from data.</em></strong></p><p>We could try various algorithms to <strong><em>predict</em></strong> <strong><em>favourite vs not so favourite movies</em></strong> for e.g. we could use <em>logistic regression</em>, <em>classification trees</em>, <em>neural network</em> etc. So this jar is filled with algorithms and techniques to solve a business problem and it is an important jar since without them finding a simple solution would be difficult and perhaps impossible for complex ones.</p><p>However, <strong><em>How do we know which algorithm is better for our task? </em></strong>We will discuss this intriguing question in the next Jar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*J0YfoUY2BA8w1ryyEwaHWw.png" /></figure><h3>6| Evaluation</h3><p>This is the final jar in the Cookie Jar framework. Once we have built the model and found a solution, we would want to test it in the real world, i.e Evaluate it, Examine it and repeat the process of the model building if our model performs poorly in the real world. There are many evaluation metrics that you might have heard of such as Accuracy, Precision, Recall , Rsquared etc..</p><p>I would suggest you read a dedicated blog of mine on the <a href="https://medium.com/convergeml/conundrums-of-the-confusion-matrix-2fa82293707a"><strong><em>confusion matrix</em></strong></a><strong><em> </em></strong>and it will familiarise you with the terminology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aPedO12rTKVxxBBGK99t-A.png" /></figure><h3>Summary</h3><p>We just went through the 6 Jars of Machine learning, the framework springs from the professors of OneFourthLabs and I think it&#39;s a great summary and a framework to help someone understand the process of Machine Learning. It’s pure engineering and science and not a magical trickery. I hope you guys liked it and if you did, please clap a little and enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56ee3d686051" width="1" height="1" alt=""><hr><p><a href="https://medium.com/convergeml/a-simple-ml-framework-of-cookie-jars-56ee3d686051">A Simple ML-Framework of Cookie Jars</a> was originally published in <a href="https://medium.com/convergeml">convergeML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A short Introduction to Pytorch using logic gates in Perceptron]]></title>
            <link>https://medium.com/convergeml/a-short-introduction-to-pytorch-using-logic-gates-in-perceptron-a8779fd93bd4?source=rss----42cc29e4dc68---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8779fd93bd4</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[perceptron]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <dc:creator><![CDATA[Shoaibkhanz]]></dc:creator>
            <pubDate>Wed, 30 Jan 2019 19:38:34 GMT</pubDate>
            <atom:updated>2019-01-30T20:45:42.230Z</atom:updated>
            <content:encoded><![CDATA[<p>A Perceptron can be thought of as an algorithm with an objective to classify the output into binary outcomes i.e. 1 or 0, True or False. It is a linear classifier, thus it uses a linear combination of weights, inputs and the bias input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*RmWOzbfRLn59XNljwac6Tg.png" /></figure><p>The above equation contains w0 x0which is called a `bias` or `intercept` and the equation can be further summarised in the following way.</p><p>The output y = 1 if the result of the dot product is ≥ 0 else if the dot product is &lt; 0 then, y = 0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*neDlDpsjCjTpeoYy525Awg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*KFvT1t8tJwVc8ba1LxQQfg.png" /></figure><p>Pytorch is an open source deep learning platform that provides a seamless path from research prototyping to production deployment. Pytorch is an extension of numpy and allows us to use GPUs to solve compute-intensive problems in research and in business. We will implement the perceptron algorithm in `Pytorch` and use logic gates to better understand the effects that weights and bias have on the output.</p><h3><strong>Logic Gates</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inCXF9LyvBDX2ogRCDRJxw.png" /></figure><p>Imagine we have 4 observations with 1 and 0 as inputs and we want to predict 1 or 0 based on some function. In the below examples we will explore AND, OR and NOT function and we will use the 4 observations to predict the correct value for each of the 3 functions.</p><p>Please focus on the equation that we described above as we will use it to create the input tensor in Pytorch. Vector x1<strong><em> </em></strong>and<strong><em> </em></strong>x2 are<strong><em> </em></strong>defined as 1 or 0 switching their places as we move down the observations, while the constant 1 is the bias i.e. x0 vector. We now have the inputs ready but we are still missing w0,w1, and<strong><em> </em></strong>w2,<strong><em> </em></strong>we need to define them for each logic function.</p><pre>#importing pytorch library<br>import torch</pre><pre>#creating input tensor <br>input_tensor = torch.tensor([(0, 0, 1), (0, 1, 1), (1, 0, 1), (1, 1, 1)],dtype = torch.double)<br>print(input_tensor)<br>#tensor([<br>#[0., 0., 1.],         <br>#[0., 1., 1.],         <br>#[1., 0., 1.],         <br>#[1., 1., 1.]], dtype=torch.float64)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*tbsjZ90DP8__e73ZU4wTQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*olXVxozaSGTzY1iqKFKX6g.png" /></figure><h3>1| AND</h3><p>We will use the above input tensor and the linear combination of weights and their inputs to give us the right output for AND function.</p><p>As we described above we need to perform the dot product between the weights and the inputs, thus the most obvious question is what are the correct weights and how do we perform the dot product in Pytorch?</p><p>Throughout this post, I will define the weights that I know will work. However, I encourage you to test other weights that might work as a solution and give us the right answer. I am using torch.tensor to create a tensor object in python, it accepts dtype parameter, I will encourage you to read more about it <a href="https://pytorch.org/docs/stable/tensors.html">here</a>.</p><pre>and_weights = torch.tensor([1, 1, -2],dtype = torch.double) #creating a tensor<br>print(and_weights.shape)<br>#torch.Size([3]) object shape</pre><p>As you will notice the shape of and_weights is torch.Size([3]) and thus to perform a valid dot product we need to reshape it into rows and columns, we will do like so and_weights.resize_(1,3) The object is now resized to torch.Size([1, 3])</p><p><em>pay attention to the underscore(</em><em>_) in </em>.resize_<em> function</em></p><blockquote>Any operation that mutates a tensor in-place is post-fixed with an _. For example: x.copy_(y), x.t_(), will change x.</blockquote><p>We now use the weights and the inputs to perform the dot product, but the catch is .dot() will not work as we are used to innumpy. We must use mm i.e matrix multiplication for 2d size tensors or more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*di8BeSirBKbRaO9UiYFg_g.png" /></figure><p>input_tensor also needs to be transposed and thus we perform a valid matrix multiplication i.e. (1 x 3) (3 x 4) for weights and input tensor respectively.</p><pre>and_matrix_mult = and_weights.mm(input_tensor.t())<br>print(and_matrix_mult)<br>#tensor([[-2., -1., -1., 0.]], dtype=torch.float64)</pre><p>The matrix output gives us some negatives and positive numbers which when passed through the above formula gives us and_output.</p><pre>and_output = [[1 if i&gt;= 0 else 0 for i in x ]for x in and_matrix_mult]<br>print(and_output)<br>[[0, 0, 0, 1]]</pre><p>That’s it for the perceptron on AND gate!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*csynckFmVpa18r-Q_Z0iog.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*9ghl14WU6-7AtPhlAXM-ig.png" /></figure><h3><strong>2| OR</strong></h3><p>The method to get OR function is similar to the AND function so I will race through the code and explain the changes. Once again, we use the same input tensor and use the linear combination of weights and inputs to give us the right OR function output. The following 4 lines code does that for us.</p><pre>or_weights = torch.tensor([2,2,-1],dtype = torch.double).resize_(1,3)<br>or_matrix_mult = or_weights.mm(input_tensor.t())<br>or_output = [[1 if i&gt;= 0 else 0 for i in x ]for x in or_matrix_mult]<br>print(or_output)<br>#[[0, 1, 1, 1]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*C5a-vW4NsQMFRXcdiP0cbw.png" /></figure><h3><strong>3| NOT</strong></h3><p>The NOT function is a bit different to other perceptrons, the NOT function only cares about one input. The operation returns a 0 if the input is 1 and a 1 if it’s a 0. The other inputs to the perceptron are ignored.</p><pre>not_weights = torch.tensor([2,-4,1],dtype = torch.double).resize_(1,3)<br>not_matrix_mult = not_weights.mm(input_tensor.t())<br>not_output = [[1 if i&gt;= 0 else 0 for i in x ]for x in not_matrix_mult]<br>print(not_output)<br>#[[1, 0, 1, 0]]</pre><h3>Summary</h3><p>You have just explored the very basic application of Pytorch using perceptrons. The operations we have explored were AND, OR and NOT, however, there are many other operations like XOR, NAND etc. The other operations can be thought of as multi-layered i.e. once you have passed them an operation you will need them to go through another operation before you get the expected result. for e.g. NAND = NOT + AND.</p><p>I hope this post was helpful, Please leave some constructive feedback and definitely few claps it keeps me motivated to write more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8779fd93bd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/convergeml/a-short-introduction-to-pytorch-using-logic-gates-in-perceptron-a8779fd93bd4">A short Introduction to Pytorch using logic gates in Perceptron</a> was originally published in <a href="https://medium.com/convergeml">convergeML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why ECDF is better than a Histogram]]></title>
            <link>https://medium.com/convergeml/why-ecdf-is-better-than-a-histogram-85deda4129ed?source=rss----42cc29e4dc68---4</link>
            <guid isPermaLink="false">https://medium.com/p/85deda4129ed</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[probability]]></category>
            <dc:creator><![CDATA[Shoaibkhanz]]></dc:creator>
            <pubDate>Sun, 20 Jan 2019 20:10:22 GMT</pubDate>
            <atom:updated>2019-01-20T20:32:22.880Z</atom:updated>
            <content:encoded><![CDATA[<p>Every exploratory analysis always has a histogram, we seek answers but histograms are less informative and biased. This may break the hearts of many who have histograms as their first weapon to perform <a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis">EDA</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*pVgpVU779qbaFMlyVGJGNw.png" /></figure><p>In this post, we will learn to draw a histogram and an ecdf using <a href="https://www.python.org/">python</a>, and then we will explore why ecdf is a better choice as a first visualization. We will use <a href="https://en.wikipedia.org/wiki/Iris_flower_data_set">iris</a> dataset to draw the histogram of setosa’s petal length.</p><h4><strong>Import all the required libraries.</strong></h4><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>from sklearn.datasets import load_iris</pre><h4><strong>Get the iris data and convert it into a dataframe.</strong></h4><pre>data = load_iris()<br>iris = pd.DataFrame(data= np.c_[data[&#39;data&#39;], data[&#39;target&#39;]],<br>                     columns= data[&#39;feature_names&#39;] + [&#39;target&#39;])</pre><h4><strong>Subset the data to setosa species and draw a histogram of its petal length with 6 bins.</strong></h4><pre>#subsetting<br>setosa = iris[iris[&#39;target&#39;] == 0]</pre><pre>#histogram<br>sns.set_style(&#39;whitegrid&#39;)<br>_ = plt.hist(setosa[&#39;petal length (cm)&#39;], bins = 6)<br>plt.title(&#39;Histogram&#39;)<br>plt.xlabel(&#39;setosa petal length (cm)&#39;)<br>plt.ylabel(&#39;counts&#39;)<br>plt.show()</pre><p>You can see the histogram from the code above. Please notice the structure of the histogram has, the bar heights are higher between the interval <em>1.3 to 1.7.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*KzoKn3zVm0gAXk0AoIMfAA.png" /></figure><p>Imagine the same histogram with 5 bins and notice how the distribution has changed. The way you bin your data can alter the way to understand it. This is called <strong>Binning Bias</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*TbDMf0yWyKIY7FBwz2pp8g.png" /></figure><h3>What’s the Alternative?</h3><p><a href="https://en.wikipedia.org/wiki/Empirical_distribution_function">ECDF</a> stands for empirical cumulative distribution function, which you should use more often to understand your data. The algorithm to build an ecdf is illustrated in the code. The output of the custom ecdf function, when applied to setosa dataframe contains 2 arrays or vectors.</p><ul><li>The 1st array<em>(e.g. setosa_sort)</em> contains the petal length in ascending order.</li><li>The 2nd array<em>(e.g. setosa_percentiles) </em>contains the percentiles for the respective value of petal length in <em>setosa_sort</em>.</li></ul><pre>#custom function for ecdf﻿<br>def empirical_cdf(data):<br>  percentiles = []<br>  n = len(data)<br>  sort_data = np.sort(data)<br>  <br>  for i in np.arange(1,n+1):<br>    p = i/n<br>    percentiles.append(p)<br>  return sort_data,percentiles</pre><pre>#use the function on the setosa iris data<br>setosa_sort, setosa_percentiles = empirical_cdf(setosa[&#39;petal length (cm)&#39;]﻿﻿﻿)</pre><p>We now use the <em>setosa_sort </em>and <em>setosa_percentiles </em>ndarrays (numpy n-dimensional arrays)<strong> </strong>to plot the ecdf, the sorted setosa data goes as the x-axis and their respective percentiles go as an input to the y-axis.</p><pre>_ = plt.plot(setosa_sort,setosa_percentiles, label=&#39;Data&#39;,color = &#39;blue&#39;)<br>plt.title(r&#39;$\bf{Empirical \ CDF}$&#39;)<br>plt.xlabel(&#39;setosa petal length (cm)&#39;)<br>plt.ylabel(&#39;percentiles&#39;)<br>plt.show()﻿﻿﻿</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*TQixGE5mL3V4GeNQGnva3g.png" /></figure><p>The plot above is an ecdf for setosa petal length. We may observe from the plot that the median(50th percentile) is around 1.5 cms, histograms don&#39;t tell you that. We are going to make the above ecdf a little more detailed.</p><pre>#data preparation for plotting﻿, capturing important statistics<br>ymedian = np.empty(len(setosa))<br>ymedian.fill(0.5)<br>ymax = np.empty(len(setosa))<br>ymax.fill(np.max(setosa_percentiles))<br>ymin = np.empty(len(setosa))<br>ymin.fill(np.min(setosa_percentiles))<br>y25 = np.empty(len(setosa))<br>y25.fill(0.25)<br>y75 = np.empty(len(setosa))<br>y75.fill(0.75)</pre><pre>fig,ax = plt.subplots()</pre><pre># Plot the data<br>setosa_ecdf = ax.plot(setosa_sort,setosa_percentiles, label=&#39;Data&#39;,color = &#39;blue&#39;)</pre><pre># Plot the lines<br>max_line = ax.plot(setosa_sort,ymax, label=&#39;Max&#39;, linestyle=&#39;--&#39;,color = &#39;#001c58&#39;)<br>y75_line = ax.plot(setosa_sort,y75, label=&#39;75Percentile&#39;, linestyle=&#39;--&#39;,color = &#39;#ff7a69&#39;)<br>med_line = ax.plot(setosa_sort,ymedian, label=&#39;Median&#39;, linestyle=&#39;--&#39;,color = &#39;#fb6500&#39;)<br>y25_line = ax.plot(setosa_sort,y25, label=&#39;25Percentile&#39;, linestyle=&#39;--&#39;,color = &#39;#4fb4b1&#39;)<br>min_line = ax.plot(setosa_sort,ymin, label=&#39;Min&#39;, linestyle=&#39;--&#39;,color = &#39;#ffbb39&#39;)</pre><pre>#annotate lines<br>ax.annotate(&#39;Max/$100^{th}$ Percentile&#39;,xy = (1,0.96),color = &#39;#001c58&#39;)<br>ax.annotate(&#39;$75^{th}$ Percentile&#39;,xy = (1,0.71),color = &#39;#ff7a69&#39;)<br>ax.annotate(&#39;Median/$50^{th}$ Percentile&#39;,xy = (1,0.46),color = &#39;#fb6500&#39;)<br>ax.annotate(&#39;$25^{th}$ Percentile&#39;,xy = (1,0.21),color = &#39;#4fb4b1&#39;)<br>ax.annotate(&#39;Min/$1^{st}$ Percentile&#39;,xy = (1.3,0.04),color = &#39;#ffbb39&#39;)</pre><pre>﻿#final and important bits<br>plt.tight_layout()<br>plt.title(r&#39;$\bf{Empirical \ CDF}$&#39;)<br>plt.xlabel(&#39;setosa petal length (cm)&#39;)<br>plt.ylabel(&#39;percentiles&#39;)<br>plt.show()</pre><p>Observe, how we have plotted all the important statistics from the data. This sort of statistics you would have seen in a <a href="https://en.wikipedia.org/wiki/Box_plot">box and whisker plot</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*lL3qc5X4qg4UG2MsqJE8-Q.png" /></figure><h3>Outliers in Histogram &amp; ECDF</h3><p>We are going to generate some random data with an outlier and visualize it on the histogram and on the ecdf.</p><pre>#data with outlier<br>outlier_data = np.random.random(100)<br>outlier_data = np.append(outlier_data,4)</pre><pre>#plotting with an outlier<br>plt.figure(figsize=(18,10))<br>plt.subplot(1,2,1)<br>_ = plt.hist(outlier_data)<br>plt.xlabel(&#39;random values&#39;)<br>plt.ylabel(&#39;count&#39;)<br>plt.title(r&#39;$\bf{Histogram \ with \  an \ Outlier}$&#39;)</pre><pre>plt.subplot(1,2,2)<br>d, p = empirical_cdf(outlier_data)<br>plt.plot(d,p)<br>plt.xlabel(&#39;random values&#39;)<br>plt.ylabel(&#39;percentiles&#39;)<br>plt.title(r&#39;$\bf{ECDF \ with \  an \ Outlier}$&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wI8M-_O-s0zCw0KntdmlJQ.png" /></figure><p>You can see how the outlier can also be spotted in an ecdf as well. Since ecdf does not depend on bins, its structure doesn&#39;t change like histograms and it still captures the outliers. I have plotted ecdf to be a line, if we visualize them as points, you will clearly see that the only value after 0.77 is 4 (which is an outlier in this case).</p><pre>Try yourself: use <strong>marker = &#39;o&#39; </strong>to visualize the ecdf as points instead of lines.</pre><h3>Bimodal Data</h3><p>We will generate bimodal data and then visualize it with a histogram and an ecdf. In fact, the reason the data is called bimodal is that it has 2(bi) peaks(modes). If you build a histogram you will see the 2 peaks, while the question is can we detect it in ecdf?</p><pre>#generate bimodal data<br>bimodal = np.append(np.random.normal(0, 1, int(0.5 * 10000)),<br>                    np.random.normal(5, 1, int(0.7 * 10000)))</pre><pre>#histogram of bimodal data<br>plt.figure(figsize=(18,10))<br>plt.subplot(1,2,1)<br>_ = plt.hist(bimodal)<br>plt.xlabel(&#39;random normal values&#39;)<br>plt.ylabel(&#39;count&#39;)<br>plt.title(r&#39;$\bf{Histogram \ of \ a \ Bimodal \ data}$&#39;)</pre><pre>#ECDF of a bimodal data<br>plt.subplot(1,2,2)<br>bm_d,bm_p = empirical_cdf(bimodal)<br>_ = plt.plot(bm_d,bm_p)<br>plt.xlabel(&#39;random normal values&#39;)<br>plt.ylabel(&#39;percentiles&#39;)<br>plt.title(r&#39;$\bf{ECDF \ of \ a \ Bimodal \ data}$&#39;)<br>plt.annotate(r&#39;$2^{nd}$&#39;+&#39;Curve --&gt; 2 Peaks \n i.e. Bimodal Data&#39;,xy = (0.5,0.5),color = &#39;#001c58&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zw_dOVbYfFNXBPgO20TBVA.png" /></figure><p>We just saw how in ecdf we can detect bimodal data as well. I hope by this time you recognize it’s merit.</p><p><strong><em>Just for fun</em></strong>, let us plot Versicolor, Virginica and Setosa species together and you will see how we can compare the distribution among different categories of species.</p><pre>#get data for other iris species<br>versicolor = iris[iris[&#39;target&#39;] == 1]<br>virginica = iris[iris[&#39;target&#39;] == 2]</pre><pre>#plotting all species of iris<br>plt.figure(figsize=(18,10))<br>plt.subplot(1,2,1)<br>_ = plt.hist(setosa[&#39;petal length (cm)&#39;], color = &#39;red&#39;,alpha = 0.4,label = &#39;setosa&#39;)<br>_ = plt.hist(versicolor[&#39;petal length (cm)&#39;], color = &#39;blue&#39;,alpha = 0.4,label = &#39;versicolor&#39;)<br>_ = plt.hist(virginica[&#39;petal length (cm)&#39;], color = &#39;green&#39;,alpha = 0.4,label = &#39;virginica&#39;)<br>plt.xlabel(&#39;random values&#39;)<br>plt.ylabel(&#39;count&#39;)<br>plt.title(r&#39;$\bf{Histogram \ with \  all \ species}$&#39;)</pre><pre>plt.subplot(1,2,2)<br>d_set, p_set = empirical_cdf(setosa[&#39;petal length (cm)&#39;])<br>d_ver, p_ver = empirical_cdf(versicolor[&#39;petal length (cm)&#39;])<br>d_vir, p_vir = empirical_cdf(virginica[&#39;petal length (cm)&#39;])<br>plt.plot(d_set,p_set, color = &#39;red&#39;, label = &#39;setosa&#39;)<br>plt.plot(d_ver,p_ver, color = &#39;blue&#39;, label = &#39;versicolor&#39;)<br>plt.plot(d_vir,p_vir, color = &#39;green&#39;, label = &#39;virginica&#39;)<br>plt.xlabel(&#39;random values&#39;)<br>plt.ylabel(&#39;percentiles&#39;)<br>plt.title(r&#39;$\bf{ECDF \ with \  all \ species}$&#39;)<br>plt.legend(bbox_to_anchor=(1, 1))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Sj6lYFB1dt_5liZXI-lig.png" /></figure><h3>How is ECDF different from CDF?</h3><p>CDF is a theoretical construct, while the <a href="https://dictionary.cambridge.org/dictionary/english/empirical">empirical</a> CDF is drawn from real data. I found few really good examples of ecdf vs cdf on StackOverflow, I will leave the <a href="https://stats.stackexchange.com/questions/239937/empirical-cdf-vs-cdf">link</a> in case you are interested to know more.</p><h3>Summary</h3><p>I hope this blog post has helped you to understand ecdf and histogram better. Please comment, If you have any constructive feedback.</p><p><a href="https://thenounproject.com/term/shield-and-sabres/966696/">The Noun Project</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85deda4129ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/convergeml/why-ecdf-is-better-than-a-histogram-85deda4129ed">Why ECDF is better than a Histogram</a> was originally published in <a href="https://medium.com/convergeml">convergeML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rapid Prototyping in Machine Learning.]]></title>
            <link>https://medium.com/convergeml/rapid-prototyping-in-machine-learning-ffe79f023aec?source=rss----42cc29e4dc68---4</link>
            <guid isPermaLink="false">https://medium.com/p/ffe79f023aec</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[rapid-prototyping]]></category>
            <dc:creator><![CDATA[Shoaibkhanz]]></dc:creator>
            <pubDate>Fri, 18 Jan 2019 21:22:38 GMT</pubDate>
            <atom:updated>2019-01-19T06:09:41.980Z</atom:updated>
            <content:encoded><![CDATA[<h3>Rapid Prototyping in Machine Learning</h3><p>Adoption of Machine Learning is rapidly growing among small and big companies. The tools are available, people are excited about them. I imagine we are ready to conquer the world <strong>But are we?…</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*udlLFM2tO-DUEWg52SEkew.jpeg" /></figure><p>In this blog post, we will discuss the following,</p><ul><li><em>How to select the right use-case for a machine learning solution?</em></li><li><em>why do we need Rapid-Prototyping in machine learning?</em></li></ul><h3>Choosing the right use-case</h3><h4><strong>| Pandora Box of use-cases</strong></h4><p>Don’t waste your time on Challenges</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FTFiCaiqWB8DI20OGew%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FTFiCaiqWB8DI20OGew%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FTFiCaiqWB8DI20OGew%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="243" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e8334df1bc4212be65b137839f4d975/href">https://medium.com/media/4e8334df1bc4212be65b137839f4d975/href</a></iframe><p>Most times even though we have every resource available at our disposal, we might not have a use-case worth solving. i.e. how do you choose a use-case that really converts to some tangible benefits?</p><blockquote><strong>With Many use-cases comes extra complexity.</strong></blockquote><p>It&#39;s great to have many use-cases to build solutions for, but it doesn&#39;t translate well in reality. In fact, solve an <strong><em>easy/simple</em></strong> use-case first, they will provide results sooner, don&#39;t get attracted towards a complex beast, their time will come.</p><h3><em>Rapid-Prototyping in machine learning</em></h3><p><strong>| Once your target is locked, what&#39;s your strategy?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbmQyQx2Yz-FJz5XDe8Vpg.jpeg" /><figcaption><strong>Strategy is needed!</strong></figcaption></figure><p>Imagine spending a long time solving a use case and finally cracking it. In the time we have been building a solution, the priority has changed; the use-case isn&#39;t important anymore for the stakeholders, or worse something happened in the market you just didn&#39;t notice or you were too far into the project to make any significant changes. <strong>That’s terrible!</strong></p><p><strong>| Rapid-Prototyping as a solution</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VGuzDMPd5rWamB4cFER7A.jpeg" /></figure><p>At the company where I work, we develop a deployable machine learning solution in 5 days. <strong>We call those 5 days as Spikes</strong>. A Data scientist involved in a spike is not allowed to engage in emails or meetings, what they are allowed to do is code, discuss solutions, brainstorm, eat pizza and sip chai/coffee…</p><p><strong>“We tell our stakeholders, we are building a hut before we build a castle.”</strong></p><blockquote><strong>note:</strong> Developing Machine Learning solutions from scratch differ greatly from Kaggle like competitions (more on it on another post, another day)</blockquote><h3><strong>| 5 days of Spike</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/138/1*Wi38HGXxsd3a6paQF_XUJg.png" /></figure><h3><strong>Day0</strong></h3><p>A Spike can have up to 10 people (5 SMEs + 5 Data Scientists) involved in solving 5 different use-cases. However, before we begin with any kind of modeling, almost a week worth of time is spent on collecting the data that could be useful to build a solution. The SME works together with the data scientist to gather the data. The SME plays a very important role as he is closer to the business and the data. i.e. <em>he has better domain knowledge than others.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*BFOiE5s7lkFxNOjQWwqa9g.png" /></figure><h3><strong>Day1</strong></h3><p>This is the day when we are sitting in the lab, away from all the distractions. We begin by looking at the objectives of the use-cases and then we create a quick <strong><em>one-pager </em></strong>to<strong><em> </em></strong>share with everyone. The document explains the following 7 items.</p><p><strong>1| Translation of the business problem into a data problem.</strong></p><p><strong>2| The top 3 questions, the business would like us to answer.</strong></p><p><strong>3| The data collection: product data, customer data etc…</strong></p><p><strong>4| Any interesting new feature that we want to test.</strong></p><p><strong>5| The Algorithms that we plan to test. </strong><em>(Trees, Regression, SVM etc..)</em></p><p><strong>6|</strong> M<strong>odel interpretability is important too. </strong><em>(PDP, LIME, SHAP)</em></p><p><strong>7 | Metrics that we will use to validate our models. </strong><em>(AUC, R2, F1-score)</em></p><p>While we explain the above points to everyone, most times people in the lab have great suggestions to give, thus this brings more collaboration and some great ideas. Following the interaction, we set up our systems, libraries and begin initial <strong><em>exploratory data analysis (EDA)</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*-6pYxaYusXT8vIFeEQygww.png" /></figure><h3><strong>Day2</strong></h3><p>At the end of day1 we have a good idea about our data, so day 2 is spent on <strong><em>feature engineering </em></strong><em>and</em><strong><em> variables selection/dimensionality reduction</em></strong>. A lot of the feature engineering comes from the SME because of his extensive domain knowledge. This process helps in creating some important features that support in building a better model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*FRYpIEQd6SIqLgQuQlEu8Q.png" /></figure><h3><strong>Day3</strong></h3><p>By the 3rd day, we have a working model, but it might have ~100 features, i.e because we started with a dataset that had ~500–900 features. Day3 is spent on further <strong>reducing the dimensionality </strong>and on <strong>model interpretability</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*7_JqU_0fwEM9aAXQYQzB7w.png" /></figure><h3><strong>Day4</strong></h3><p>It&#39;s model graduation day because we <strong>summarise </strong>all our findings, model performance and its interpretability in a document on this day, we also explore a bit more to see if we can get any other potential insight from the data that stakeholders never asked for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/124/1*QJPex6gMOO4PDlSLSiINKg.png" /></figure><h3><strong>Day5</strong></h3><p>This is the final day of the Spike, where we go back to our stakeholders to explain them everything we have been building for the last 4 days. We get a lot of questions, which is a good sign! If the stakeholders are confident about the model, then we make plans to deploy and test it, with a plan to improve it further with new data.</p><h4>We built the hut we promised, but a lot needs to be done…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXqBmrHDEYp-g3NZaxhr_w.jpeg" /></figure><h3>Summary</h3><p>The Rapid Prototyping helps in creating a minimum viable product(MVP) in a short time. If we and the users like it, we iterate and build a better one. Thus this way you keep improving without losing sight of the market and everyone has the right expectation.</p><p>I hope you like the blog, please comment if you have any constructive feedback.</p><ul><li><a href="https://thenounproject.com/linseedstudio/collection/dog-eared-sheets/?i=627528">Noun Project</a></li><li><a href="https://thenounproject.com/term/users/76078/">The Noun Project</a></li><li><a href="https://thenounproject.com/search/?q=final%20day&amp;i=92669">Noun Project</a></li><li><a href="https://thenounproject.com/term/company-skype-meeting/580239/">The Noun Project</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffe79f023aec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/convergeml/rapid-prototyping-in-machine-learning-ffe79f023aec">Rapid Prototyping in Machine Learning.</a> was originally published in <a href="https://medium.com/convergeml">convergeML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conundrums of the Confusion Matrix]]></title>
            <link>https://medium.com/convergeml/conundrums-of-the-confusion-matrix-2fa82293707a?source=rss----42cc29e4dc68---4</link>
            <guid isPermaLink="false">https://medium.com/p/2fa82293707a</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Shoaibkhanz]]></dc:creator>
            <pubDate>Sun, 13 Jan 2019 14:51:18 GMT</pubDate>
            <atom:updated>2019-01-16T19:59:40.038Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine a problem where we want to predict rain since it’s always cloudy, not how much rain, but will it rain or not. The first task for us is to collect the data, clean it, encode <strong>rain = 1</strong> and <strong>no rain= 0</strong> and make it ready for statistical modeling.</p><p><strong>But wait! How do we know our model is better than any other model?</strong></p><p>This blog post will explain the metrics used to test the robustness of the model but before we discuss them, we will dive into the most basic components of a confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*EL2E-7Wm0-j1b5VmId7pqA.png" /></figure><p>In the first iteration, we perhaps want to build a base model, a logistic regression model is a good choice. The model classifies each day into 1 or 0 i.e. (rain or no rain), but remember our logistic regression model will give us probabilities of 1 and 0, e.g. for a particular day, it might predict rain with <strong>90%</strong> certainty and no rain with<strong> 10%</strong>. It’s upon us to decide which percentage threshold we will consider as <strong>1.</strong></p><blockquote>We may classify a day as a rainy day whenever our model is at least 50% confident of predicting rain.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78V1PFGlLnYaPRjeJogftA.png" /><figcaption>In this example 50% threshold is arbitrary, it depends on the problem</figcaption></figure><p>We now have a foundation to discuss the <strong>confusion matrix, </strong>but before we do, please find its definition from Wikipedia which I have further simplified.</p><blockquote>In classification problems, a <strong>confusion matrix</strong>, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm.</blockquote><p>Most commonly, the <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">ROC curve</a> is built as a visualization to understand the performance of the model, but in this post, we will not discuss the ROC, but the metrics derived from the confusion matrix, which allows us to get those visualizations. <em>“Post on ROC curve is reserved for another day.”</em></p><p>The terminology you will find most often used for confusion matrix has 2 components. i.e. positive class and negative class, because we are interested in predicting rain thus, <strong>rain = positive</strong> <strong>class</strong> and <strong>no rain= negative class</strong>. From these 2 components, we derive 4 terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*9AxFA74C3K6iTlI3QVnOMA.png" /><figcaption>TP = True Positives, FN = False Negatives, TN = True Negatives, FP = False Positives</figcaption></figure><p>To understand the confusion matrix, It’s extremely important to understand the 4 components. Let’s see how these 4 components will be evaluated for rain prediction.</p><blockquote><strong>note: </strong>please read, <strong>^</strong> = prediction, i.e. <strong>P(predicted value|actual value)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*mNrcA3TkWX_Z_HadU9meqA.png" /><figcaption>TP = True Positive</figcaption></figure><p>When our model is at least 50% confident of rainfall, we will predict <strong>1</strong> (rain) as the outcome and in fact, if it turns its true, then it&#39;s a true-positive prediction.</p><blockquote>When we predict the positive class correctly, it’s called <strong>True Positive</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/1*ZkQJfWY7yOfWS562fzy8RA.png" /><figcaption>TN = True Negative</figcaption></figure><p>Imagine the negative case, when our model is at least 50% confident of no rain, we predict <strong>0</strong>(no rain) as the outcome and in fact, if it is true, then it’s a true negative prediction.</p><blockquote>When we predict the negative class correctly, it’s called <strong>True Negative</strong>.</blockquote><h3><strong>Things get interesting for the next 2 types of predictions. What would happen if we are wrong?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*zXdJAD8rf8pxzLAmb2okrg.png" /><figcaption>FP = False Positive</figcaption></figure><p>Imagine, our model predicts it will rain, but to our surprise, no sign of rain. i.e. our model got it wrong. It predicted the positive class (rain) however, the correct value was the negative class, i.e. it was a day with no rain.</p><blockquote>When we predict the positive class incorrectly, it’s called <strong><em>False Positive.</em></strong></blockquote><h4>Next is the 4th component of the confusion matrix.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*nnuHSjPyiPSbYGMaFm84Lw.png" /><figcaption>FN = False Negative</figcaption></figure><p>Compare the image for false positive and false negative, you will find events are rearranged, In this scenario, our model predicted the negative class (no rain) but the correct value was a positive class.</p><blockquote>When we predict the negative class incorrectly, It’s called <strong><em>False Negative.</em></strong></blockquote><p><strong>Below picture is the best example of a False outcome (wrong prediction). </strong><em>(Please ignore if the patient in the below picture looks like </em><a href="https://en.wikipedia.org/wiki/Neil_deGrasse_Tyson"><em>Neil deGrasse Tyson</em></a><em>)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*DHHCDARetAdFC1bjErHVGg.png" /><figcaption>Illustration of False Positive and False Negative</figcaption></figure><h3><strong>Deriving metrics from Confusion Matrix</strong></h3><blockquote><strong>Remember this: </strong>Positive Class(P) = TP + FN and Negative Class(N) = TN + FP</blockquote><p>Let’s predict rain for 50 days and then arrange our predictions against their correct values in a square matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_x6uYNhMD_hXKyeHVG3r-A.png" /><figcaption>Confusion Matrix</figcaption></figure><p><strong>1|TPR (True Positive Rate)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/1*cFJyDeNeTzY7yCv6qob9Uw.png" /><figcaption>TPR = True Positive Rate</figcaption></figure><p>The <strong><em>True Positive rate </em></strong><em>is given by </em><strong>TPR = TP/P</strong><em>, i.e. True positive values over all the positive values.</em><strong> </strong>Remember, <strong>P= TP+FN.</strong></p><p>Thus, <strong>TPR = TP/(TP+FN). </strong>This metric is also known as<em> Sensitivity</em> in statistics or <em>Recall</em> in machine learning.</p><p>This metric helps us understand how well we are predicting our positive case. In our example <strong>TPR = 20/(20+5) = 0.8, </strong>i.e. our model correctly predicts rain <strong>80% </strong>of the time, when it rained.</p><p><strong>2|TNR (True Negative Rate)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*BVhu6jLlKTEG9DbODuzjLg.png" /><figcaption>TNR = True Negative Rate</figcaption></figure><p>The <strong><em>True Negative rate </em></strong><em>is given by </em><strong>TNR = TN/N.</strong></p><p><em>i.e. True negative values over all the negative values.</em></p><p>Remember, <strong>N= TN+FP.</strong></p><p>Thus, <strong>TNR = TN/(TN+FP). </strong>This metric is also known as <em>Specificity</em> in statistics.</p><p>This metric helps us understand how well we are predicting our negative case. In our example <strong>TNR = 15/(15+10) = 0.6, </strong>i.e. our model correctly predicts no rain <strong>60% </strong>of the time, when there was no rain.</p><blockquote><strong>quick revision:</strong> TPR = Sensitivity/Recall, TNR = Specificity</blockquote><p><strong>3 |False Positive Rate</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*k8_xANF0tcjiLQIs0U2vgw.png" /><figcaption>FPR = False Positive Rate</figcaption></figure><p>We might be interested to know, how many times the model predicted the positive class or rain but there was no rain for that day.</p><p>It’s defined as <strong>FP/N, </strong>i.e.<strong> </strong><em>False positive values over all the negative values.</em></p><p>If we use the fact that <strong>N = TN + FP </strong>then,<strong> FPR = FP/(TN+FP)</strong>, In our example, <strong>FPR = 10/(15+10) = 0.4. </strong>i.e our model incorrectly predicted rain <strong>40% </strong>of the time, when there was no rain.</p><p><strong>4 |Accuracy</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*SEetdmoeSciklUr72XfEow.png" /><figcaption>accuracy = TPR + FNR</figcaption></figure><p>When we want to know how well our model predicts the positive class <strong>&amp; </strong>the negative class, we turn to accuracy. It’s defined as <strong>(TP + TN)/(P + N).</strong></p><p>In the above example, our model’s accuracy is <strong>(20 + 15)/(20+5+15+10) = 0.7, </strong>i.e. our model correctly predicts the rain and no rain <strong>70% </strong>of the time.</p><p><strong>5 |Error Rate</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*0zjIFIed_WXU1boDxVTMTw.png" /><figcaption>error rate or misclassification rate = FPR + FNR</figcaption></figure><p>The Error rate is also called the <em>misclassification rate</em>. It’s the opposite of <em>accuracy</em> or if represented mathematically the <strong>error rate = (1- accuracy) = (FP + FN)/(P+N).</strong></p><p>The Error rate for our rain example is <strong>(1 – 0.7) = 0.3, </strong>i.e. we 30% percent of the time we didn&#39;t get the correct prediction. I will not define the <strong>FNR (False Negative Rate)</strong> because I believe if you are new to the confusion matrix you can easily get confused, this post already has 6 metrics. However, I believe after learning from this post you will figure it out yourself.</p><p><strong>6 |PPV (Positive Predicted Value)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/1*-PELXLwbgFN0H1ZYmCp15Q.png" /><figcaption>PPV = Precision, P(actual value|predicted value)</figcaption></figure><p>When we want to know, how many correct predictions we made for the positive class from all the predictions we made for the positive class, then we know we are looking for precision in our model’s prediction.</p><p>Hence, the <strong>Positive Prediction Value</strong> is also called <em>Precision</em> in machine learning.</p><p>It is defined as <strong>TP/(TP+FP).</strong> For the example above, our model’s <strong>PPV or Precision = 20/(20+10) = 0.66…, </strong>i.e. our model predicted rain correctly <strong>66 %</strong> of the time whenever it predicted rain.</p><h3><strong>Conclusion</strong></h3><p>TP, TN, FP, FN are the building blocks of all the classification metrics that are most commonly <a href="https://en.wikipedia.org/wiki/Confusion_matrix">used</a>. It’s mostly the <em>False</em> in the <em>positive</em> or <em>True</em> in the <em>Negative</em> etc. that cause confusion if you read them or the first time, so don&#39;t be disheartened if you struggle a bit, I hope this blog post helped you to understand the confusion matrix.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fa82293707a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/convergeml/conundrums-of-the-confusion-matrix-2fa82293707a">Conundrums of the Confusion Matrix</a> was originally published in <a href="https://medium.com/convergeml">convergeML</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>