<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prasan Cumar N on Medium]]></title>
        <description><![CDATA[Stories by Prasan Cumar N on Medium]]></description>
        <link>https://medium.com/@prasancumarn?source=rss-4ad743ca8d05------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*IwTnXQLzhLKGZMFZ.</url>
            <title>Stories by Prasan Cumar N on Medium</title>
            <link>https://medium.com/@prasancumarn?source=rss-4ad743ca8d05------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:40:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prasancumarn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Estimating model performance in production]]></title>
            <link>https://medium.com/@prasancumarn/estimating-model-performance-in-production-37c22cf0a4bd?source=rss-4ad743ca8d05------2</link>
            <guid isPermaLink="false">https://medium.com/p/37c22cf0a4bd</guid>
            <category><![CDATA[clt]]></category>
            <category><![CDATA[bootstrapping]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[model]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Prasan Cumar N]]></dc:creator>
            <pubDate>Wed, 10 Jul 2019 16:25:38 GMT</pubDate>
            <atom:updated>2019-07-10T16:44:27.201Z</atom:updated>
            <content:encoded><![CDATA[<h3>Estimating model performance in Production</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*1EXBfl-yvcxQzaS8rE_27A.jpeg" /></figure><p>Once a model has been tested satisfactorily by making sure that the model meets the required accuracy level then the model will be dropped in production environment. After moving it to production if the model misbehaves imagine what would happen? The misbehaving would be due to several reasons like sample data not being a true representative of the real data or model being overfit. Thus a crucial task of data scientist would be statistically assessing models likely performance in production and this has to be done with some degree of confidence.</p><p>Since the model is yet to face the production data, it will be risky to give point estimates such as “<strong>the model accuracy will be 98% in the production</strong>” . Instead, it expressed as a range estimate, since a range is specified, that indicates some degree of uncertainty. Hence, the range has to be specified with a degree of confidence for e.g. “the true performance of the model in production data is likely to be in the given range 95% of the time”.</p><p>To estimate a range with some confidence, the model has to be executed multiple times on different data sets and that is a practical challenge as we may not always have access to such volumes of data. We have to do with what we possess. This is where the bootstrap sampling becomes handy. Bootstrap sampling involves taking ‘m’ random samples from the given data of size ‘m’ observations. Each ‘m’ random samples can be used to test the model on. If I have access to 1000 data points, I create say, 50 sets of data of size 1000 using random sampling with replacement (bootstrap sampling). Each set will have a combination of unique and redundant observations.</p><p>To estimate the performance range with 95% confidence, we use statistical approach given below –</p><blockquote>Run this model large number of times on bootstrapped data (sampling with replacement) and collect the accuracy measures for each run and plot the histogram reflecting the frequency with which the various accuracy levels were observed across the sample data sets, the shape of distribution will tend to become normal when the number of data sets created tends to infinity.</blockquote><p>Let us represent the distribution mean as M and standard deviation as Std). The “M” is the average performance in the test runs with a standard deviation (average difference of Std). We are satisfied with the accuracy we have in the test environment.</p><blockquote><strong><em>Note:</em></strong> The “M” is an indicator of accuracy in the real world. The more the testing we do, the more performance scores we collect in the test to find the “M”, the closer the “M” gets to the real-world performance. This concept in statistics is known as the Central Limit Theorem (CLT).</blockquote><p>Assume we executed our model on 30 different data sets created from the original sample data through bootstrap method and got the following accuracy scores (X).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*TfZPjRVDRBkarrlHKzQg_A.png" /><figcaption>Fig — 1 model performance data on 30 sample datasets</figcaption></figure><p>As per the CLT given a distribution of performance scores, the distribution tends to acquire properties of a normal curve with increase in number of sample sets (we have 30 as of now) and the performance score distribution looks like a perfect bell curve</p><p>Imagine we had access to the true normal distribution of the scores in the population data. Suppose the normal distribution represented on Z scores was centered at Mnorm (Z = 0), with standard deviation Stdnorm. 95% confidence would be represented by the area between Z score of 1.96 Standard Deviation on either side of the mean</p><p>In this case we could have anchored our range estimates on Mu and expressed the range as</p><blockquote>Equation 1.0 →</blockquote><blockquote>[Mnorm- (1.96*Stdnorm) ]- -[Mnorm+(1.96*Stdnorm)] @ 95% confidence</blockquote><blockquote>Equation 2.0 →</blockquote><blockquote>{[M- (1.9*Stdnorm)] &lt;= Mnorm &lt;= [M + (1.96*Stdnorm)]}</blockquote><blockquote>Equation 3.0 →</blockquote><blockquote>{[M — Tc (Std / sqrt(n))] &lt;=Mnorm &lt;= [M + Tc (Std / Sqrt(n)]}</blockquote><ol><li>M - Scores distribution mean</li><li>2. Tc - the T distribution threshold that gives 95% confidence (assuming we wish to be 95% confidence). Equivalent of 1.96 in normal distribution</li><li>n- the number of times we collected the performance scores</li><li>Mnorm - the elusive mean performance in the population / universe which we are unable to predict and hence satisfy by giving a range within which it lies.</li><li>Std - Standard distribution of the scores</li><li>Std/sqrt(n) - Standard error</li></ol><p>So, from the Fig -1 (xls sheet screenshot), on substituting the values into the equation 3.0, the range comes to</p><blockquote>79.8 — (2.045 * 1.34) &lt;= Mnorm &lt;= 79.8 + (2.045 * 1.34)</blockquote><blockquote>79.8–2.7403 &lt;= Mnorm &lt;= 79.8 + 2.7403</blockquote><blockquote>77.05 &lt;= Mnorm &lt;= 82.54</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*GPnSy_Jd20P5xIOiNbRcOA.png" /></figure><blockquote>Thus, when we put this model in production, we are <strong>95% confident</strong> the true performance (accuracy) will lie in the range of <strong>77.05</strong> to <strong>82.54</strong>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37c22cf0a4bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>