<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[dsnaiplusui - Medium]]></title>
        <description><![CDATA[Official publications of DSN AI+ UI, a Data Science Nigeria community in the University of Ibadan - Medium]]></description>
        <link>https://medium.com/dsnaiplusui?source=rss----311bc712981e---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>dsnaiplusui - Medium</title>
            <link>https://medium.com/dsnaiplusui?source=rss----311bc712981e---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:46:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dsnaiplusui" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When your data is not normal: A quick introduction to non-parametric statistical methods]]></title>
            <link>https://medium.com/dsnaiplusui/when-your-data-is-not-normal-a-quick-introduction-to-non-parametric-statistical-methods-75762b7f85e5?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/75762b7f85e5</guid>
            <dc:creator><![CDATA[Adeyinka Oresanya]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 08:47:22 GMT</pubDate>
            <atom:updated>2020-07-07T08:46:33.740Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*x8HyKgT6mjY5lU_S.png" /></figure><p><em>Welcome to day 7 of Seven Days of Statistics organised by </em><a href="https://twitter.com/dsn_ui"><em>AI+ Club, University of Ibadan</em></a><em>. You can find the other parts of the series </em><a href="https://medium.com/dsnaiplusui"><em>here</em></a><em>.</em></p><p>Statisticians and data scientists love to work with data that conform to a number of assumptions regarding the nature of population from which the sample data was drawn, for instance, the level of measurement, sample size, and distribution of the dataset. This is because they can exploit the entire collection of statistical methods developed for data assuming a normal distribution (Pearson Correlation coefficient, ANOVA, linear regression). These methods are called parametric statistics because they focus on specific parameters of the population, usually, the mean and variance.</p><p>However, in the real world, not all data come with a known or identifiable distribution. Data may contain outliers, multiple peaks, ordinal variables and some other forms that can make data not possess an easily identifiable shape. A quick example is the salary distribution of workers which tend to have a right-skewed distribution. Does that mean we can’t conduct accurate analysis on this type of data? No, there is always a way out!</p><p>While data scientists can go as far as using data preprocessing methods to conform data with an unknown distribution into having a known distribution, especially a normal one, so that they can utilise the well-understood parametric statistical methods, there are alternative methods designed to work well with this type of data.</p><p>Enter non-parametric statistics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*VD73XR8X9ZBj51G4.png" /><figcaption>Skewed-distribution usually require non-parametric statistical methods</figcaption></figure><p>Non-parametric statistical methods are techniques that don’t follow the same ‘rules’ as parametric methods, that is, they do not require the strict assumptions made about the population from which the sample data was drawn (although, this does not imply that they are assumption-free). They are also referred to as ‘distribution-free’ statistics.</p><h3>Types of Non-parametric Statistical methods</h3><p>In elementary statistics, Chi-square test is usually the main non-parametric method discussed, however there are several others, most of which are equivalent of parametric methods applied when data do not conform to the assumptions required by parametric methods. Examples are:</p><p><strong>1</strong>. <strong>Kruskal-Wallis Test</strong>: This is a non-parametric alternative to one-way ANOVA, used to find out if the median of two or more groups are different. For this test, ranks of the data points are used for the estimation, rather than the data points themselves.</p><p><strong>2</strong>. <strong>Mann-Whitney U Test</strong>: This is the non-parametric alternative to the two-sample t-test. It is used to compare differences between two independent groups when the dependent variable is either ordinal or continuous.</p><p><strong>3. Spearman’s Rank-Order Correlation: </strong>This is the non-parametric version of the Pearson Product-Moment Correlation. The Spearman’s correlation coefficient measures the strength and direction of association between two ordinal/ranked variables.</p><p><strong>4. Goodman and Kruskal’s Gamma:</strong> This is also used to determine the strength and direction of association that exist between two ordinal variables measured on an ordinal scale. However, it is more appropriate when the data contain many tied ranks.</p><p><strong>5.</strong> <strong>Chi-Square Test: </strong>This is used to determine whether there is an association between two categorical variables.</p><p><strong>6. One-Sample Sign Test</strong>: This isused to estimate the median of a population which is then compared to a hypothesized value k.</p><p><strong>7.</strong> <strong>Mood’s Median Test</strong>: This is used to test the equality of medians from two or more populations. It is used instead of one-sample sign test when there two independent samples.</p><p><strong>8.</strong> <strong>Friedman Test</strong>: This is the non-parametric alternative to the one-way ANOVA with repeated measures, used to test for differences between groups when the dependent variable being measured is ordinal.</p><p><strong>9. Mann-Kendall Trend Test</strong>: This is used to detect trends in a time-series data.</p><h3>When to use non-parametric statistics</h3><p>1. When there is little to no assumption concerning the distribution of the population.</p><p>2. When the distribution of the dependent (target) variable is not normal.</p><p>3. When there are very small sample sizes.</p><p>4. When there is unequal variance across groups.</p><p>5. When the data possess significant outliers.</p><p>6. When the level of measurement is nominal or ordinal and the main focus is on the ranking or frequencies of data.</p><p>7. When hypotheses are stated using ranks, medians, or frequencies of data.</p><p>The ability of non-parametric methods to relax assumptions about the sample data make its application wider in scope than parametric statistics. However, in scenarios where the use of parametric methods is more appropriate, non-parametric methods become less efficient. Thus, it is important for professionals working with data to be aware of the statistical techniques available and know when to apply them.</p><p><em>For further reading on non-parametric methods,see </em><a href="https://www.amazon.com/Practical-Nonparametric-Statistics-3rd-Conover/dp/0471160687"><em>Conover W.J’s Practical Nonparametric Statistics</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75762b7f85e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/when-your-data-is-not-normal-a-quick-introduction-to-non-parametric-statistical-methods-75762b7f85e5">When your data is not normal: A quick introduction to non-parametric statistical methods</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ESTIMATION STATISTICS AT A GLANCE]]></title>
            <link>https://medium.com/dsnaiplusui/estimation-statistics-at-a-glance-ca925b40aff?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/ca925b40aff</guid>
            <category><![CDATA[tolerance-interval]]></category>
            <category><![CDATA[estimation-statistics]]></category>
            <category><![CDATA[meta-analysis]]></category>
            <category><![CDATA[effect-size]]></category>
            <category><![CDATA[prediction-interval]]></category>
            <dc:creator><![CDATA[Ayodele Awokoya]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 10:31:34 GMT</pubDate>
            <atom:updated>2020-07-06T10:31:34.476Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*UNsjZCU_P_8fJEgEU5Y95A.jpeg" /></figure><p>Estimation is a technique in statistics used to quantify the effect of sample data on a population parameter. Estimation statistics is the process or methods for measuring the true value of a population parameter using the sample statistics.</p><p>This article, as part of the #7days_of_stats by DSN AI+ club University of Ibadan, promises to be enlightening as it provides insight into</p><ul><li>Why Estimation Statistics?</li><li>Effect Size</li><li>Interval Estimate and</li><li>Meta Analysis</li></ul><h3>WHY ESTIMATION STATISTICS?</h3><p>In statistics, the probability value called p-value has been used to inform the statistical significance of a hypothesis. This approach has been misinterpreted over the years due to the fact that it only presents the likelihood that an effect exist without commenting on the size of the effect, knowing fully well that it is possible for a result to be statistically significant and trivial and also statistically non-significant and important. Hence, the need to know the effect size of an experiment in order to relate better with the outcome.</p><h4>Types of Estimation Statistics</h4><p>There are three major types of estimation statistics. They are:</p><ol><li>Effect size</li><li>Interval Estimation, and</li><li>Meta Analysis</li></ol><p>Let us begin by exploring each of these estimates in statistics.</p><h3>EFFECT SIZE</h3><p>Effect in statistics is the actual quantification of difference or relationship of the result in an experiment.</p><p>Effect size method, which is also referred to as point estimate, refers to a suite of statistical tools from the field of estimation statistics which helps in quantifying the exact size of the effect in the result of an experiment. Effect sizes generally depends on the assumption that ‘control’ and ‘experimental‘ group values are normally distributed and have the same standard deviations. Effect size methods can therefore be grouped based on the type of effect to be quantifies. The two main groups are:</p><ul><li>Association and</li><li>Difference</li></ul><p>The result of an effect size can be interpreted based on goals of the interpretation and statistical method used. In practice, effect size tells the reader the magnitude with which an event is likely to happen. For example, using the prevalent covid-19 situation, if an experiment is carried out on a newly discovered treatment and the statistical test shows that it has the likelihood of curing the disease, effect size will give us an exact quantifiable measure of how much a dose of the treatment can cure covid-19. In general, knowing the likelihood that a treatment is viable and also the extent of its viability is more insightful.</p><h4>Calculating and Interpreting Effect Size</h4><p>Effect sizes either measure the sizes of associations between variables or the sizes of differences between group means. This measure must be chosen based on the goal of the interpretation. This include:</p><ul><li>Standard Result e.g Cohen’s d calculation</li><li>Unit Free Result e.g Correlation coefficient</li></ul><h4>Cohen’s d</h4><p>This is an appropriate effect size for comparison between two means. It can be used for example, to accompany the reporting of t-test and ANOVA results. It is widely used in meta-analysis. In general the rule of thumb in cohen’s d suggest that d=0.2 is considered ‘small’ effect size, 0.5 ‘medium’ and 0.8 a ‘large’ effect size. This implies that if two groups’ mean differ with 0.2, the difference is considered small, though it is statistically not significant.</p><p>Cohen’s d calculation is not provided in python, but we can do it manually thus:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2F9aL2sLJ25XobH7PTrFFMFC&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2F9aL2sLJ25XobH7PTrFFMFC&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="579" height="463" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5757fd743f494dd45263a752ac64198/href">https://medium.com/media/d5757fd743f494dd45263a752ac64198/href</a></iframe><p>In python, cohen’s d can be defined as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55a9342625309f261d88b2e15580129c/href">https://medium.com/media/55a9342625309f261d88b2e15580129c/href</a></iframe><h4>Pearson r Correlation</h4><p>The association between variables is often referred to as the ‘r family’ of effect size methods. The pearson correlation coefficient measures the degree of linear association between two real-valued variables. The rule of thumb for interpreting pearson’s r is:</p><ul><li>-1.0: perfect negative relationship</li><li>-0.7: strong negative relationship</li><li>-0.5: moderate negative relationship</li><li>-0.3: weak negative relationship</li><li>0.0: No relationship</li><li>0.3: weak positive relationship</li><li>0.5: moderate positive relationship</li><li>0.7: strong positive</li><li>1.0: perfect positive</li></ul><p>The pearson’s correlation coefficient can be calculated in python using the pearsonr () spicy function. For example, quantify the size of association between two samples of random gaussian numbers in python as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/766096607c555070039404865e3f996a/href">https://medium.com/media/766096607c555070039404865e3f996a/href</a></iframe><h3>INTERVAL ESTIMATE</h3><p>This is the use of sample data to calculate a range of possible values of unknown population parameter. An interval estimate is defined by two numbers between which a population parameter is said to lie. For example, a&lt;µ&lt;b is an interval estimate for the population mean µ. It indicates that the population mean is greater than a but lesser than b.</p><p>Interval estimate is the use of sample data to calculate the interval of possible values of an unknown population parameter, which is in contrast to point estimate, a single number. There are 3 types of interval estimation.</p><ul><li>Confidence Interval</li><li>Tolerance Interval</li><li>Prediction Interval</li></ul><h4>Confidence Interval</h4><p>Confidence interval is the range of values in which our true values actually lies (our predicted population parameter). It is a range of values which is used to estimate the population parameter from a sample statistics. Confidence interval, as an estimate interval with a specific level, is the probability that the interval estimate will contain the parameter.</p><p>Confidence interval can be calculated using the formula below</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FsSzFQHXEQzkkMbgQ4ECzBt&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FsSzFQHXEQzkkMbgQ4ECzBt&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="388" height="294" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fc35e318f5818a02ae03aefdee35276/href">https://medium.com/media/6fc35e318f5818a02ae03aefdee35276/href</a></iframe><h4>Prediction Interval</h4><p>Prediction interval is an estimate of an interval in which a future observation will fall within a certain probability given what has already been observed. Prediction intervals are used in regression models to generate prediction based on specific predictor values. Like confidence interval, prediction intervals are upper bound and lower bound. Unlike confidence intervals, prediction intervals predict the spread of individual observations rather than the mean.</p><p>The prediction interval for new response ynew, with predictor’s value xh can be determined using the general formula</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2Fic138sofrJ5gSFpmRKcgAk&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2Fic138sofrJ5gSFpmRKcgAk&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="551" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/cedff964e2da682e635fab47f3a87906/href">https://medium.com/media/cedff964e2da682e635fab47f3a87906/href</a></iframe><p>click <a href="https://online.stat.psu.edu/stat462/node/151/">here</a> for more on how to calculate prediction interval .</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4ab0ddebff196c8b90403f99bec81f2/href">https://medium.com/media/a4ab0ddebff196c8b90403f99bec81f2/href</a></iframe><h4>Tolerance Interval</h4><p>Tolerance interval is a statistical interval within which with some confidence level, a specific proportion of a sample population falls. A tolerance interval defines the upper and /or lower bounds within which a certain percent of the process output falls with a stated confidence. Minimum percentage of the population that the interval should cover and a confidence level is needed to generate the tolerance interval.</p><p>Click <a href="https://www.statisticshowto.com/tolerance-intervals/">here</a> for more on how to calculate tolerance Interval</p><h4>Difference between Confidence, Prediction and Tolerance Intervals.</h4><ol><li>These are distinct approaches to determine the range of an estimate in statistical analysis. Confidence interval gives a range of the likely location of a true population parameter, but as the sample size increases, the interval eventually converges to a single value, the true population parameter.</li><li>Prediction interval informs about the distribution of individual values as opposed to the uncertainty in estimating the population mean and it will not converge to a single value as the sample size increases.</li><li>To compute tolerance interval, it is necessary to specify one percentage that expresses how sure you want to be (confidence level) and another percentage that expresses the fraction of the population the interval will contain (population coverage).</li><li>If the first value (confidence level) is set to 50% then tolerance interval is simple the same as prediction interval. If the confidence level is set to a higher value say(90%-99%) then the tolerance interval is wider than a prediction interval. Tolerance intervals do not converge to a single value as the sample size increases just like prediction interval</li><li>Prediction intervals are always wider than confidence intervals due to the added uncertainty involved in predicting a single response versus the mean response.</li></ol><h3>META-ANALYSIS</h3><p>Meta-Analysis is a quantitative approach used to systematically assess previous research studies in order to derive conclusion about the body of the researches. Meta analysis are useful when many similar studies with conflicting findings are combined into a stronger finding, than any single study, using statistical methods.</p><p>The aim is to combine these independent observations into an average effect size and draw an overall conclusion regarding the direction and magnitude of real-world effects.</p><p>Although, not often used in applied machine learning, it is useful to note meta-analyses as they form part of this trust of new statistical methods.</p><p>Conclusively, estimation statistics, which are majorly divided into two distinct parts called point and interval estimate, can with statistical test give better insight for generalizing about a particular population parameter using sample statistics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca925b40aff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/estimation-statistics-at-a-glance-ca925b40aff">ESTIMATION STATISTICS AT A GLANCE</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LESSON 5 : STATISTICAL HYPOTHESIS TESTS]]></title>
            <link>https://medium.com/dsnaiplusui/objectives-understand-the-basic-concepts-of-statistical-hypothesis-testing-understand-how-to-set-up-5cb3a85a6d69?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/5cb3a85a6d69</guid>
            <dc:creator><![CDATA[Taiwo Olatunde]]></dc:creator>
            <pubDate>Sun, 05 Jul 2020 13:52:47 GMT</pubDate>
            <atom:updated>2020-07-05T13:52:47.876Z</atom:updated>
            <content:encoded><![CDATA[<p>LESSON 5 : STATISTICAL HYPOTHESIS TESTS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*jGEFWP9fIzqB1brIJsFICA.jpeg" /></figure><p>INTRODUCTION<br>Statistical hypothesis testing is an intrinsic tool for making inference about a defined population in Statistics. It entails drawing conclusion from sample data based on statistical statements made about the sample data. This statement or claim is called Statistical hypothesis.</p><p>Listed as lesson 5 of the 7days of Statistics challenge from the desk of <a href="https://medium.com/u/695170cdb34d">DSN AIPlus UI</a> below are the objectives of the lesson</p><p>OBJECTIVES</p><ul><li>Understand the basic concepts of statistical hypothesis testing.</li><li>Understand how to set up an hypothesis from a problem statement.</li><li>Understand the concept of p-value in hypothesis testing.</li><li>Understand the concept of Type 1 and Type 2 error.</li></ul><p>TYPES OF HYPOTHESIS</p><ul><li>Null hypothesis – as the name implies ‘NULL’, this is the statement of no difference between the hypothesised value and the observed value. It renders a statistical hypothesis insignificant and it is symbolically denoted as Ho.</li><li>Alternative hypothesis – this is the hypothesis contrary to the null hypothesis. The alternative hypothesis renders a statistical hypothesis significant, which means, it assumes that the hypothesised value significantly deviates from the observed value. It is symbolically denoted as H1.</li></ul><p>TYPES OF ERROR</p><p>Errors are likely made when drawing conclusions from statistical hypothesis testing, the errors are of two types namely;</p><ul><li>Type 1 error - this type of error is committed when when a true null hypothesis is rejected in favour of the false alternative hypothesis, meaning accepting H1 is true when actually Ho is true.</li><li>Type 2 error – this type of error is committed when a false null hypothesis is accept in denial of a true alternative hypothesis, meaning accepting Ho is true when actually H1 is true.</li></ul><p>P-VALUE<br>p-value is the probability of getting the sample value while Ho is true, it a probability value ranging from 0 to 1 which measures how strong or how weak the observed value will be gotten assuming that Ho is true. This implies that the higher the p-value, the stronger the chances of getting observed value and vice versa. P-value is compared directly with the stated level of significance which is mosly taken as 5%(0.05).</p><h3>Below is a real life example to determine if the mean year a university student admitted in year 2016 will graduate is the same irrespective of year of study and extra year(ranging from 1 to 3) at 5percent level of significance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*CjLeadpsYoL54HG4Q1UeQQ.jpeg" /></figure><p>From above, comparing the p-value directly with the level of significance (0.05). p-value &gt; 0.05. It is safe to accept the null hypothesis and conclude that the statistical hypothesis is insignificant.</p><p>TASK</p><ul><li>Think of a real life problem statement,simulate random numbers with respect to the problem statement, run statistical hypothesis test and explain your conclusion.</li><li>Read more about sided tests.</li><li>Read further about statistical hypothesis testing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cb3a85a6d69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/objectives-understand-the-basic-concepts-of-statistical-hypothesis-testing-understand-how-to-set-up-5cb3a85a6d69">LESSON 5 : STATISTICAL HYPOTHESIS TESTS</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lesson 04: Correlation between Variables.]]></title>
            <link>https://medium.com/dsnaiplusui/lesson-04-correlation-between-variables-d60b0a7cb04a?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/d60b0a7cb04a</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[correlation]]></category>
            <category><![CDATA[eda]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <dc:creator><![CDATA[Isaac Adegbayibi]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 09:53:00 GMT</pubDate>
            <atom:updated>2020-07-04T09:53:00.234Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Lesson 04: Correlation between Variables.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n4F0C56zMzdqrKf8kBoKyg.jpeg" /><figcaption>Image from investopedia.com</figcaption></figure><p>In a study, Exploratory Data Analysis is performed and an interest in studying the relationships of some variables involved in the study arises! Then, a very important question comes for an answer: Can one statistic measure both the strength and direction of a linear relationship between two variables? Sure! In Statistics, we use the <em>coefficient of correlation </em>to measure the strength and direction of the linear relationship between two numerical variables, say, <em>X </em>and <em>Y</em>. The correlation coefficient for a sample of data is denoted by <em>r.</em></p><p>In this lesson, the concept of correlation will be looked at, starting from its fundamental theory to its application in machine learning.</p><p><strong>Nexus between Correlation and Machine Learning.</strong></p><p>It is no longer news that AI+Club, University of Ibadan is training members on Machine Learning to use the Jupyter Notebook! Correlation in Statistics is important to further complete our understanding of the model built through Machine Learning.</p><p>Data correlation is how one set of data may correspond to another set. In Machine Learning, think of how your features correspond with your output.</p><p>For example, the image below visualizes a dataset of brain size versus body size. Notice that as the body size increases, so does the brain size. This is known as a linear correlation. With a simple definition of linear correlation, the data follows a straight line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Q-G7TBgR7AhkDYRq" /><figcaption>Figure 1: Brain Weight Vs. Body Weight.</figcaption></figure><p><strong>Theory.</strong></p><p>Correlation describes the strength of the linear relationship between two variables. A numerical value, called the <em>coefficient of correlation</em> which always takes a value between -1 and +1, tells us about the relationship between the two variables.</p><p>Correlation between two variables can be in three instances. We can get a scatter plot from our EDA on the two variables to show us the general direction and nature of the relationship between the two variables. The three instances that can be observed are:</p><ol><li>Positive Correlation: Both variables change in the same direction. This implies that for every increase in the independent variable, there is a corresponding increase in the value of the dependent variable.</li><li>Neutral/Zero Correlation: No relationship in the change of the variables. Here, there is no relationship between the two variables, they are independent of one another. Also, it can be said that the independent variable cannot explain or predict the dependent variable in the regression model.</li><li>Negative Correlation: For this instance, it tells us that there is an inverse proportional relationship between the variables. This implies that as the values in one variable (independent or dependent) increases, there is a decrease in the other variable.</li></ol><p>The coefficient of correlation for a simple linear regression model (Y = a +bX) is usually denoted by <em>r </em>and we read its values as thus:</p><ol><li>The relationship is perfectly linear when R is either -1 or +1.</li><li>If the relationship is strong and positive, R will be near +1 (close to +1).</li><li>If the relationship is strong and negative, R will be close to -1.</li><li>If there is no apparent linear relationship, R will be close to 0.</li></ol><p><strong>Illustration.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*SWXL0aSJEgMrpzfm.png" /><figcaption>Figure 1: Image showing correlation instances between two variables, say, Y and X.</figcaption></figure><p>For simplicity, let’s consider the image above. Two variables are under study in each of the six sub-images above and all six sub-images are obtained from the EDA of the variables.</p><ul><li>Starting from the top left, it is observed that there is a strong, positive correlation between the two variables as <em>r </em>= 0.7. For the bottom left, that scenario is the inverse of the former as there exists a strong, negative correlation between the variables as <em>r</em> =-0.7.</li><li>For the centreimages, the top centreshows us a weak, positive correlation, close to zero with <em>r</em> = 0.3. The bottom centre is the reciprocal of that, showing weak, negative correlation, <em>r =</em> -0.3.</li><li>The sub-images to the right are obviously the same! Both are showing us when the two variables aren’t related. This is Zero/Neutral Correlation with <em>r = </em>0.</li></ul><p>That’s it for the illustration!</p><p><strong>Computation.</strong></p><p>An easy, well-used approach to compute the mathematical values of the <em>coefficient of correlation</em> is <em>Pearson’s product-moment coefficient, </em>defined below<em>:</em></p><p>The population correlation coefficient,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/36/0*HG7KE5xamo8Cure8" /></figure><p>between two random variables</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/16/0*-BJ2rWXi-FomiO_w" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/14/0*YsPpl-sP2iNUnBfd" /></figure><p>with expected values</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/24/0*Dc5OVzkUGXhZYpoY" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/23/0*YEDWbaoQCCde2WnA" /></figure><p>and standard deviations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/24/0*snNSUxH6psuCxsEb" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/23/0*tObLkdjzh6faFv3B" /></figure><p>is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*5V8ug-U1yuWftr91" /></figure><p>where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/13/0*IgAgNWkXEDDAirYr" /></figure><p>is the expected value operator,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/27/0*DWwtpJoegWwq_oEY" /></figure><p>means covariance, and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/32/0*I27yJUpV0KoKEvTh" /></figure><p>It is a widely used alternative notation for the coefficient of correlation. The Pearson correlation is defined only if both standard deviations are finite and positive.</p><p>We do not need to memorize the formulas here as some lines of code in our console can whip up the values in a very short runtime.</p><p><strong>Multicollinearity.</strong></p><p>Most real-life scenarios would have multiple independent variables trying to explain or predict a dependent variable.</p><p>The technique used for the estimation that is derived from the<em> Simple Linear Regression </em>is <em>Multiple Regression</em>.</p><p>But, in Multiple Regression, there is always a problem of the independent variables being correlated and even showing a stronger relationship between themselves than they should with the dependent variable, this problem is known as Multicollinearity.</p><p>A proper definition: Multicollinearity is a situation when there violates a major assumption of the classical linear regression model which holds that there is no intercorrelation between two or more independent variables. This leads to negative consequences on the estimated regression coefficients when this holds.</p><p>The following are major sources of multicollinearity:</p><ul><li>An overdetermined model: Multicollinearity exists here because the model has much more independent variables than the number of observations in the sample data set used for estimation or prediction.</li><li>Analyzing time series models: In time series analysis, the problem of autocorrelation is more pronounced than multicollinearity, and time-series data is unique because an independent variable involved is time. If time is included as an independent variable in your model, you would have to use time series methods for your analysis or you would run into severe multicollinearity.</li><li>Data collection method: sampling over a limited range of values for the independent variables in the population can cause multicollinearity as some parts of the population are not represented in the sample.</li><li>Other sources are constraints on the model or in the population being sampled, model specification error, etc.</li></ul><p>Consequences of Multicollinearity are:</p><ul><li>The Ordinary Least Squares estimators of the parameters of the model which are the best, linear, unbiased estimators have large variances and covariances which makes precise estimation difficult.</li><li>The covariance of estimates of parameters of collinear variables will be significantly larger.</li><li>OLS estimators and their standard errors can be sensitive to very small changes in the sample&#39;s data, etc.</li></ul><p>Ways to remove/reduce multicollinearity are:</p><ul><li>Increase the sample size.</li><li>Combining cross-sectional data with time-series data.</li><li>Transforming the variables into a much simpler form for estimation, etc.</li></ul><p>Thank you for reading! Happy learning from myself and <a href="https://medium.com/u/695170cdb34d">DSN AIPlus UI</a> .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d60b0a7cb04a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/lesson-04-correlation-between-variables-d60b0a7cb04a">Lesson 04: Correlation between Variables.</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lesson 03: Gaussian Distribution and Descriptive Statistics]]></title>
            <link>https://medium.com/dsnaiplusui/lesson-03-gaussian-distribution-and-descriptive-statistics-1b41af5f746d?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/1b41af5f746d</guid>
            <category><![CDATA[distribution]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nigeria]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Giwayusuf]]></dc:creator>
            <pubDate>Fri, 03 Jul 2020 06:14:49 GMT</pubDate>
            <atom:updated>2020-07-03T06:14:49.714Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gaussian Distribution and Descriptive Statistics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*MZZrOlj-kr7OfRKGIFbFtg.png" /><figcaption>image by William Iven via Re splash</figcaption></figure><p>Usually, when dealing with data, data collection is the first task. After which we make inferences from the data. Inferences can only be made when we understand the data distribution and parameters used to properly describe the data distribution.</p><p><em>The 7 days of statistics challenge by </em><a href="https://medium.com/u/695170cdb34d"><em>DSN AIPlus UI</em></a><em> , is meant to explain essential theoretical statistics, which also includes descriptive statistics and data distribution.</em></p><p>Data distributions are graphical methods of organizing and displaying useful information about data. There are several data distributions including Gaussian distribution, the binomial distribution, and the Poisson distribution.</p><p>This article will focus on the most important distribution, Gaussian, or normal distribution. By the end of this article you will:</p><ul><li>Become familiar with simple descriptive statistics</li><li>Identify Gaussian distribution and its use cases</li><li>Implement the calculation of sample mean and variance from scratch using python.</li></ul><p><strong>Descriptive statistics</strong></p><p>Descriptive statistics are used to describe the basic features of data in a study. They do this by providing simple quantitative summaries about a sample or population.</p><p>Before I continue, we need to be familiar with two terms:</p><ul><li><strong>Population</strong>: includes all the elements from a set of data. Here is the overall list of all entities from which inference can be drawn.</li><li><strong>Sample</strong>: a sample is a smaller group of members of a population selected from the population. It is usually selected randomly.</li></ul><p>For example, if I am studying the factors that affect relegated teams in the English Premier League; my population will be in the Premier league table from its inception in 1992 to date. I can study the table from 2010 to 2020, which will be the new sample for the study.</p><p>We usually cannot get data about an entire population. Due to this limitation, samples are used to make statistical inferences about the population.</p><p><strong>The mean</strong></p><p>A great attempt to understand distribution is to locate where most of the entries are concentrated.</p><p>The mean is the numerical average of entities in a data set. It describes the central location within a data set.</p><p>It can be calculated by dividing the sum of all the entities in a sample by the total number of entries in the sample.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FowjWDH9BLZeTyJwExhjpjF&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FowjWDH9BLZeTyJwExhjpjF&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="273" height="157" frameborder="0" scrolling="no"><a href="https://medium.com/media/a18972db148a8201aec706401aaa80c5/href">https://medium.com/media/a18972db148a8201aec706401aaa80c5/href</a></iframe><p><strong>Standard deviation</strong></p><p>Data distributions can be described by how far each entity is from the mean. This is known as a measure of variation.</p><p>Standard deviation is a measure of the amount of variation or dispersion of a set of values. Theoretically, it represents the average distance of each entity from the mean.</p><p>It can be calculated by these steps:</p><ol><li>Calculate the mean of the dataset</li><li>From each entry in the list, subtract the mean…</li><li>and square the differences</li><li>Find the average of the squared differences</li><li>Find the square root of the average .</li></ol><p>Here is the formula for calculating sample standard deviation.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FuAuVNh2CvDj2r66fA8nsjh&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FuAuVNh2CvDj2r66fA8nsjh&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="136" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f78816c1dd32cc761929b59edb56536/href">https://medium.com/media/0f78816c1dd32cc761929b59edb56536/href</a></iframe><p><strong>Gaussian distribution</strong></p><p>The Gaussian distribution also called normal distribution is a continuous probability distribution that is symmetrical about the mean.</p><p>The Gaussian distribution is the most popular and most important distribution because it fits several natural phenomena like heights, blood pressure and some other continuous variables.</p><p>It is symmetrically shaped with most of the data are clustered around the central peak/mean giving it a bell-shaped curve.</p><p>The curve is equally spread on both sides of the peak. The standard deviation of any point at one side of the mean is equal to the standard deviation at the same point at the other side of the mean.</p><p>Gaussian distribution can be represented by two parameters: the mean and the standard deviation.</p><p>Examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*Incq79YguE4iNc_NReaPcw.png" /></figure><p>The distribution in this example fits real data collected from 14-year-old girls during a study. Y-axis represents the frequency density, X-axis represents that height.</p><p>The graph is concentrated around the mean, 1.512 and reduces gradually on both sides with a standard deviation of 0.00741.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*YHyh3FJ7UOzt5UodegfEhQ.jpeg" /><figcaption>Graph via Mathsbitsnotebook</figcaption></figure><p>This graph shows normal distributions and parameters for different data using different colors. Note that small standard deviation corresponds to smaller spreads and taller peaks.</p><p>Now, after identifying the data as a normal distribution, we can make statistical generalizations such as the empirical rule of estimation.</p><p>The empirical rule is a statistical rule that states that for a normal distribution:</p><ol><li>99.7% of the data set falls within 3 standard deviations of the mean.</li><li>95% falls within 2 standard deviations of the mean.</li><li>68% falls within 1 standard deviation of the mean.</li></ol><p><strong>Note that no data set can occur as perfectly normal. But this abstraction only makes it easier to understand the distribution and make generalizations about it</strong>.</p><p><strong>Implementation of mean and standard deviation from scratch with python</strong></p><p>Only basic knowledge of coding is required here.</p><p><strong>Mean</strong></p><p>To calculate the mean, I created the function <em>calc_mean. </em>The function takes in one argument, “data_l” which is the list of entities. It returns the mean of the list as a float.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18095a02694785e4d4d70c58a0b384e9/href">https://medium.com/media/18095a02694785e4d4d70c58a0b384e9/href</a></iframe><p>The mean is equal to the sum of the list divided by total number of entries.</p><p>We do not want the mean to be returned as an integer, so I multiplied it by 1.0 to return the mean as float.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fef2e963b6ef177d8484b198979d6a4/href">https://medium.com/media/3fef2e963b6ef177d8484b198979d6a4/href</a></iframe><p><strong>Standard deviation</strong></p><p>To calculate the standard deviation, I created a function called <em>calc_sdev</em>. The function takes in the list of sample data as the argument and returns the standard deviation of the list as a float.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55586d6d1ad83d9e43362df251c37206/href">https://medium.com/media/55586d6d1ad83d9e43362df251c37206/href</a></iframe><p>First, We need to store the fundamental variables which will be used in the standard deviation formula.</p><ul><li>I used the function <em>calc_mean</em> to calculate the mean of the data set and stored it as a variable named “mean”.</li><li>I calculated n which is N — 1 for sample.</li><li>I created the variable “sigma” and stored it as 0. This will help us store our standard deviation.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b715c317cd8d54318d7e5ec71d5ea02f/href">https://medium.com/media/b715c317cd8d54318d7e5ec71d5ea02f/href</a></iframe><p>Next, I implemented the formula in two steps by looping through all elements in the data set as “s”.</p><ul><li>In the first step: sigma is equal to the sum of square of the difference between s and the mean, for all s.</li><li>Then, the standard deviation is equal to the square root of sigma divided by n.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ebfb744b7b6347b7165c62e898b2d24/href">https://medium.com/media/9ebfb744b7b6347b7165c62e898b2d24/href</a></iframe><p>This functions can then be used to calculate mean and sample standard deviation of any data set.</p><p><strong>Execution</strong></p><p>In this section, we will calculate the mean and standard deviation of 100 numbers using the functions we created.</p><p>First, I will generate a list of 100 random numbers with the following:</p><ul><li>I imported seed to initiate a random number generator</li><li>I imported randn to generate random numbers</li></ul><p>You can use any seed value to generate your random numbers, for example i used seed(1).</p><p>I stored my random numbers in the list, <em>data_list.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ae724637e090a58ebe93d84966fdb4d/href">https://medium.com/media/2ae724637e090a58ebe93d84966fdb4d/href</a></iframe><p>Finally, I used the functions <em>calc_mean</em> and <em>calc_sdev</em> to print the mean and standard deviation for <em>data_list.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7786e2678aa63f9d74775d62ea58a35a/href">https://medium.com/media/7786e2678aa63f9d74775d62ea58a35a/href</a></iframe><p>This can be used on any data set as long as we store them as lists.</p><p>Note that we created a function to calculate standard deviation for the sample.</p><p>Here is a challenge: create a function to calculate standard deviation for population data.</p><p>Hint: Change N-1 to N.</p><p><strong>Conclusion</strong></p><p>The importance of understanding data distributions and the parameters used to summarize them cannot be overemphasized, and we have discussed them in this article.</p><p>All distributions have specific parameters describing them just as Normal distribution can be summarized by numerical values of its mean and standard deviation.</p><p>You can read about other distributions and the parameters that summarize them via this <a href="https://en.wikipedia.org/wiki/List_of_probability_distributions">link</a>.</p><p><em>Thanks for reading, happy learning from myself, and </em><a href="https://medium.com/u/695170cdb34d"><em>DSN AIPlus UI</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b41af5f746d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/lesson-03-gaussian-distribution-and-descriptive-statistics-1b41af5f746d">Lesson 03: Gaussian Distribution and Descriptive Statistics</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lesson 2: Introduction to Statistics]]></title>
            <link>https://medium.com/dsnaiplusui/lesson-2-introduction-to-statistics-1e0a14da1ed9?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/1e0a14da1ed9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Oluwafadekemi Ogundiya]]></dc:creator>
            <pubDate>Thu, 02 Jul 2020 09:30:10 GMT</pubDate>
            <atom:updated>2020-07-02T09:30:10.654Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nszY2TbKdACp5q6ZvBRbOA.jpeg" /></figure><p>Statistics is a subfield of mathematics. It helps us to make educated guesses of the unknown and find useful information in an ocean of data. But despite its usefulness, many people struggle with statistics. What is it? How does it work?</p><p>Bringing sense out of data can be cumbersome and alarming for every machine learning enthusiast. As part of my contribution to the 7 days of statistical theory challenge by <a href="https://twitter.com/dsn_ui/status/1276528030454894592?s=19">AI+Club UI</a>, a student community under Data Science Nigeria, I shall write about on Introduction to Statistics.</p><h3><strong>What is Statistics?</strong></h3><p>It refers to a collection of methods for working with data and using data to answer questions. The field comprises a grab bag of methods for working with data that can seem large and amorphous for beginners. It&#39;s hard to see the line between methods that belong to statistics and methods that belong to other fields of study.</p><p>The first thing we will delve into is what type of data we will encounter when dealing with Statistics. Roughly we can divide data into two distinct classes namely Categorical data and Numerical data. Now, this sounds somewhat self-explanatory because they are, but I’ll explain.</p><p><strong>Numerical data:</strong> This is data that is measurable, such as time, height, weight, amount, and so on. And it is further divided into two which are discrete and continuous data.</p><p><strong>Discrete data: </strong>This is data that<strong> </strong>is numerical data that represents countable items. They take on values that can be grouped into a list where the list may be finite or infinite. Continuous data is numerical data which represents measurements that describe their value as intervals on an actual number line, rather than take counting numbers. We may further divide continuous data into intervals and ratios.</p><p><strong>Categorical data:</strong> This is data that may be divided into groups such as race, sex, age group, and so on. And it is further divided into two categories namely nominal and ordinal data. Nominal data is data used to name variables providing no numerical value. This data type is a subcategory of categorical data and sometimes called “labeled” or “named” data. Ordinal data is a data type with a set order or scale to it. However, this order does not have a standard scale on which the difference in variables in each scale is measured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*l7U1ugLRag4maxLdCqeVYA.png" /><figcaption>Photo Credit: Ask Data science</figcaption></figure><p>That being said, for statistical tools that we use it&#39;s helpful to divide the field of statistics into two large groups of methods: Descriptive statistics for summarizing data and Inferential statistics for drawing conclusions from a sample of data.</p><p><strong>Descriptive Statistics: </strong>Descriptive statistics refer to methods for summarizing raw observations into information that we can understand and share. Descriptive statistics describe a sample. That’s straightforward.</p><p>You take a group you’re interested in, record data about the group members, and then use summary statistics and graphs to present the group properties. With descriptive statistics, there is no uncertainty because you are describing only the people or items that you measure. You’re not trying to infer properties about a larger population.</p><p>The process involves taking a potentially large number of data points in the sample and reducing them down to a few meaningful summary values and graphs. This procedure allows us to gain more insights and visualize the data than pouring through row upon row of raw numbers!</p><p><strong>Inferential Statistics:</strong> Inferential statistics is a fancy name for methods that aid in quantifying properties of the domain or population from a smaller set of obtained observations called samples.</p><p>Inferential statistics take data from a sample and make inferences about the larger population from which the sample was drawn. Because the goal of inferential statistics is to draw conclusions from a sample and generalize them to a population, we need to have confidence that our sample accurately reflects the population. This requirement affects our process. At a broad level, we must do:</p><ol><li>Define the population we are studying.</li><li>Draw a representative sample from that population.</li><li>Use analyses that incorporate sampling errors.</li></ol><p>We don’t get to pick a convenient group. Instead, random sampling allows us to have confidence that the sample represents the population. This process is a primary method for obtaining samples that mirrors the population on average. Random sampling produces statistics, such as the mean, that do not be too high or too low.</p><p>Using a <a href="https://statisticsbyjim.com/glossary/sample/">random sample</a>, we can generalize from the sample to the broader population. Unfortunately, gathering a random sample can be a complicated process.</p><p><strong><em>Conclusion</em></strong></p><p>A famous writer and philosopher <strong>George Santayana</strong> once said</p><blockquote>“Those who do not learn history are doomed to repeat it.”</blockquote><p>Statistics help you to make a calculated inference from the past and help you to predict what is to come (future) and make sense of data.</p><p>Thank you <a href="https://twitter.com/dsn_ui/status/1276528030454894592?s=19">AI+Club UI</a> for the opportunity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e0a14da1ed9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/lesson-2-introduction-to-statistics-1e0a14da1ed9">Lesson 2: Introduction to Statistics</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Statistics for Machine Learning]]></title>
            <link>https://medium.com/dsnaiplusui/statistics-for-machine-learning-539e92e74be3?source=rss----311bc712981e---4</link>
            <guid isPermaLink="false">https://medium.com/p/539e92e74be3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machinelerning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[modeling]]></category>
            <dc:creator><![CDATA[Odeajo Israel]]></dc:creator>
            <pubDate>Wed, 01 Jul 2020 09:40:15 GMT</pubDate>
            <atom:updated>2020-07-01T09:23:06.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*cp8K4xEqLhh7vzO13EpNQQ.jpeg" /></figure><h3>Introduction</h3><p><strong>Statistics is </strong>a prerequisite to the field of applied <strong>machine learning</strong>. We <strong>need statistics</strong> to help transform observations into information and to answer questions about samples of observations.</p><p><strong>Statistics</strong> are important because today we live in the information world and much of this information is determined mathematically by <strong>Statistics</strong> Help. It means to be informed, correct data, and static concepts are necessary. They use their <strong>statistical skills</strong> to collect relevant data.</p><p>In this article, I will discuss the importance of Statistics to Machine Learning for Data Scientist &amp; Machine Learning Engineer and any other person who sees <strong>Data </strong>as the new gold.</p><p>The following are the highlights of the effects of Statistics on Machine Learning model as a solution</p><h3>1. Statistics in Data Preparation</h3><p>Statistical methods are required in preparing training and testing data for your machine learning model.</p><p>This includes techniques for:</p><ul><li>Outlier detection.</li><li>Missing value imputation.</li><li>Data sampling.</li><li>Data scaling.</li><li>Variable encoding.</li></ul><p>A basic understanding of data distributions, descriptive statistics, and data visualization is required to help you identify the methods to choose when performing these tasks.</p><h3>2. Statistics in Model Evaluation</h3><p>Statistical methods are required when evaluating the skill of a machine learning model on data not seen during training.</p><p>This includes techniques for:</p><ul><li>Data sampling.</li><li>Data resampling.</li><li>Experimental design.</li></ul><p>Re-sampling techniques such as k-fold cross-validation are often well understood by machine learning practitioners, but the rationale for why this method is required is not.</p><h3>3. Statistics in Model Selection</h3><p>Statistical methods are required when selecting a final model or model configuration to use for a predictive modeling problem.</p><p>These include techniques for:</p><ul><li>Checking for a significant difference between results.</li><li>Quantifying the size of the difference between results.</li></ul><p>This might include the use of statistical hypothesis tests.</p><h3>4. Statistics in Model Presentation</h3><p>Statistical methods are required when presenting the skill of a final model to stakeholders.</p><p>This includes techniques for:</p><ul><li>Summarizing the expected skill of the model on average.</li><li>Quantifying the expected variability of the skill of the model in practice.</li></ul><p>This might include estimation statistics such as confidence intervals.</p><h3>5. Statistics in Prediction</h3><p>Statistical methods are required when making a prediction with a finalized model on new data.</p><p>This includes techniques for:</p><ul><li>Quantifying the expected variability for the prediction.</li></ul><p>This might include estimation statistics such as prediction intervals.</p><h3>Conclusion</h3><p>For a successful Machine Learning life cycle, the above facts about statistics are <strong>IMPORTANT </strong>to build a productive machine learning solution.</p><p>I write this article as regards the <strong><em>7 days of Statistics challenge</em></strong> by AI+ Club, University of Ibadan <a href="https://medium.com/u/695170cdb34d">DSN AIPlus UI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=539e92e74be3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dsnaiplusui/statistics-for-machine-learning-539e92e74be3">Statistics for Machine Learning</a> was originally published in <a href="https://medium.com/dsnaiplusui">dsnaiplusui</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>