<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Willam Green on Medium]]></title>
        <description><![CDATA[Stories by Willam Green on Medium]]></description>
        <link>https://medium.com/@dskswu?source=rss-1ed8f83521bf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xmW9A9RDughudJ9L.</url>
            <title>Stories by Willam Green on Medium</title>
            <link>https://medium.com/@dskswu?source=rss-1ed8f83521bf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 05:51:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dskswu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Machine Learning with a Heart: Predicting Heart Disease]]></title>
            <link>https://medium.com/@dskswu/machine-learning-with-a-heart-predicting-heart-disease-b2e9f24fee84?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2e9f24fee84</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Wed, 01 Aug 2018 03:37:05 GMT</pubDate>
            <atom:updated>2018-08-01T03:45:45.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*6WGnPZ5lkiT2QgK-JP1DFw.png" /></figure><p>Machine learning (ML) is causing quite the buzz in the healthcare industry as a whole . Payers to healthcare companies around the world are taking advantage of ML today. In this post, I will demonstrate a use case and show how we can harness the power of ML and apply it real world problems. We’ll walk through a very simple baseline model for predicting heart disease from patient data, how to load the data, and make some predictions.</p><p>In previous discussion I shared my notes on Deep Learning Book Part I: <a href="https://www.deeplearningbook.org/contents/part_basics.html">Applied Math and Machine Learning Basics</a>. This is a good segue to apply some of those concepts discussed in the book. Let’s take a look at <a href="https://www.drivendata.org/competitions/54/machine-learning-with-a-heart/page/108/">Warm Up: Machine Learning with a Heart</a> competition hosted by <a href="https://www.drivendata.org/">DrivenData</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAqzvCxPjpDN8RC9HQw45w.png" /><figcaption>Source: <a href="https://machinelearning-blog.com/2017/11/19/fsgdhfju/">https://machinelearning-blog.com/2017/11/19/fsgdhfju/</a></figcaption></figure><p>The figure above is a great visual of a machine learning project A to Z. The first step before we began coding is to understand the problem we are trying to solve and get the available data. In this project, we will work with publicly available data from <a href="https://www.drivendata.org/">DrivenData</a>.</p><h4>About Heart Disease</h4><p>Preventing heart disease is important. Good data-driven systems for predicting heart disease can improve the entire research and prevention process, making sure that more people can live healthy lives.</p><p>In the United States, the Centers for Disease Control and Prevention is a good resource for information about heart disease. According to <a href="https://www.cdc.gov/heartdisease/facts.htm">their website</a>:</p><ul><li>About 610,000 people die of heart disease in the United States every year–that’s 1 in every 4 deaths.</li><li>Heart disease is the leading cause of death for both men and women. More than half of the deaths due to heart disease in 2009 were in men.</li><li>Coronary heart disease (CHD) is the most common type of heart disease, killing over 370,000 people annually.</li><li>Every year about 735,000 Americans have a heart attack. Of these, 525,000 are a first heart attack and 210,000 happen in people who have already had a heart attack.</li><li>Heart disease is the leading cause of death for people of most ethnicities in the United States, including African Americans, Hispanics, and whites. For American Indians or Alaska Natives and Asians or Pacific Islanders, heart disease is second only to cancer.</li></ul><p>For more information, you can look at the website of the Centers for Disease Control and Prevention: <a href="https://www.cdc.gov/heartdisease/prevention.htm">preventing heart disease</a></p><h4>Problem description</h4><p>The goal is to predict the binary class heart_disease_present, which represents whether or not a patient has heart disease:</p><ul><li>0 represents no heart disease present</li><li>1 represents heart disease present</li></ul><h4>Dataset</h4><p>There are 14 columns in the dataset, where the patient_id column is a unique and random identifier. The remaining 13 features are described in the section below.</p><ul><li>slope_of_peak_exercise_st_segment (type: int): the slope of the peak exercise <a href="https://en.wikipedia.org/wiki/ST_segment">ST segment</a>, an electrocardiography read out indicating quality of blood flow to the heart</li><li>thal (type: categorical): results of <a href="https://www.ucsfbenioffchildrens.org/tests/007201.html">thallium stress test</a> measuring blood flow to the heart, with possible values normal, fixed_defect, reversible_defect</li><li>resting_blood_pressure (type: int): resting blood pressure</li><li>chest_pain_type (type: int): chest pain type (4 values)</li><li>num_major_vessels (type: int): number of major vessels (0-3) colored by flourosopy</li><li>fasting_blood_sugar_gt_120_mg_per_dl (type: binary): fasting blood sugar &gt; 120 mg/dl</li><li>resting_ekg_results (type: int): resting electrocardiographic results (values 0,1,2)</li><li>serum_cholesterol_mg_per_dl (type: int): serum cholestoral in mg/dl</li><li>oldpeak_eq_st_depression (type: float): oldpeak = <a href="https://en.wikipedia.org/wiki/ST_depression">ST depression</a> induced by exercise relative to rest, a measure of abnormality in electrocardiograms</li><li>sex (type: binary): 0: female, 1: male</li><li>age (type: int): age in years</li><li>max_heart_rate_achieved (type: int): maximum heart rate achieved (beats per minute)</li><li>exercise_induced_angina (type: binary): exercise-induced chest pain (0: False, 1: True)</li></ul><h4>Step 2: Clean and Prepare</h4><p>Datasets in a perfect world is a perfectly curated group of observations with no missing values or anomalies. However, this is not true. Real world data comes in all shapes and sizes. It can be messy, which means it needs to be clean and wrangles. Data cleaning is a necessary part in data science problems. Machine learning models learn from data. It is crucial, however, that the data you feed them is specifically pre processed and refined for the problem you want to solve. This includes data cleaning, preprocessing, feature engineering, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*scRPugIFIt8PZ1mO8V6xlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/1*XEhUEeAip53Kt1KCc4YmMA.png" /></figure><h4>Visual Exploratory Data Analysis (EDA)</h4><p>It’s time to visualize our data with a little help from the seaborn package.</p><p>From the frequency plot of heart disease below, we see that the two classes (‘Heart Disease’ and ‘No Heart Disease’) are approximately balanced, with 45% of observations having heart disease and the remaining population not having heart disease.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*IjY1iijDn-OY6-3ZTD49pA.png" /></figure><p>The data is relatively well-balanced, so we won’t take any steps here to equalize the classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r45VQG76b--AtrGlDUtlOg.png" /></figure><p>The pairplot above allows us to see the distribution and relationship of numerical variables. The diagonal shows kernel density plots showing the rough distributions of the two populations. The scatter plots show relationship between plots. We can make a couple observations from the pairplot:</p><ul><li>Resting blood pressure tends to increase with age regardless of heart disease.</li><li>We can see that max heart rates are significantly lower for people without heart disease.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*WOovBUmxxIdozirJoluUeg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*gqId5tHpmWswarolq74tMQ.png" /></figure><h4>Step 3 and 4: Train and Test Model</h4><p>Model selection is the process of combining data and prior information to select among a group of statistical models. When it comes to classic first pass models, few can contend with logistic regression. This linear model is fast to train, easy to understand, and typically does pretty well “out of the box”.</p><p>The Scikit Learn logistic regression model works well combined with Pipeline and GridSearchCVpreprocessing tools. This will help to streamline the process of model training and hyperparameter optimization.</p><p>Sklearn’s pipeline functionality makes it easier to repeat commonly occuring steps in your modeling process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*u1ztp6NZY5F7fugsH8XS8Q.png" /><figcaption>Pipeline and GridSearchCV preprocessing</figcaption></figure><p><strong>GridSearchCV</strong> allows you to construct a grid of all the combinations of parameters, tries each combination, and then reports back the best combination/model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*eatL9HmnBwBV_Wgdhc92Zw.png" /></figure><p>The figure above is a confusion matrix for parameter <strong><em>C</em></strong> and penalty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*rvg4QcAhP6eatGpWDxGJNg.png" /></figure><p>Looking at the <strong><em>C</em></strong> (Inverse Regulation Strength) plot and print out above, we can see that the highest accuracy achieved is when the parameter <strong><em>C</em></strong> is set to 0.1 and the penalty parameter is set to ‘l2’ (<strong>Ridge Regression</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*jhv6svlof9VelayZ53d7Zw.png" /></figure><p>From the <strong>confusion matrices</strong> above, we see that the most ‘accurate’ model tends to make the worse kind of mistake. Like above, correct predictions appear on the main diagonal, whereas all off-diagonal values correspond to incorrect classifications.</p><p>Accuracy is measured on how well all cases were classified. The two types of misclassifications:</p><ul><li>labeling a healthy person as unhealthy,</li><li>labeling an unhealthy person as healthy</li></ul><p>are not equally bad. It’s definitely worse to mislabel a patient as healthy if they actually have heart disease, as their heart disease would go untreated, and they may continue with an unhealthy diet or pursue dangerous activity levels.</p><p>In terms of classification metrics, that kind of prediction mistake is a <strong>False Negative</strong>. The alternative mistake (a <strong>False Positive</strong>) is to label someone as unhealthy when they are actually healthy, which would lead to someone unnecessarily changing their diet and lifestyle, which would be unpleasant, but not potentially lethal, as in the other case.</p><h4>Step 5: Improve</h4><p>One way to improve our model is to reduce the number of features in your data matrix by picking those with the highest predictive value is advisable.</p><p>Let’s build a model that takes the cost of mistakes into account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*LIN41NaFspq2xZNYUYZyFg.png" /></figure><p>The number of significant features is less than the total number of features, so the unimportant features are eliminated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*s6Fevj9wIQnLdXEQsBNztg.png" /></figure><p>The l1 logistic regression (<strong>Lasso Regression</strong>) was smart enough to assign 0 importance to features below the significance threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*eAXKk9eknQeWkJpGxB94zw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*YBSZjlZxAv-8NpV54fvUNg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*vkoAEctLHD4edNP6H99i_A.png" /></figure><p>The confusion matrices directly below as well as the confusion matrices a bit further up were made by fitting classifiers with 30 different slices of the data and generating a confusion matrix for each data slice.</p><p>The confusion matrices below used a classifier with parameters {<strong><em>C</em></strong>=1, penalty =L1}, only used the 13 features that the recursive feature elimination found to be significant, and used parameters that were selected by the <strong>recall</strong> scoring measure rather than simple accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*4gfuCGqdM9wyvwqJN4a6dA.png" /></figure><p>Over the 30 runs and over a sample of 45 people, this model labeled (on average)<strong> 3.27</strong> unhealthy people as being healthy, which is slightly better than the above model, which labeled <strong>3.53</strong> unhealthy people as healthy, but this model labeled <strong>2.63</strong> healthy people as unhealthy while the above model labeled <strong>1.80</strong> healthy people as unhealthy.</p><p>Considering the differing costs of mistakes, this is an improvement.</p><h4>Summary</h4><p>In summary, we demonstrated a use case and showed how we can harness the power of ML and apply it real world problems. The <a href="https://www.drivendata.org/competitions/54/machine-learning-with-a-heart/page/108/">Warm Up: Machine Learning with a Heart</a> is a good dataset to practice applying ML algorithms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2e9f24fee84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Oh My Guudness….. Machine Learning II]]></title>
            <link>https://medium.com/@dskswu/oh-my-guudness-machine-learning-ii-1068a7abc09b?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/1068a7abc09b</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Sun, 22 Jul 2018 04:22:20 GMT</pubDate>
            <atom:updated>2018-07-22T04:22:20.126Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hSeRWsqUiBtHybSm-MN7Gg.png" /></figure><p>This blog post a continuation to <a href="https://medium.com/@dskswu/oh-my-guudness-machine-learning-46fc4acf745d">Oh My Guudness….. Machine Learning</a>.</p><p><strong><em>Please note that this post is for my future self to review the materials on this book without reading it all over again.</em></strong></p><p>There are many statistical tools that help us to achieve machine learning goal of solving a task not only on training data but also to generalize. Estimators, bias and variance are foundational concepts useful to characterize notions of generalization, underfitting and overfitting.</p><p><strong>Point estimation</strong> is the relation between input and target variables. We want to estimate a parameter. The true value of the parameter : 𝜽 (fixed and unknown). The estimated value of parameter : 𝜽 (a function of the data, random variable). Let { 𝑥 1 , … , 𝑥 𝑚 } be a set of 𝑚 independent and identically distributed (i.i.d.) data points . A point estimator or statistic is any function of the data: 𝜃 𝑚 = 𝑔(𝑥(1) , … , 𝑥(𝑚) ). <strong>Function estimation</strong> trys to predict a variable <em>y</em> given an input vector <em>x. </em>It describes the realtionship between<em> y </em>and<em> x.</em></p><p><strong>Bias</strong> of a model is the difference between the expected value of the estimator and the actual value. The difference between the expectation of the means of the samples we get from a population with mean <em>θ</em> and that population parameter, <em>θ</em>, itself is zero, because the sample means will be all distributed around the population mean. None of them will be the population mean exactly, but the mean of all the sample means will be exactly the population mean.</p><p>This is not the case for other parameters, such as the variance, for which the variance observed in the sample tends to be too small in comparison to the true variance. So if we want to estimate the population variance from the sample we divide by <em>n</em>−1, instead of <em>n</em> (<a href="https://en.wikipedia.org/wiki/Bessel%27s_correction">Bassel’s correction</a>) to correct the bias of the sample variance as an estimator of the population variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*55Mxz2FrRd60QQSu7CfLoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*9Y7qTIDxcoEXtigGd_p8qA.png" /></figure><p>Variance and standard error is another important property of an estimator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*8YwbSQOEyIE7eWvxf_pCDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*SRH0FTd66A-DDtFD_M-EQQ.png" /></figure><p>If we resampled the dataset and repeat the learning process, what is the variance in each of our learned parameters? A high variance implies that the parameters we learn are highly dependent on the dataset, which may indicate overfitting. A low variance implies that the parameters don’t change much when the dataset they train on is different, which implies that there’s less overfitting. We often compute the generalization error by takin the sample mean of the error on the test set.</p><p>Bias-Variance is a <strong>tradeoff</strong> between a model’s ability to minimize <strong>bias</strong> and <strong>variance</strong>. It is essential to understand how different sources of error that lead to bias and variance, which helps us to improve the data fitting process resulting in more accurate models. Bias-variance are actually side effects of one factor: the complexity of our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*1fW0LFbf5X5u-lwgTTxShg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*UnosBxRsd7yXIuWI_cAbhw.png" /></figure><p>Consistency is the number of data points <em>m </em>in the dataset increase, the point estimates converge to the true value of the corresponding parameters. It ensures that the bias induced by the estimator diminishes as the number of data examples grows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*gDZwS600aT-OBhHAOPxsEA.png" /></figure><p>Maximum likelihood estimation is a method in statistics for estimating parameter(s) of a model for given data. The basic intuition behind MLE is that the estimate which explains the data best, will be the best estimator.</p><p>The main advantage of MLE is that it has asymptotic property. It means that when the size of the data increases, the estimate converges faster towards the population parameter. We use MLE for many techniques in statistics to estimate parameters.</p><p>Another interpretation of the MLE is that it minimizes teh ference between 2 probability distributions: one which si given by your learned parameters <strong><em>θ</em></strong> and the other which is the true underlying data generating distribution. The <strong>KL divergence</strong> is a measure of difference between two probability distributions, and is closely related to the <strong>cross entropy</strong> loss. Minimizing this is equivalent to maximizing the likelihood.</p><p>Conditional Log-Likelihood and MSE</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*0NBvm9MxvdodFX3hll8ing.png" /></figure><p><strong>Supervised Learning Algorithms</strong> are learning algorithms that learn to associate some input with some output, given an training set of examples of inputs <strong><em>x</em></strong> and outputs <strong><em>y</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*OqOzEP0pLmqvEBirKAjPXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*nX0HhKiocvP0Lndyvb-GnQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*FjhVxyGH9RQ9S-EgqwcuCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*2OTAShwZj2Hm0CAu5g4fKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*sQPTRdAG2HomJUM_An5Mqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*2ojyZD4JWaYKXwdK7-ONTw.png" /></figure><p><strong>Unsupervised Learning Algorithms </strong>s cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*9KTGyoibhawIRFseJ9aW7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*VN_qPzuk394z6VoArKI8RA.png" /></figure><p><strong>Stochastic Gradient Descent </strong>is an iterative optimization algorithm that can be applied to functions that are a linear combination of differentiable functions. Main idea is that computing the true gradient may be inefficient or impractical, especially on a very large deep learning dataset, since we need to sum across all of the training examples, and there could be millions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*IWrqSXwT4qusc_OcRPCDzg.png" /></figure><h4>Conclusion</h4><p>This wraps up Chapter 5. If any errors are found, please email me at dskswu@gmail.com. Meanwhile follow me on my twitter <a href="https://twitter.com/William33712308">here</a>. The log for journey can be found<a href="https://github.com/MinuteswithMetrics/100-Days-Of-ML-Code/blob/master/100DaysOfML_log.md"> here</a>.</p><h4>Reference</h4><ol><li>Ian Goodfellow, Yoshua, Bengio, and Aaron Courvlle. <em>Deep Learning</em> MIT Press, 2016. <a href="http://www.deeplearningbook.org/">http://www.deeplearningbook.org</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1068a7abc09b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Oh My Guudness….. Machine Learning]]></title>
            <link>https://medium.com/@dskswu/oh-my-guudness-machine-learning-46fc4acf745d?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/46fc4acf745d</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Sat, 21 Jul 2018 05:20:05 GMT</pubDate>
            <atom:updated>2018-07-21T05:20:05.017Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*53io1VgKlHKgYEJs65B0nw.png" /></figure><p>This blog post is my progress for day 7 in the <a href="https://github.com/llSourcell/100_Days_of_ML_Code">#100DaysofMLCode</a>. Going back to the basics with ML has helped me to make better sense of what I have learned. In this blog post I will be sharing my notes on <a href="http://www.deeplearningbook.org/contents/numerical.html">Deep Learning Book Chapter 5: Machine Learning Basics</a>. This chapter will be discussed in two parts.</p><p><strong><em>Please note that this post is for my future self to review the materials on this book without reading it all over again.</em></strong></p><p>Machine learning is fun, challenging, puzzling, and even a bit scary if you’re one of those people that believe robots will someday steal our jobs and rule the world. This article will focus on my notes from Deep Learning book Chapter 5: Machine Learning Basics. First question, what is machine learning? Machine Learning is a mix between math, statistics and a few of AI that teaches computers to do what comes naturally to humans: learn from experience. It uses algorithms to “learn” from data without relying on a predetermined equation as a model. The algorithms adaptively improve their performance as the number of samples available for learning increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/1*LBqzYOv153c6LcFBwQvDzg.jpeg" /></figure><p>Machine learning algorithms learns from experience (E) to do a task (T) with performance (P) measures.</p><p>The task, <strong>T</strong> enables us to tackle tasks that are too hard to solve with written written programs designed by human beings. Common <strong>T </strong>are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0YftJQEIAlKbvLfj.png" /></figure><p>The performance measure, <strong>P </strong>evaluates the ML algorithm abilities using quantitative measure of its performance. In most cases the performance measure <strong>P</strong> is specific to<strong> T</strong> being carried out. For example, for a task like classification, we can measure the <strong>accuracy</strong> of the model by the proportion of correct outputs. We can also measure the <strong>error rate</strong>, which is the proportion of examples for which the model produces an incorrect output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Aj2tFfls2G2ABPyrgea00g.png" /><figcaption>Split data into training and testing set. Normally 80/20, no set rule.</figcaption></figure><p>The model is evaluated on the test set, which is data that has not been trained on. We want to know how well the model performs on data it has not seen before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/0*GxO2ahUJg52_1mtJ.jpg" /><figcaption>No cheating</figcaption></figure><p>The experience, <strong>E </strong>is categorized into unsupervised or supervised algorithms based on what kind of experience they are allowed to have during the learning process. Simply, algorithms are given access to the training dataset. There are 2 types of learning algorithms: <strong>supervised</strong> and <strong>unsupervised</strong> learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/0*JBdn3Fpb_cSqtzch.png" /></figure><p>The dataset contains features that describe each individual example and also labels/targets. Usually involves learning/estimating a probability distribution <strong>p(<em>y</em>∣<em>x</em>)</strong>.Supervised learning train the machine using data which is well labeled that means some data is already tagged with correct answer. Whereas unsupervised learning wants to learn <strong>p(<em>x</em>)</strong>. Unsupervised learning data is not labeled with the related belonging class. The system, therefore, develops and organizes the data, looking for common characteristics among them, and changing them based on their internal knowledge. The main difference is that in unsupervised learning, we are not provided the labels <strong><em>y</em></strong>.</p><p>Some machine learning algorithms do not just experience a ﬁxed dataset. for example, <strong>reinforcement learning</strong> algorithms don’t experience the same dataset over and over, they interact with an environment and learn from a feedback loop/reward signal. The dataset can be described as a <strong>design matrix</strong>, where each row contains a different example and each column contains a different feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*VnMtt-fBTJVOnwEu.jpg" /><figcaption><strong>Reinforcement learning</strong></figcaption></figure><p>Example: Linear Regression model is one member of the supervised classification algorithm family. Below is the most accurate and well-defined definition of logistic regression from Wikipedia.</p><p><em>“Logistic regression measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a </em><a href="https://en.wikipedia.org/wiki/Logistic_function"><em>logistic function</em></a><em>” </em>(<a href="https://en.wikipedia.org/wiki/Logistic_regression">Wikipedia</a>)</p><p>To better understand the definition, it uses a black box function to understand the relation between the categorical <strong>dependent variable </strong>and the <strong>independent variables</strong>. This black box function is popularly known as the <strong>Softmax function</strong>. Our goal is to predict the target class (dependent variable) using the independent variables.</p><p>To show you how regression algorithm works we’ll use a simple regression with one parameter — a home’s living area — to predict price. To keep it simple, assume that there is a linear relationship between area and price. A linear relationship is represented by a linear equation:</p><p>To show you how regression algorithm works we’ll take into account only one parameter — a home’s living area — to predict price. It’s logical to suppose that there is a linear relationship between area and price. And as we remember from high school, a linear relationship is represented by a linear equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*UA2YbS-vfkDUKGab2rVJsQ.png" /></figure><p>In this example y = price and x equals area. Predicting the price of a home is as simple as solving the equation (where k0 and k1 are constant coefficients): <strong>price = k0 + k1 * area</strong></p><p>We calculate the coefficients (k0 and k1) using regression. For example, we have 1000 known house prices in a given area. Using a learning technique, we can find a set of coefficient values. Once the coefficient values are found, we can plug in different area values to predict the resulting price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*gsQKnXlEgo6Oj_3yW3bPvA.gif" /><figcaption>[In this graph, y is price and x is living area. Black dots are our observations. Moving lines show what happens when k0 and k1 change.]</figcaption></figure><p>There is always a deviation, or difference between a predicted value and an actual value. The total deviation is calculated by summing the deviations for each k0 and k1 combination. Regression takes every possible value for k0 and k1 and minimizes the total deviation; this is the idea of regression in a nutshell.</p><p>Mean Squared Error (MSE) measures the average squared difference between an observation actual and predicted values. The output is a single number representing the cost, or score, associated with our current set of weights. Our goal is to minimize MSE to improve the accuracy of our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*paR729xftE1ZmUIkxIOZug.png" /></figure><p>Overfitting and underfitting are the two biggest challenge in ML. Our model (algorithm) must perform well on new unseen inputs — not just on the data on which our model was trained. Generally we train to minimize the test error. However, we want our model to perform well on on inputs it has never seen before. This is referred to as <strong>generalization</strong>. Generalization/Test error is the expected value of error on a new input.</p><p><strong>Statistical learning</strong> theory gives us some ideas on how to get a better expected testing error. We want to ensure that the training and test data both came from the same underlying probability distribution. ML assumes that the dataset are independent and identically distributed (drawn form the same probability distribution).</p><p>The factors determining how well a ML algorithm performs are its ability to make the training error-small and make the gap between training and test error small. These two factors correspond to the machine learning challenges: <strong>underfitting</strong> and <strong>overfitting</strong>. Underfitting is when the model is not able to obtain a sufficient low error value on the training set. Overfitting is when the gap between the training error and test error is too large.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bzm5fZga3Aw3KwkORGu2XA.png" /></figure><p><strong>Capacity</strong> refers to its ability to fit a wide variety of functions. For example, a polynomial model has higher capacity than a linear model. A ML algorithms performs best when the capacity is proportional to the complexity of its task and the input of the training data set.When a model’s capacity is too low for a particular task, they tend to underfit, when it’s too high, they tend to overfit.</p><p>There are several different ways to control capacity. One way to change it, is to change the number of input features it has and simultaneously add new parameters with those features. Set the functions that the learning algorithm is allowed to select as being the solution. This is called <strong>hypothesis space.</strong> Forecast resource saturation or demand.</p><p>Regularization is a technique which is used to solve the overfitting problem of the <strong>machine learning</strong> models. There are two types of regularization as follows:</p><ul><li>L1 Regularization or Lasso Regularization</li><li>L2 Regularization or Ridge Regularization</li></ul><p>L1 Regularization or Lasso Regularization adds a penalty to the error function. The penalty is the sum of the <strong>absolute</strong> values of weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*B17Ta-7H0bQdjrKx.png" /></figure><p>p is the tuning parameter which decides how much we want to penalize the model.</p><p>L2 Regularization or Ridge Regularization also adds a penalty to the error function. But the penalty here is the sum of the <strong>squared</strong> values of weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/0*esuFL1NyrGs879e_.png" /></figure><p>Similar to L1, in L2 also, p is the tuning parameter which decides how much we want to penalize the model.</p><p><strong>No free lunch theorem</strong> states that no machine learning model is better than another when classifying examples over all possible data generating distributions — all models do the same when evaluated over <strong>all possible tasks</strong>. This means that there’s no machine learning algorithm that’s universally better than another one, but certain algorithms are obviously better for certain tasks.</p><p>Hyperparameters and validation sets are used to control the algorithms behavior. Algorithms may contain several hyperparametes (such as model capacity or λ value for regularization) and these parameters must be set, and aren’t generally learned. Is not practical to select hyperparameters based on their performance on the training dataset, because the learning procedure will then choose a hyperparameter setting that maximizes model capacity to fit the training dataset, which leads to overfitting.</p><p>For example, given a training dataset we can always fit it better by selecting a model capacity that is higher and no regularization, but this basically defeats the purpose of regularization. Generally we set a validation dataset aside to not train/learn on, but to validate the hyperparameters. Cross-validation can be used when the dataset is small.</p><h4>Conclusion</h4><p>In this post, I shared my notes on Deep Learning Book Chapter 5: Machine Learning Basics notes. I will pick up with Estimators, Bias, and Variance in part II.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*Jcd5zOtD9YnONxU0" /></figure><h4>Reference</h4><ol><li>Ian Goodfellow, Yoshua, Bengio, and Aaron Courvlle. <em>Deep Learning</em> MIT Press, 2016. <a href="http://www.deeplearningbook.org/">http://www.deeplearningbook.org</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46fc4acf745d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WAYEMENT…. Calculus!]]></title>
            <link>https://medium.com/@dskswu/wayement-calculus-e7577fabd58f?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7577fabd58f</guid>
            <category><![CDATA[calculus]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Thu, 19 Jul 2018 16:31:46 GMT</pubDate>
            <atom:updated>2018-07-19T16:31:49.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*k_9dyrw7PTS0u5fvavvF_A.jpeg" /></figure><p>This blog post is my progress for day 6 in the <a href="https://github.com/llSourcell/100_Days_of_ML_Code">#100DaysofMLCode</a>. Going back to the basics with ML has helped me to make better sense of what I have learned. In this blog post I will be sharing my notes on <a href="http://www.deeplearningbook.org/contents/numerical.html">Deep Learning Book Chapter 4: Numerical Computation</a>. I’m definitely feeling the meme above.</p><p><strong><em>Please note that this post is for my future self to review the materials on this book without reading it all over again.</em></strong></p><p>Computing large numerical computation is common in ML. For example, Bayesian Statistics often requires computing very large multidimensional integrals.</p><h4>Overflow and Underflow</h4><ol><li>Underflow occurs when numbers<strong> near zero</strong> are <strong>rounded to zero</strong>.</li><li>Overflow occurs when numbers with <strong>large magnitude</strong> are approximated as <strong>∞</strong> or <strong>-∞</strong>.</li></ol><p>Softmax example of overflow and underflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*_xqQkRq76wPzuILT8swUTQ.png" /></figure><p>We have an issue if our x’s are very negative, because by exponentiating them we push them very close to zero, possibly resulting in underflow. Similarly, with huge x, we might get overflow. We can do a bit of <strong>normalization</strong> to overcome this: by subtracting max(x), we don’t change the computation, but we ensure that in the numerator everything is &lt;0, and in the denominator, at least one of the entries is exp(0)=1, which means we’re not going to get underflow!</p><h4>Poor Conditioning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*_Hy8LUisV73335Cmn6QfHg.png" /></figure><p>Conditioning refers to how rapidly a function changes with a small change in input. Rounding errors can rapidly change the output. Consider f(x)=A-1x — A ε <strong>Rnxn has a eigendecomposition.</strong> Its condition no. is , i.e. ratio of largest to smallest eigenvalue . When this large, the output is very sensitive to input error. Poorly conditioned matrices amplify pre-existing errors when we multiply by its inverse.</p><h4><strong>Gradient Based Optimization</strong></h4><p>Deep learning algorithms use optimization techniques all the time. We minimize loss, or error, or maximize <strong>some kind </strong>of score functions. Most optimization problems are often phrased in terms of <strong>minimize f</strong>(<strong>x</strong>). Maximization is accomplished by minimizing -<strong>f</strong>(<strong>x</strong>).</p><ol><li>Consider function <strong>y=f (x), x, y</strong> real number.</li><li>Derivative of function denoted: <strong>f</strong> ’<strong>(x)</strong> or as <strong>dy/dx</strong></li></ol><ul><li>Derivative <strong>f’(x)</strong> gives the slope of<strong> f (x)</strong> at point x</li><li>It specifies how to scale a small change in input to obtain a corresponding change in the output: <strong>f (x + ε) ≈ f (x) + ε f’ (x)</strong></li></ul><p>3. It tells how you make a small change in input to make a small improvement in <strong><em>y</em></strong>.</p><p>4. We know that <strong>f (x — ε sign (f’(x)))</strong> is less than <strong>f (x)</strong> for small <strong>ε</strong>. Thus we can reduce <strong>f (x) </strong>by moving x in small steps with opposite sign of derivative. This technique is called <strong>gradient descent</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*I9oziM19T96Nxbt9.png" /></figure><p>Points where <strong>f’(x) = 0 </strong>are called<strong> critical</strong> or <strong>stationary</strong> points. Types of critical points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*pZenw-WYJVVqaGkqiVq7Rg.gif" /></figure><p>When dealing with multiple inputs, we use <strong>partial derivatives</strong> to tell you how a multivariable function changes as you tweak just one of the variables in its input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/1*9ra9XZ9EM5E34XwotNPy1w.png" /></figure><p>The gradient generalizes the notion of derivative to case where the derivative is w.r.t. a vector: the gradient of <strong><em>f </em></strong><em>i</em>s the vector containing all the partial derivatives, denoted ∇x<em>f</em>(x).</p><p>The <strong>directional derivative</strong> is the rate at which the function changes at a point in the <strong>direction</strong> . The minimization occurs when the gradient points directly uphill, and negative gradient points directly downhill. It tell you how a multivariable function changes as you move along some vector in its input space.</p><p>The gradient points directly uphill, and the negative gradient points directly downhill . We can decrease <strong><em>f</em></strong> by moving in the direction of the negative gradient. This method is called <strong>Steepest descent, or </strong>gradient descent<strong>. </strong>Steepest descent proposes a new point:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*H9J5Utse-fShdjXiGfyh0w.png" /></figure><p>where <strong>ε</strong> is the learning rate, a positive scalar determining the size of the step. this method converges when every element of the gradient is zero(or very close to zero). The general concept of repeatedly making a small move in the locally best direction can be generalized to discrete spaces (<strong>hill climbing</strong>). In simple terms, we may be able to avoid iterative algorithm and jump to the critical point by solving the equation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*DTGelaryKCiLz42ZlbwwkA.png" /></figure><p>for <strong><em>x</em></strong>.</p><h4>Jacobian and Hessian Matrices</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*XR_-nMjuVTEsp_UnIsHS0w.png" /><figcaption>Source: <a href="https://github.com/AlejandroMllo/DeepLearningBookSummary/blob/master/Chapters/deep-learning-chapter-4.pdf">https://github.com/AlejandroMllo/DeepLearningBookSummary/blob/master/Chapters/deep-learning-chapter-4.pdf</a></figcaption></figure><h4>Constrained Optimization</h4><p>Limit the minimum and maximum values of <em>x </em>to have to be in some set, known as the feasible points. It’s common to impose a norm constraint such as ||<em>x|| &lt; = 1. </em>One way of satisfying this constraint is to use <strong>projected gradient descent</strong>, i.e. take the gradient update normally and project it back into the feasible set. Another way is to rewrite the constrained problem as a modified unconstrained problem. For example, if we want to find the minimum <em>x </em>for some <em>f</em>(<em>x</em>) such that <em>x</em> has unit norm, we could rewrite this as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/1*ZLmiSyPhuNfYrGUhgq11og.png" /></figure><p>and return:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*io9l0IFm_6lex23cSL1dAg.png" /></figure><p>as our solution.</p><h4>KKT approach</h4><p><strong>Karush–Kuhn–Tucker(KKT)</strong> approach provides a general framework for writing constrained optimization problems as unconstrained ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tha4uM7V5csFsXplvaHJEA.png" /><figcaption>Source: <a href="https://uclaacmai.github.io/deeplearning-book-notes/Ch4-Numerical-Computation">https://uclaacmai.github.io/deeplearning-book-notes/Ch4-Numerical-Computation</a></figcaption></figure><h4>Conclusion</h4><p>In this post, I shared my notes on Deep Learning Book Chapter 4: Numerical Computation Notes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*9ipPqyyhu4iWolyvoOahBg.gif" /></figure><h4><strong>Reference</strong></h4><ol><li>Ian Goodfellow, Yoshua, Bengio, and Aaron Courvlle. <em>Deep Learning</em> MIT Press, 2016. <a href="http://www.deeplearningbook.org/">http://www.deeplearningbook.org</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7577fabd58f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chapter 3: Probability and Information Theory]]></title>
            <link>https://medium.com/@dskswu/chapter-3-probability-and-information-theory-ecad8b1aeefe?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecad8b1aeefe</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Mon, 16 Jul 2018 20:09:44 GMT</pubDate>
            <atom:updated>2018-07-16T20:09:44.048Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/543/0*CYu-eDN2gyBbovux.png" /></figure><p>This blog post is my progress for day 5 in the <a href="https://github.com/llSourcell/100_Days_of_ML_Code">#100DaysofMLCode</a>. Going back to the basics with ML has helped me to make better sense of what I have learned. In this blog post I will be sharing my notes on <a href="http://www.deeplearningbook.org/contents/prob.html">Deep Learning Book Chapter 3: Probability and Information Theory</a>.</p><p><strong><em>Please note that this post is for my future self to review the materials on this book without reading it all over again.</em></strong></p><p>Everything thing we observe around us presents itself to some degree of uncertainty. <a href="https://www.britannica.com/science/probability-theory"><strong>Probability theory</strong></a>, a branch of mathematics concerned with the analysis of random phenomena. It provides a mathematical framework for representing uncertainty. In the application of Artificial Intelligence, probability theory is used in two ways:</p><ul><li>It specifies how AI should handle reasoning.</li><li>Use probability and statistics to analyze the behaviour of proposed AI systems.</li></ul><p>Why Probability? ML makes heavy use of probability theory. We often ask our ML algorithm how likely a given result is. The answer is often expressed in terms of probability. There are two types of probability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r26IMjrebWUPaKYVcAEyyQ.png" /></figure><p><strong>Frequentist Probability:</strong> defines probability of some event in terms of the relative frequency with which the event tends to occur. It refers to the rates at which events occur.</p><p><strong>Bayesian Probability:</strong> defines probability in more subjective terms — as a measure of the strength of your belief regarding the true situation. It refers to more qualitative levels of certainty.</p><p>A random variables, usually written X, is a <strong>variable</strong> whose possible values are numerical outcomes of a <strong>random</strong> phenomenon. There are two types of random variables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THXqVXlDlE6loHXykzihCA.png" /></figure><ul><li><strong>Discrete:</strong> The number of states are finite or countably infinite. <em>Example</em>: Let <em>X</em> represent the sum of two dice.</li><li><strong>Continuous:</strong> It is associated with a real value. The probability distribution of a continuous random variable is shown by a <strong>density curve</strong>. The probability that X is between an interval of numbers is the area under the density curve between the interval endpoints. The probability that a <strong>continuous</strong> <strong>random variable</strong> <em>X</em></li></ul><p>Probability distribution is random variable or a set of random variables that take on each of its possible states. We describe probability distributions on whether the variables is discrete or continuous. The chart above is a visualization of both discrete and continuous distribution.</p><p><strong>Probability mass function (PMF)</strong> gives you probabilities for discrete random variables. Its counterpart is the <strong>probability density function (PDF)</strong>, which gives probabilities for continuous random variables. The PMF equations is P(X = x). It basically means “the probability that X takes on some value x”. By itself it’s not useful; What’s more useful is an equation that tells the probability of an event happening.</p><p><strong>Example: </strong>P(X=1) = 0.3 * 0.3.</p><p>A more formal definition is:</p><p><em>The probability mass function, f(x) = P(X = x), of a discrete random variable X has the following properties:</em></p><ol><li><em>All probabilities are positive: fx(x) ≥ 0.</em></li><li><em>Any event in the distribution (e.g. “scoring between 20 and 30”) has a probability of happening of between 0 and 1 (e.g. 0% and 100%).</em></li><li><em>The sum of all probabilities is 100% (i.e. 1 as a decimal): Σfx(x) = 1.</em></li><li><em>An individual probability is found by adding up the x-values in event A. P(X Ε A) =</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/80/1*scyqg1gVtHLFpxWI5SN3BQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*WR-ZKEDJ2Mt1juj-BHQz9Q.png" /></figure><p>The <strong>probability density function </strong>shown above shows all possible values for Y, which for this case, has an infinite amount of possibilities. For example, the random variable Y could equal 175 cm, 180 cm or 186.9999999999 cm. We can use PDF to answer a question like: <strong>What is the probability that a person height will be between 175 cm and 186 cm </strong>The<strong> w</strong>ritten in notation, the question becomes: P(175 &lt; Y &lt; 186).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*c70v8eL7ewd167IYMgGEHw.png" /></figure><p>Looking at the shaded area. it looks like it’s about 75 percent. So, P(175 &lt; Y &lt; 186) = 75%.</p><p>Marginal probability or marginal probability distribution is a subset of a collection of random variables.</p><p><em>If X and Y are discrete random variables and f (x,y) is the value of<br>their joint probability distribution at (x,y), the functions given by:<br>g(x) = Σ</em>y<em> f (x,y) and h(y) = Σ</em>x<em> f (x,y) are the marginal distributions of X and Y , respectively.</em></p><p>A frequency table can be used to find the margins of probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*vAxnFchNHJNWzdUW.jpg" /><figcaption>A <strong>marginal distribution</strong> gets it’s name because it appears in the <em>margins</em> of a probability distribution table.</figcaption></figure><p>Marginal distributions follow a couple of rules:</p><ul><li>The distribution must be from bivariate data. Bivariate is just another way of saying “two variables,” like X and Y. In the table above, the random variables i and j are coming from the roll of two dice.</li><li>A marginal distribution is where you are only interested in <em>one</em> of the random variables . In other words, either X <strong>or </strong>Y. If you look at the probability table above, the sum probabilities of one variable are listed in the bottom row and the other sum probabilities are listed in the right column. So this table has <em>two </em>marginal distributions.</li></ul><p>Conditional probability is one of the most important concept in probability theory. It’s the probability that one event occurring with some relationship to one or more other events. For example:</p><ul><li>Event A is that it is snowing outside, and it has a 0.3 (30%) chance of snow today.</li><li>Event B is that you will need to go outside, and that has a probability of 0.5 (50%).</li></ul><p>A conditional probability would look at these two events in relationship with one another, such as <strong>the probability that it is both snowing <em>and </em>you will need to go outside. </strong>The formula for conditional probability is: P(B|A) = P(A and B) / P(A). You can also rewrite as: P(B|A) = P(A∩B) / P(A).</p><p>The chain rule of conditional probabilities is really important . It permits the calculation of any member of the joint distribution of a set of random variables using only <strong>conditional probabilities.</strong></p><p>Iindependence extends to dealing with collections of more than <strong>two </strong>events or <strong>random variables</strong>, in which case the events are pairwise <strong>independent</strong> if each pair are <strong>independent</strong> of each other, and the events are mutually <strong>independent</strong> if each event is <strong>independent</strong> of each other combination of events. two random variables are independent if <strong>either of the following statements are true</strong>:</p><ol><li>P(x|y) = P(x), for all values of X and Y.</li><li>P(x∩y) = P(x) * P(y), for all values of X and Y.</li></ol><p><strong>The first statement, P(x|y) = P(x), for all values of X and Y, </strong>is stating “the probability of x, given y, is x.” In other words, knowing y should make no difference on the probability, x — it’s still going to be just x no matter what the value of y.</p><p>Conditional independence are two events A and B are conditionally independent given a third event C if P(A^B| C) = P(A|C)P(B|C) or, equivalently, P(A|B^C) = P(A|C).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*XYbI9sgxP9IMYx9RsAdJHQ.png" /></figure><p>The pictures on the left represent the probabilities of events A, B and C by the areas shaded red, blue and yellow respectively with respect to the total area. In both examples A and B are conditionally independent given C because: P(A^B| C) = P(A|C)P(B|C) BUT A and B are NOT conditionally independent given ~C, as: P(A^B|~ C) != P(A|~C)P(B|~C)</p><p>Expectation is the average value of some random variable X. It is also called the expected value. The mean, expected value, or expectation of a random variable X is written as E(X) or µX. If we observe N random values of X, then the mean of the N values will be approximately equal to E(X) for large N. The expectation is defined differently for continuous and discrete random variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*iH-ihNOG2ZbTV-_MXybwkg.png" /></figure><p>Variance of a random variable X is a measure of how spread out it is. Are the values of X clustered tightly around their mean, or can we commonly observe values of X a long way from the mean value? The variance measures how far the values of X are from their mean, on average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*5ThVBmalp3w60EE_nCdDtA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/0*S9L4sIEZT9HcQXbw.jpg" /><figcaption>Lower the variance, more peakier the distribution. The square root of variance is called <strong>standard deviation</strong>.</figcaption></figure><p>Covariance is a measure of the association or dependence between two random variables X and Y . Covariance can be either positive or negative. (Variance is always positive.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*t9H7NcxvQQcKoaEpCeDL7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*B5bKZfQyqJEqcOZr.gif" /></figure><p>The <strong>correlation </strong>coefficient of X and Y is a measure of the linear association between X and Y . It is given by the covariance, scaled by the overall variability in X and Y . As a result, the correlation coefficient is always between −1 and +1, so it is easily compared for different quantities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*JsUeOi2tR0vUsqQxTOqlhg.png" /></figure><p>The correlation measures linear association between X and Y . It takes values only between −1 and +1, and has the same sign as the covariance. The correlation is ±1 if and only if there is a perfect linear relationship between X and Y , i.e. corr(X, Y ) = 1 ⇐⇒ Y = aX + b for some constants a and b. The correlation is 0 if X and Y are independent, but a correlation of 0 does not imply that X and Y are independent.</p><p>Common probability distributions describe the probability of each outcome. They come in many shapes, but in only one size: probabilities in a distribution always add up to 1.</p><p><strong>Bernoulli Distribution:</strong> The Bernoulli distribution is the probability distribution of a random variable having the probability density function. Its single binary random variable controlled by the probability <em>ϕ</em></p><pre><em># Simulate data</em><br>np<strong>.</strong>random<strong>.</strong>seed(123)<br><br>nobs <strong>=</strong> 100<br>theta <strong>=</strong> 0.3<br>Y <strong>=</strong> np<strong>.</strong>random<strong>.</strong>binomial(1, theta, nobs)</pre><pre><em># Plot the data</em><br>fig <strong>=</strong> plt<strong>.</strong>figure(figsize<strong>=</strong>(7,3))<br>gs <strong>=</strong> gridspec<strong>.</strong>GridSpec(1, 2, width_ratios<strong>=</strong>[5, 1]) <br>ax1 <strong>=</strong> fig<strong>.</strong>add_subplot(gs[0])<br>ax2 <strong>=</strong> fig<strong>.</strong>add_subplot(gs[1])<br><br>ax1<strong>.</strong>plot(range(nobs), Y, &#39;x&#39;)<br>ax2<strong>.</strong>hist(<strong>-</strong>Y, bins<strong>=</strong>2)<br><br>ax1<strong>.</strong>yaxis<strong>.</strong>set(ticks<strong>=</strong>(0,1), ticklabels<strong>=</strong>(&#39;Failure&#39;, &#39;Success&#39;))<br>ax2<strong>.</strong>xaxis<strong>.</strong>set(ticks<strong>=</strong>(<strong>-</strong>1,0), ticklabels<strong>=</strong>(&#39;Success&#39;, &#39;Failure&#39;));<br><br>ax1<strong>.</strong>set(title<strong>=</strong>r&#39;Bernoulli Trial Outcomes $(\theta=0.3)$&#39;, xlabel<strong>=</strong>&#39;Trial&#39;, ylim<strong>=</strong>(<strong>-</strong>0.2, 1.2))<br>ax2<strong>.</strong>set(ylabel<strong>=</strong>&#39;Frequency&#39;)<br><br>fig<strong>.</strong>tight_layout()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*MMDRkKY8XmgNuajg.png" /></figure><p><strong>Multinoulli Distribution:</strong> The <strong>multinomial distribution</strong> arises from an extension of the binomial experiment to situations where each trial has k ≥ 2 possible outcomes.</p><pre><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br><strong>from</strong> <strong>scipy</strong> <strong>import</strong> stats<br><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br><br>x = np.linspace(0, 5, 1000)<br>fig,ax = plt.subplots(1,1)<br><br>linestyles = [&#39;:&#39;, &#39;--&#39;, &#39;-.&#39;, &#39;-&#39;]<br>deg_of_freedom = [1, 2, 3, 4]<br><strong>for</strong> df, ls <strong>in</strong> zip(deg_of_freedom, linestyles):<br>  ax.plot(x, stats.chi2.pdf(x, df), linestyle=ls, label=r&#39;$df=<strong>%i</strong>$&#39; % df)<br><br>plt.xlim(0, 5)<br>plt.ylim(0, 0.5)<br><br>plt.xlabel(&#39;$\chi^2$&#39;)<br>plt.ylabel(r&#39;$f(\chi^2)$&#39;)<br>plt.title(r&#39;$\chi^2\ \mathrm{Distribution}$&#39;)<br><br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/0*Mf9IWrxFbEOSsmhu.png" /></figure><p><strong>Gaussian Distribution</strong>: Is commonly called the “<strong>normal distribution</strong>” and is often described as a “bell-shaped curve”. If the probability of a single event is p = and there are n = events, then the value of the <strong>Gaussian distribution</strong> function at value x = is x 10^ .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*-CQ31FFhW2zK4KUngY87rQ.png" /></figure><p><strong>Exponential and Laplace Distributions:</strong> The difference between two independent identically <strong>distributed exponential</strong> random variables is governed by a <strong>Laplace distribution</strong>, as is a Brownian motion evaluated at an <strong>exponentially distributed</strong> random time. We can place a sharp peak anywhere via the <strong>Laplace</strong> distribution, which adds in a translation parameter μ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*aAer3TcmxlCMb4Kb5-4wjg.png" /></figure><pre>loc, scale = 0., 1.<br>s = np.random.laplace(loc, scale, 1000)</pre><pre>import matplotlib.pyplot as plt<br>count, bins, ignored = plt.hist(s, 30, normed=True)<br>x = np.arange(-8., 8., .01)<br>pdf = np.exp(-abs(x-loc)/scale)/(2.*scale)<br>plt.plot(x, pdf)</pre><pre>g = (1/(scale * np.sqrt(2 * np.pi)) * np.exp(-(x — loc)**2 / (2 * scale**2)))<br>plt.plot(x,g)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*yH3hF_rJMdypL7GPveD4mA.png" /></figure><p><strong>Dirac Distribution</strong> and <strong>Empirical Distribution </strong>is a<strong> </strong>way to specify a probability distribution where we put all of the probability mass on one point, and zero everywhere else, but the function is still a valid probability because it integrates to one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*FNnQIYNYcKoxNFcTMxDrkw.gif" /></figure><p><strong>Mixtures of distributions:</strong> A way of combining probability distributions. In this case, <strong><em>x </em></strong>can be thought of as a random variable that is sampled from a variable distribution. For example, you could sample <strong><em>x</em></strong> from a unit Gaussian with <strong>0</strong> mean with probability <strong>0.5</strong>, and sample it from a unit Gaussian with 1 mean with probability <strong>0.5</strong>. Then <em>x </em>would be a random variable that comes from a mixture of 2 Gaussians. In general, the probability of <strong><em>x</em></strong> is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*N0tQkZp0IjEG6od8HyJKPA.png" /></figure><p>It introduces the concept of a <strong>latent variable</strong>, which we cannot observe directly<strong><em>. </em></strong>In the above case, <strong><em>c</em></strong> is a latent variable and cannot be directly observe, but it affects <strong><em>x</em></strong>’s probabilities.</p><p>Gaussian mixture model (GMM) is very powerful mixture model, where each of the component distributions is a gaussian, with a <strong>prior probability</strong> associated with each gaussian. The word “prior” indicates that it expresses the model’s beliefs about <strong><em>c</em></strong> <em>before</em> it has observed <strong><em>x</em></strong>. Gaussian mixture model is a <strong>universal approximator</strong>.</p><p><strong>Useful properties of Common Functions </strong>are functions that are often employed while working with the probability distributions in deep learning models, one such being the <strong>logistic sigmoid</strong>, given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*_pP4TLRVkflvRDVLAaXgjg.png" /></figure><p>A <strong>sigmoid function</strong> is a <a href="https://en.wikipedia.org/wiki/Function_(mathematics)">mathematical function</a> having a characteristic “S”-shaped curve or <strong>sigmoid curve</strong>.</p><p>Another commonly used function is the <strong>softplus function</strong>: The derivative of the softplus function is the logistic function. ReLU and Softplus are largely similar, except near 0(zero) where the softplus is enticingly smooth and differentiable. It’s much easier and efficient to compute ReLU and its derivative than for the softplus function which has log(.) and exp(.) in its formulation.</p><p>Range(0,∞)</p><p>The softplus function can be approximated by max function (or hard max) i.e max(0,x+N(0,1))</p><p><strong>Bayes Theorem </strong>interpret <strong>y</strong> as being the class labels and <strong><em>x</em></strong> as being our feature vectors. Bayes theorem tells us that we can obtain the posterior probabilities of yy given the likelihood <strong>p(<em>x</em>∣<em>y</em>)</strong>, and the priors <strong>p(<em>y</em>)</strong>, and the evidence <strong>p(<em>x</em>)</strong>. Given these, we can estimate the most likely class that x belongs to by computing argmaxy <strong>p(y∣x)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/0*Rk4zb61dYE80cyOe.png" /></figure><p><strong>Technical Details of Continuous Variables measure theory</strong> provides characterization of the sets of sets we can compute probabilities on. M<strong>easure zero</strong> is a set of negligibly small size. A property that holds <strong>almost everywhere</strong> holds throughout all space except for a set of measure zero.</p><p><strong>Information Theory </strong>is branch of applied mathematics that resolves around quantifying how much information is present in a signal. It was first invented to study sending messages from discrete alphabets over noisy channel such as radio transmission. information theory tells how to design optimal codes and calculate the expected length of messages sampled from speciﬁc probability distributions using various encoding schemes. In ML, we can applied this theory to continuous variables where some message length interpretations does not apply.</p><ul><li><strong>Information: </strong>I(<em>x</em>)=-P(<em>x</em>)</li><li><strong>Entropy:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*w1ulO63Xx_xTpKse5OmHvg.png" /></figure><ul><li><strong>KL divergence:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*o0heRD4BRQtCG0LdF6Ih6Q.png" /></figure><p><strong>Structured Probabilistic Models: </strong>Machine learning algorithms often involve probability distributions over a very large number of random variables. Often, these probability distributions involve direct interactions between relatively few variables.</p><p><strong>Directed</strong> models represent the factorization into probability distribution where there is a flow of influence between nodes. Directed models contain one factor for each of the variables and the factor consists of a conditional probability distribution over the factor, given its parents</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*UlvAmgqAkt-IbQFWW7dIEg.png" /></figure><p><strong>Undirected models</strong> represent factorizations into a set of functions, usually <strong>not </strong>probability distributions of any kind, i.e. there is no constraint that the factor must sum to 1, but must be non-negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*1ilQLwHOLN6zXxxt4Z893g.png" /></figure><h3>Summary</h3><p>In this post, I shared my notes on Chapter 3 Probability and Information Theory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/0*4PxgD2K5kM9pRCTa.png" /></figure><p><strong>Reference</strong></p><ol><li>Ian Goodfellow, Yoshua, Bengio, and Aaron Courvlle. <em>Deep Learning</em> MIT Press, 2016. <a href="http://www.deeplearningbook.org/">http://www.deeplearningbook.org</a></li><li><a href="http://matatat.org/p-values-statistical-testing.html">http://matatat.org/p-values-statistical-testing.html</a></li><li><a href="https://github.com/ChadFulton/tsa-notebooks/blob/master/bernoulli_trials_bayesian.ipynb">https://github.com/ChadFulton/tsa-notebooks/blob/master/bernoulli_trials_bayesian.ipynb</a></li><li><a href="https://uclaacmai.github.io/deeplearning-book-notes/Ch3-Probability-Information-Theory">https://uclaacmai.github.io/deeplearning-book-notes/Ch3-Probability-Information-Theory</a></li><li><a href="https://github.com/AlejandroMllo/DeepLearningBookSummary/blob/master/Chapters/deep-learning-chapter-3.pdf">https://github.com/AlejandroMllo/DeepLearningBookSummary/blob/master/Chapters/deep-learning-chapter-3.pdf</a></li><li><a href="https://github.com/dalmia/Deep-Learning-Book-Chapter-Summaries/blob/master/Chapter%203%20-%20Probability%20and%20Information%20Theory.ipynb">https://github.com/dalmia/Deep-Learning-Book-Chapter-Summaries/blob/master/Chapter%203%20-%20Probability%20and%20Information%20Theory.ipynb</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecad8b1aeefe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Algebra! Ain’t Nobody Got The Time For That]]></title>
            <link>https://medium.com/@dskswu/linear-algebra-aint-nobody-got-the-time-for-that-ee0c98cfdf25?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee0c98cfdf25</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Fri, 13 Jul 2018 20:51:09 GMT</pubDate>
            <atom:updated>2018-07-13T20:51:09.746Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*F-ggLok6h3SvekHzx6pk9Q.jpeg" /></figure><p>This post is a continuation of my previous post <a href="https://medium.com/@dskswu/scalars-vectors-and-matrices-oh-my-9054d2bda1f9"><strong>Scalars, Vectors and Matrices, oh my</strong></a>.<strong> </strong>We will dive right in and begin with linear dependence and span.</p><h4>Linear Dependence and Span</h4><p>The first thing to ask when are faced with system of linear equations: What is the number of solutions? Three cases can represent the number solutions for system of linear equations <strong>Ax=b. </strong>Can<strong> </strong>there ever be more than one solution and less than a infinite number of solutions?<strong> </strong>No, two lines cannot cross more than once but can be either parallel or superimposed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*v54qCvGasNZV7g7oRNCyOw.jpeg" /></figure><p>To better understand this concept, let’s take a look at a visual representation of two dimensions and two equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0RISQunPkJvuHatLGHtag.png" /></figure><p>Looking at the figure above, you can see that the solutions correspond to the intersection of the lines. In no solution, the two lines run parallel and never cross. The second option is that the lines cross once. And finally, the last option of lines are is superimposed.</p><p>From this point forward, I will paste my notes in png format since LATex is not compatible with this platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*6ifhVOq5QlUzRliJPCEcKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*iKju73DlYm-FEjKN2xnY5w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*5VOp12AQZBBI0V9uFVHpWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*f9vypi-Ym75Iher2SIZejw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*9Js40Iz_kd9AbusB5dt8ag.png" /></figure><h4>Norms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*baILEiXYFVSehgQjqhIznA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*L6eFsEuZKFoCJNf9Ljgo3Q.png" /></figure><h4>Special Matrices</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*LhbyK2pg2eA1Y8moCh6Tvg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*ELQme5V7GmITUnal0mje_A.png" /></figure><h4>Eigendecomposition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*STJ5RaxmPBB9aSJSCj0XyA.png" /></figure><h4>Optimization with Eigenvalues and Eigenvectors</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*Nk-aJpbMmDc4X2gHrGr0Jg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n9GVwmjsolqsJnUK5wUwKw.png" /></figure><h4>Singular Value Decomposition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*xJYS9QTS8BtvylJpxlcbBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*-lCN506xJ1aY1bQIkB_uCg.png" /></figure><h4>Trace &amp; Determinant</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*jeBAtoWlCRKp1IWstkxXSA.png" /></figure><h4>Final Words</h4><p>This wraps up Chapter 2. If any errors are found, please email me at dskswu@gmail.com. Meanwhile follow me on my twitter <a href="https://twitter.com/William33712308">here</a>. The log for journey can be found<a href="https://github.com/MinuteswithMetrics/100-Days-Of-ML-Code/blob/master/100DaysOfML_log.md"> here</a>.</p><h3><strong>Reference</strong></h3><pre>@book{Goodfellow-et-al-2016,<br>    title={Deep Learning},<br>    author={Ian Goodfellow and Yoshua Bengio and Aaron Courville},<br>    publisher={MIT Press},<br>    note={\url{http://www.deeplearningbook.org}},<br>    year={2016}<br>}</pre><p><a href="https://store.chrisalbon.com/">Chris Albon</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee0c98cfdf25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scalars, Vectors and Matrices, oh my!]]></title>
            <link>https://medium.com/@dskswu/scalars-vectors-and-matrices-oh-my-9054d2bda1f9?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9054d2bda1f9</guid>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Wed, 11 Jul 2018 04:08:52 GMT</pubDate>
            <atom:updated>2018-07-11T04:08:52.212Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*51NRhb1t4ip6Pdo7cZKd1A.jpeg" /></figure><p><strong>Day 3</strong> and 4 was spent studying the foundational and math side of machine learning. Machine learning can be applied without mathematics. However, applying machine learning is more effective knowing what these systems are doing under the hood. It helps to work through introductory books to gain an understanding of the foundational side of machine learning.</p><p>On day day 3 and 4, I read<a href="http://www.deeplearningbook.org/contents/linear_algebra.html"> <strong>Deep Learning Chapter 2</strong>: <strong>Linear Algebra</strong></a> written by Ian Goodfellow. Linear Algebra is everywhere in machine learning and can be seen in the basic materials. It’s widely used throughout engineering and science. Linear algebra is used to frame optimization algorithms and solving linear systems of constraints. So this blog post is my notes on topics needed to understand what will come next.</p><h4><strong>scalars</strong>, Vectors and Matrices</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*r6vg0CSPiPN7fJ-9ku1lVg.jpeg" /></figure><ul><li>A scalar is a single number</li><li>A vector is an array of numbers.</li><li>A matrix is a 2-D array</li><li>A tensor is A tensor is a<em> n</em>-dimensional array with <strong><em>n</em> &gt;2</strong></li><li>scalars are written in lowercase and italics. For instance: <em>n</em></li><li>vectors are written in lowercase, italics and bold type. For instance: <strong><em>x</em></strong></li><li>matrices are written in uppercase, italics and bold. For instance: <strong><em>X</em></strong></li></ul><p><strong>Code example</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*pt-Zm_TZ64qENMHxsOt0xQ.jpeg" /></figure><p><strong>Matrix</strong> operations can be applied to linear algebra objects. These operations are addition, subtraction, transpose, multiplication, and inverse.</p><p><strong>Addition : </strong>adding 2 vectors requires them to have the same number of elements and equal shape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*9EKi6B4m1-3rBrUZRM47bw.jpeg" /></figure><p><strong>Subtraction:</strong> subtracting 2 vectors requires them to have the same number of elements and equal shape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*XQslCXt3I1GoPeJ3d-NqPA.jpeg" /></figure><p><strong>Transpose: </strong>Transpose of a Matrix is an operator which flips a matrix over its main diagonal like a mirror image. This can be done by calling numpy.transpose function or T method in numpy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*_UMA3UJySHT81IeO_4z3Sg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*Eqm2m4rL4dW6f86IlTs_7A.jpeg" /></figure><p><strong>Multiplication: </strong>NumPy uses numpy.dot function for multiplication of both vectors and matrices. Matrix multiplication is <strong>not commutative</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*zGouH4ld2j4dnuofRllZkQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*_SB6g92EYheWxAq9UcztXw.png" /></figure><p><strong>Inverse: </strong>Inverse operation only applies to square matrices(matrix with same number of columns and rows). To compute inverse using numpy we need to use numpy.linalg.inv function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*sHK5kLblMq8QgBlQfoIeng.jpeg" /></figure><p><strong>Transpose: </strong>converting a row vector to a column vector and vice versa. It’s basically a mirror image. This can be done by calling numpy.transpose function or T method in numpy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*6_7n3XofOQToNj3SQs_YAw.jpeg" /></figure><h3>Summary</h3><p>In this post, I discussed mathematical objects of Linear ALgebra that is used in ML. I briefly discussed addition, subtraction, transpose, multiplication, and inverse. Furthermore, you have learned important properties of matrices and how we can enable efficient computations. Day three and four of 100 day ML code challenge was spent reviewing the basic foundations. In the next few days I plan to complete <a href="http://www.deeplearningbook.org/contents/part_basics.html">Part I: Applied Math and Machine Learning Basics</a> and document my progress.</p><h4>Final Words</h4><p>It was fun discussing part I of Linear Algebra. Stay tune for Part II where I discuss types of matrices. If any errors are found, please email me at dskswu@gmail.com. Meanwhile follow me on my twitter <a href="https://twitter.com/William33712308">here</a>. The log for journey can be found<a href="https://github.com/MinuteswithMetrics/100-Days-Of-ML-Code/blob/master/100DaysOfML_log.md"> here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*kM2tF_0wPVrsyaM8xDm_qg.jpeg" /><figcaption>The Wiz</figcaption></figure><h3>Resources</h3><pre>@book{Goodfellow-et-al-2016,<br>    title={Deep Learning},<br>    author={Ian Goodfellow and Yoshua Bengio and Aaron Courville},<br>    publisher={MIT Press},<br>    note={\url{http://www.deeplearningbook.org}},<br>    year={2016}<br>}</pre><p><a href="https://legacy.gitbook.com/book/leonardoaraujosantos/artificial-inteligence/details">Artificial Inteligence · GitBook</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9054d2bda1f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Siraj Raval: 100 Days of ML Coding Challenge]]></title>
            <link>https://medium.com/@dskswu/siraj-raval-100-days-of-ml-coding-challenge-7b985d8a4e82?source=rss-1ed8f83521bf------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b985d8a4e82</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Willam Green]]></dc:creator>
            <pubDate>Tue, 10 Jul 2018 04:02:13 GMT</pubDate>
            <atom:updated>2018-07-10T04:12:59.982Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*h7Soza5ysqQdx9U1OPU5eg.jpeg" /><figcaption>“ Machine Learning is the most transformative technology of our time” Siraj Raval</figcaption></figure><h3><strong>Challenge Accepted</strong></h3><p>On Friday, July 06, 2018, I embarked on 100 day journey to expand my knowledge and sharpen my machine learning (ML) skills. To start the challenge, I began by setting up all the required resources and tools that I need to complete this challenge and also selected a few projects that gain my interest.</p><p><strong>Day 1</strong> began<strong> </strong>with the <a href="https://www.drivendata.org/competitions/2/warm-up-predict-blood-donations/">Warm Up: Predict Blood Donations</a> dataset. This is a good place to start and to work with a less complex dataset. The goal for this <a href="https://www.drivendata.org/">DrivenData</a> challenge is to predict whether or not a donor will give blood the next opportunity. The figure below is the process our team used during the <a href="https://www.kaggle.com/c/data-science-bowl-2018">2018 Data Science Bowl</a>. It is important to note that this is not the say all be all way of tackling a problem. However, it is good tool that I use to stay on course</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OcupeV_f3-zwgdW7ztujqg.jpeg" /><figcaption>My ML process map</figcaption></figure><p>My strategy for this particular problem was to run several different ML models and assess their performance using cross validation scores. The range between accuracy scores was assessed by taking the mean and comparing against each model used. I used a grid search to locate the optimal parameters for the model. The average score was used as a metric to analyze model’s performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*p0vyOtxZu70GreCHdHmr4Q.png" /><figcaption>Visualization of Model differences</figcaption></figure><p>The ROC curve of the model was used to make a comparison of the models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*ycCeHRMmFy3IrVcPKbyVCQ.png" /></figure><p>Looking at the charts above, it was clear that Random Forest (RF) out perform the other models. Using the RF model, I was able to place 935/4532. This is a good start, but there is more that can be done to improve our outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*zERopHABls0d5fwonqLTNA.jpeg" /></figure><h4><strong>Final Words</strong></h4><p>It was fun to play around with this dataset. If any errors are found, please email me at dskswu@gmail.com. Meanwhile follow me on my twitter <a href="https://twitter.com/William33712308">here</a>. The log for journey can be found<a href="https://github.com/MinuteswithMetrics/100-Days-Of-ML-Code/blob/master/100DaysOfML_log.md"> here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b985d8a4e82" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>