<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Machine Learning Algorithms from scratch - Medium]]></title>
        <description><![CDATA[This publication is a group of important Machine learning algorithms which are implemented from scratch in Python. Maths behind every frequently used machine learning algorithm is discussed and made easy. - Medium]]></description>
        <link>https://medium.com/machine-learning-algorithms-from-scratch?source=rss----2f01df7a83b0---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Machine Learning Algorithms from scratch - Medium</title>
            <link>https://medium.com/machine-learning-algorithms-from-scratch?source=rss----2f01df7a83b0---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:30:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/machine-learning-algorithms-from-scratch" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Digit Recognition from 0–9 using Deep Neural Network from scratch]]></title>
            <link>https://medium.com/machine-learning-algorithms-from-scratch/digit-recognition-from-0-9-using-deep-neural-network-from-scratch-8e6bcf1dbd3?source=rss----2f01df7a83b0---4</link>
            <guid isPermaLink="false">https://medium.com/p/8e6bcf1dbd3</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[digit-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[pavan kalyan urandur]]></dc:creator>
            <pubDate>Fri, 22 Feb 2019 20:03:43 GMT</pubDate>
            <atom:updated>2019-02-27T17:11:49.138Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*MEt6kC9qyI2y2qQIaIte5w.jpeg" /></figure><p>In Machine learning, Artificial Neural Networks (ANN) play a major role in showcasing the power of statistics and mathematics to solve complex and non-linear problems. ANN has been there in the field of machine learning and Artificial Intelligence for a long time, but recent improvements in computational power and big data is helping them to show how powerful they are. They can be used to solve both supervised and unsupervised, classification and regression problems, computer vision etc. In this article, we shall be implementing an ANN from scratch and apply it to solve a simple problem of detecting digits from 0–9.</p><p>Neural Network is similar to logistic regression (perceptron) but with more layers and hidden units. Here’s is what a single layer perceptron (logistic regression) looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*Hm3q4CM-uDVGR31Uu2Atow.png" /></figure><p>Here x1, x2,x3 are the features of the input, w1, w2, w3 are the parameters, b is the bias. Now Z= w1 X1+w2 X2+w3 X3+b. g(z) is the activation function. There are many available activation functions like sigmoid, tanh, relu, leaky relu etc. If the problem is a classification problem, the most common choice of activation function is sigmoid. The formulae and plots of these activation functions are:</p><p><strong>Sigmoid:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*a12RZYZ9y7bpibt9WChuBw.png" /></figure><p><strong>Tanh:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*UVH4I5v7-hfzxnN7twDcAg.png" /></figure><p><strong>Relu (Rectified Linear Unit):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*QQkLph_hxQ2WXBbq3gUk1Q.png" /></figure><p>Similar to the perceptron model, a simple one-layer neural network looks like :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*yGWOw0V4bOBxlA6V6I_VSg.png" /></figure><p>The input layer is where the features of the input features (X) is fed, the Hidden layer will be the weighted sum of the inputs with parameters (W) followed by an activation function. There will always be bias (b) for each hidden layer. The output of the neural network will be the weighted sum of outputs of previous hidden layer followed by an activation function. For a two-class classifier problem, output layer will be the probability that given training example belongs to the class (cᵢ).</p><p>Similarly, a 4 layered deep neural network looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2vEZQVp8nMlNwr8ovpyKA.jpeg" /></figure><p>Now for a single-layered neural network, at hidden layer:</p><p>Z₁= W₁ . X+b₁, where Z₁ is the weighted sum of inputs and b₁ is the bias.</p><p>X is the input matrix where each training example is stacked horizontally via columns. So dimensions of X are (Nₓ,m) where Nₓ is the number of features, m is the number of training examples. If the number of units in the hidden layer is N₁, then dimensions of W₁ will be (N₁, Nₓ). So dimensions of Z₁ will be (N₁,m). b₁ is a scalar quantity and when we add it in python, it will be added by broadcasting rules. The output of the hidden layer is the result of passing Z₁ to the activation function. Similarly for an N layer neural network, at any hidden layer, dimensions of Wₙ will be (Nₙ, Nn -1).</p><p>So g(Z) can be any activation function such as RELU or sigmoid. In practice, we always choose RELU activation function for hidden layers and sigmoid function for output layers. This is because using sigmoid will lead to vanishing gradients. It means derivative of sigmoid dg(z)=g(z)*(1-g(z)). So if z is large, g(z) tends to 1. So the derivative becomes 0. Similarly, if z is small, g(z) tends to 0 leading to derivative becoming 0 again.</p><p><strong>Steps involved in Developing a NN model:</strong></p><ol><li><strong>Architecture: </strong>Choose the neural network architecture i.e number of hidden layers, number of units in hidden layers. In fact, these two are the hyperparameters which are needed to be tuned for better results and improved accuracy.</li><li><strong>Parameter Initialization:</strong> Initialize parameters randomly. While implementing, we use dictionaries to store parameters and gradients. i.e W₁ is equivalent to parameters[“W”+str(1)] and dW₁ is equivalent to grads[“dW”+str(1)].In python, to initialize parameters, we use:</li></ol><pre>parameters[&#39;W&#39; + str(l)] = np.random.randn(layer_dims[l],layer_dims[l-1])*0.01<br>parameters[&#39;b&#39; + str(l)] = np.zeros((layer_dims[l],1))</pre><p><strong>3. Forward propagation</strong>: Once weights are initialized, input X is propagated through hidden units at each layer and produce the output yhat. It usually contains two steps:</p><p>3. a: Linear Forward: At any layer l,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*6czJwpRcYZitc_ZIVIBeLw.png" /></figure><p>where A[0]=X.</p><p>3.b: Linear-Activation Forward: Result of Linear Forward step is processed into activation function to produce the output of the layer. Activation function at hidden layers is generally chosen as RELU to avoid vanishing gradient concept and activation function at the output layer is sigmoid as sigmoid’s range will always be within the range of [0,1] which can be interpreted in a probabilistic way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*eO5V8owltxLRIm5D-Lc82w.png" /></figure><p><strong>4.</strong> <strong>Compute Cost</strong>: We use cost function which is used in logistic regression (Cross-Entropy cost). Once we have final layer’s output Aˡ,</p><p>Cost= (-1/m)* (y* log(Aˡ)+(1-y)*log(1-Aˡ)). where y is the true output and Aˡ is the output of the network. m is the number of training examples.</p><p><strong>5. Backward propagation: </strong>This is the most important step in training as it gives the gradients of the parameters and bias which are used in gradient descent step to update weights. Again this step contains two substeps:</p><p>5.a.Linear backward: Suppose you have already calculated the derivative dZˡ, we want to get dWˡ, dbˡ, dA[l-1].</p><p>We use these three formulae:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*rKIgrYBZwxuddDB6_OGvEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*LD6kFqpcicdCRgQ9mvOLFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/1*Z1bIDemI8BorgSd1dqhxww.png" /></figure><p>5.b. Linear-Activation backward: Now to compute dZˡ, we use the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*l8nemkzx0-FdoQHDc10mDA.png" /></figure><p>where g’ (Zˡ) is the derivative of g(Z) and it can be relu or sigmoid based on the layer for which we are computing. i.e if layer for which we are computing dZˡ is final output layer, then g(Z) is sigmoid and for other layers, g(Z) is relu.</p><p><strong>6. Update Weights:</strong></p><p>The final step of the training process is to update weights which we get during backward propagation. We use gradient descent to update weights. Update equation is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*t4p8aD4sEzbXS7le_myLjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*L6w6Z34LiQDYG7MZzSmjyA.png" /></figure><p>where α is the learning rate which is a hyperparameter and needs to be tuned properly to achieve better performance.</p><p><strong>Implementation from Scratch:</strong></p><p><strong><em>Data preprocessing Steps:</em></strong></p><ol><li>Importing libraries</li></ol><pre><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong></pre><p>2. Load the digits data sets from sklearn.datasets</p><pre><strong>from</strong> <strong>sklearn.datasets</strong> <strong>import</strong> load_digits<br>digits=load_digits()</pre><p>3. Let’s see the first image in the dataset:</p><pre><strong>import</strong> <strong>pylab</strong> <strong>as</strong> <strong>pl</strong><br>pl.gray()<br>pl.matshow(digits.images[0])<br>pl.show()</pre><p>The result will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*SwnSYnSH9jQSsQ4RYPFD_g.png" /></figure><p>4. Let’s see how our computer looks at this image:</p><pre>digits.images[0]</pre><pre>&quot;&quot;&quot;</pre><pre>array([[ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.],<br>       [ 0.,  0., 13., 15., 10., 15.,  5.,  0.],<br>       [ 0.,  3., 15.,  2.,  0., 11.,  8.,  0.],<br>       [ 0.,  4., 12.,  0.,  0.,  8.,  8.,  0.],<br>       [ 0.,  5.,  8.,  0.,  0.,  9.,  8.,  0.],<br>       [ 0.,  4., 11.,  0.,  1., 12.,  7.,  0.],<br>       [ 0.,  2., 14.,  5., 10., 12.,  0.,  0.],<br>       [ 0.,  0.,  6., 13., 10.,  0.,  0.,  0.]]) &quot;&quot;&quot;</pre><p>So each image is an 8x8 pixel image.</p><p>5. The target digit of this image is:</p><pre>digits.target[0]<br>&quot;&quot;&quot;<br>  <br>0</pre><pre>&quot;&quot;&quot;</pre><p>6. Let’s look at first 15 digits images and their corresponding targets:</p><pre>images_and_labels=list(zip(digits.images,digits.target))<br>plt.figure(figsize=(5,5))<br><strong>for</strong> index,(image,label) <strong>in</strong> enumerate(images_and_labels[:15]):<br>    plt.subplot(3,5,index+1)<br>    plt.axis(&#39;off&#39;)<br>    plt.imshow(image,cmap=plt.cm.gray_r,interpolation=&#39;nearest&#39;)<br>    plt.title(&#39;<strong>%i</strong>&#39; % label)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*aHlv_t4rNJbHRv7QYhH_vg.png" /></figure><p>7. Let’s define some variables:</p><pre><em>#Define variables</em><br>n_samples=len(digits.images)<br>print(&quot;Number of samples in the data set is :&quot;+ str(n_samples))<br><br>x=digits.images.reshape((n_samples,-1))<br>print(&quot;Shape of input matrix x is : &quot;+str(x.shape))</pre><pre>y=digits.target<br>print(&quot;Shape of target vector y is :&quot;+str(y.shape))</pre><pre>&quot;&quot;&quot;<br>Number of samples in the data set is :1797<br>Shape of input matrix x is : (1797, 64)<br>Shape of target vector y is : (1797,)<br>&quot;&quot;&quot;</pre><p>8. Split the data into training and testing sets:</p><pre><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 0)</pre><p>9. Feature Scaling:</p><pre><em># Feature Scaling</em><br><strong>from</strong> <strong>sklearn.preprocessing</strong> <strong>import</strong> StandardScaler<br>sc = StandardScaler()<br>X_train = sc.fit_transform(X_train)<br>X_test = sc.transform(X_test)</pre><p>10. Convert each training example to a column vector.</p><pre>X_train=X_train.T<br>X_test=X_test.T<br>y_train=y_train.reshape(y_train.shape[0],1)<br>y_test=y_test.reshape(y_test.shape[0],1)<br>y_train=y_train.T<br>y_test=y_test.T</pre><p>11. The output layer of our model will have 10 units. Among those, the predicted digit is the index of the unit with the highest value. That is for example if the 7th unit in the output layer has the highest value, then the predicted digit will be 6 (as the 7th unit will be at index 6). So we need to convert each training and testing example to have the shape (10,1).</p><pre>Y_train_=np.zeros((10,y_train.shape[1]))<br><strong>for</strong> i <strong>in</strong> range(y_train.shape[1]):<br>    Y_train_[y_train[0,i],i]=1</pre><pre>Y_test_=np.zeros((10,y_test.shape[1]))<br><strong>for</strong> i <strong>in</strong> range(y_test.shape[1]):<br>    Y_test_[y_test[0,i],i]=1</pre><p>Now with this step, we are done with data preprocessing, let’s implement the Artificial neural network from scratch:</p><p><strong>Implementation:</strong></p><ol><li>Initialize the parameters for Neural network: It includes all W’s and b’s for every hidden layer:</li></ol><pre><em># initialize parameters for deep neural networks</em><br><strong>def</strong> initialize_parameters_deep(layer_dims):<br>    np.random.seed(3)<br>    parameters = {}<br>    L = len(layer_dims) <br>    <strong>for</strong> l <strong>in</strong> range(1, L):<br>        parameters[&#39;W&#39; + str(l)] = np.random.randn(layer_dims[l],layer_dims[l-1])*0.01<br>        parameters[&#39;b&#39; + str(l)] = np.zeros((layer_dims[l],1))<br>        <br>        <strong>assert</strong>(parameters[&#39;W&#39; + str(l)].shape == (layer_dims[l], layer_dims[l-1]))<br>        <strong>assert</strong>(parameters[&#39;b&#39; + str(l)].shape == (layer_dims[l], 1))<br>    <strong>return</strong> parameters</pre><p>Here layer_dims is a list. For example, if we need a neural network with one layer and 6 units in the hidden layer, we need to pass [10,6,10] as layer_dims. Here first 10 corresponds to the number of input features, and last 10 corresponds to the number of units in the output layer.</p><p>2. Forward propagation without activation for a single layer (linear_forward):</p><pre><strong>def</strong> linear_forward(A, W, b):<br>    Z = np.dot(W,A)+b<br>    <strong>assert</strong>(Z.shape == (W.shape[0], A.shape[1]))<br>    cache = (A, W, b)<br>    <br>    <strong>return</strong> Z, cache</pre><p>3. Useful activation functions and their derivatives</p><pre><em># use ful activation functions and their derivatives</em><br><strong>def</strong> sigmoid_(Z):<br>    <strong>return</strong> 1/(1+np.exp(-Z))<br><br><strong>def</strong> relu_(Z):<br>    <strong>return</strong> Z*(Z&gt;0)<br><br><strong>def</strong> drelu_(Z):<br>    <strong>return</strong> 1. *(Z&gt;0)<br><br><strong>def</strong> dsigmoid_(Z):<br>    <strong>return</strong> sigmoid_(Z)*(1-sigmoid_(Z))<br><br><strong>def</strong> sigmoid(Z):<br>    <strong>return</strong> sigmoid_(Z),Z<br><br><strong>def</strong> relu(Z):<br>    <strong>return</strong> relu_(Z),Z</pre><p>4. Forward propagation with activation for a single layer: If the type of activation is sigmoid, it performs sigmoid activation function else performs relu activation function.</p><pre><strong>def</strong> linear_activation_forward(A_prev,W,b,activation):<br>    <strong>if</strong> activation == &quot;sigmoid&quot;:<br>        Z, linear_cache = linear_forward(A_prev,W,b)<br>        A, activation_cache = sigmoid(Z)<br>        <br>    <strong>elif</strong> activation == &quot;relu&quot;:<br>        Z, linear_cache = linear_forward(A_prev,W,b)<br>        A, activation_cache = relu(Z)<br>        <br>    <strong>assert</strong> (A.shape == (W.shape[0], A_prev.shape[1]))<br>    cache = (linear_cache, activation_cache)<br>    <br>    <strong>return</strong> A, cache</pre><p>5. Put all together and implement forward propagation for all L layers in the network:</p><pre><em># implementation of forward propogation for L layer neural network</em><br><strong>def</strong> L_model_forward(X, parameters):<br>    caches = []<br>    A = X<br>    L = len(parameters) // 2   <br>    <strong>for</strong> l <strong>in</strong> range(1, L):<br>        A_prev = A <br>        A, cache = linear_activation_forward(A_prev,parameters[&#39;W&#39;+str(l)],parameters[&#39;b&#39;+str(l)],&quot;relu&quot;)<br>        caches.append(cache)<br>    AL, cache = linear_activation_forward(A,parameters[&#39;W&#39;+str(L)],parameters[&#39;b&#39;+str(L)],&quot;sigmoid&quot;)<br>    caches.append(cache)<br>    <em>#assert(AL.shape == (1,X.shape[1]))</em><br>    <strong>return</strong> AL, caches</pre><p>So L_model_forward takes input X and parameters as arguments and output the final predicted vector of the output layer and some cache information which is used for backpropagation. For the first L-1 layers, we use relu as activation function and for the last layer, we use sigmoid activation function.</p><p>6. Next step is to compute the cost function for the output AL:</p><pre><em># cost function</em><br><strong>def</strong> compute_cost(AL, Y):<br>    m=Y.shape[1]<br>    cost = -(1/m)*np.sum((Y*np.log(AL)+(1-Y)*np.log(1-AL)))<br>    cost=np.squeeze(cost)<br>    <strong>assert</strong>(cost.shape == ())<br>    <strong>return</strong> cost</pre><p>7. Now let’s move to the final part which is backpropagation. Similar to forward propagation, let’s go step by step starting from linear backward for one particular layer:</p><pre><strong>def</strong> linear_backward(dZ, cache):<br>    A_prev, W, b = cache<br>    m = A_prev.shape[1]<br>    dW = (1/m)*np.dot(dZ,A_prev.T)<br>    db = (1/m)*np.sum(dZ,axis=1,keepdims=<strong>True</strong>)<br>    dA_prev = np.dot(W.T,dZ)<br>    <br>    <strong>assert</strong> (dA_prev.shape == A_prev.shape)<br>    <strong>assert</strong> (dW.shape == W.shape)<br>    <strong>assert</strong> (db.shape == b.shape)<br>    <br>    <strong>return</strong> dA_prev, dW, db</pre><p>8. To compute dZ in the above function, we need activation function specific backward propagation method:</p><p>For relu layer:</p><pre><strong>def</strong> relu_backward(dA,activation_cache):<br>    <strong>return</strong> dA* drelu_(activation_cache)</pre><p>Similarly for sigmoid layer:</p><pre><strong>def</strong> sigmoid_backward(dA,activation_cache):<br>    <strong>return</strong> dA* dsigmoid_(activation_cache)</pre><p>9. For one particular layer, overall backward propagation function is implemented as:</p><pre><strong>def</strong> linear_activation_backward(dA, cache, activation):<br>    linear_cache, activation_cache = cache<br>    <strong>if</strong> activation == &quot;relu&quot;:<br>        dZ = relu_backward(dA,activation_cache)<br>        dA_prev, dW, db = linear_backward(dZ,linear_cache)<br>    <br>    <strong>elif</strong> activation == &quot;sigmoid&quot;:<br>        dZ = sigmoid_backward(dA,activation_cache)<br>        dA_prev, dW, db = linear_backward(dZ,linear_cache)<br>    <strong>return</strong> dA_prev,dW,db</pre><p>Here dA is the derivative of one particular layer’s activation. The linear_activaton_backward function takes dA and corresponding cache from linear_activation_forward function and outputs the derivative of the activations of the previous layer along with gradients dW and db. This dW, db are used for adjusting parameters W using gradient descent.</p><p>10. Now for L layers, backward propagation is computed as follows:</p><pre><em># back propogation for L layers</em><br><strong>def</strong> L_model_backward(AL, Y, caches):<br>    grads = {}<br>    L = len(caches)<br>    m = AL.shape[1]<br>    <em>#Y = Y.reshape(AL.shape)</em><br>    <br>    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL))<br>    <br>    current_cache = caches[L-1]<br>    grads[&quot;dA&quot; + str(L-1)], grads[&quot;dW&quot; + str(L)], grads[&quot;db&quot; + str(L)] = linear_activation_backward(dAL,current_cache,&quot;sigmoid&quot;)<br>    <br>    <strong>for</strong> l <strong>in</strong> reversed(range(L-1)):<br>        current_cache = caches[l]<br>        dA_prev_temp, dW_temp, db_temp = linear_activation_backward(grads[&quot;dA&quot;+str(l+1)],current_cache,&quot;relu&quot;)<br>        grads[&quot;dA&quot; + str(l)] = dA_prev_temp<br>        grads[&quot;dW&quot; + str(l + 1)] = dW_temp<br>        grads[&quot;db&quot; + str(l + 1)] = db_temp<br>    <strong>return</strong> grads</pre><p>11. With backpropagation is successfully implemented we need to adjust the weights with the help of computed derivatives (dW and db).</p><pre><em>#update parameters</em><br><strong>def</strong> update_parameters(parameters, grads, learning_rate):<br>    L = len(parameters) // 2 <br>    <strong>for</strong> l <strong>in</strong> range(L):<br>        parameters[&quot;W&quot; + str(l+1)] = parameters[&quot;W&quot; + str(l+1)]-(learning_rate)*grads[&quot;dW&quot;+str(l+1)] <br>        parameters[&quot;b&quot; + str(l+1)] = parameters[&quot;b&quot; + str(l+1)]-(learning_rate)*grads[&quot;db&quot;+str(l+1)]<br>    <strong>return</strong> parameters</pre><p>12. Now putting all the above steps, we shall implement a neural network with 3 hidden layers and train the model.</p><pre><em># N layer neural network</em><br>layers_dims=[n_x,60,10,10]<br><br><strong>def</strong> L_layer_model(X, Y, layers_dims, learning_rate = 0.005, num_iterations = 3000, print_cost=<strong>False</strong>):<br>    np.random.seed(1)<br>    costs = [] <br>    <br>    parameters = initialize_parameters_deep(layers_dims)<br>    <br>    <strong>for</strong> i <strong>in</strong> range(0, num_iterations):<br>        AL, caches = L_model_forward(X, parameters)<br>        cost = compute_cost(AL, Y)<br>        grads = L_model_backward(AL, Y, caches)<br>        parameters = update_parameters(parameters, grads, learning_rate)<br>        <strong>if</strong> print_cost <strong>and</strong> i % 1000 == 0:<br>            print (&quot;Cost after iteration <strong>%i</strong>: <strong>%f</strong>&quot; %(i, cost))<br>        <strong>if</strong> print_cost <strong>and</strong> i % 1000 == 0:<br>            costs.append(cost)<br>    <em># plot the cost</em><br>    plt.plot(np.squeeze(costs))<br>    plt.ylabel(&#39;cost&#39;)<br>    plt.xlabel(&#39;iterations (per tens)&#39;)<br>    plt.title(&quot;Learning rate =&quot; + str(learning_rate))<br>    plt.show()<br>    <br>    <strong>return</strong> parameters</pre><p><strong>Training, Prediction, and Visualization:</strong></p><p>We have successfully implemented all the required functions like forward propagation, computing cost function, backward propagation, and update parameters. Now let’s train the model and check how cost function is decreasing with every iteration.</p><pre>parameters = L_layer_model(X_train, Y_train_, layers_dims, num_iterations = 50000, print_cost = <strong>True</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*6FBwAASiNpltDVM6512EqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*Y174Si67sJCo0Dt6AvZGIw.png" /></figure><p>We can see that after every 1000 iterations, the cost is decreasing. This shows our model is trained well, but let’s see how it will predict the unseen data and analyze some of the implementation tips.</p><p><strong>Prediction</strong>:</p><pre><strong>def</strong> predict_L_layer(X,parameters):<br>    AL,caches=L_model_forward(X,parameters)<br>    prediction=np.argmax(AL,axis=0)<br>    <strong>return</strong> prediction.reshape(1,prediction.shape[0])</pre><p>Let’s test with Training examples to know the percentage of training accuracy.</p><pre>predictions_train_L = predict_L_layer(X_train, parameters)</pre><pre>print(&quot;Training Accuracy : &quot;+ str(np.sum(predictions_train_L==y_train)/y_train.shape[1] * 100)+&quot; %&quot;)</pre><pre>&quot;&quot;&quot; Output<br>Training Accuracy : 100.0 %<br>&quot;&quot;&quot;</pre><p>Wow, so it looks our model has been trained perfectly well to recognize all the training set images. Now let’s try our luck with testing set images to know how accurately our model is predicting the unseen images.</p><pre>predictions_test_L=predict_L_layer(X_test,parameters)<br>print(&quot;Testing Accuracy : &quot;+ str(np.sum(predictions_test_L==y_test)/y_test.shape[1] * 100)+&quot; %&quot;)</pre><pre>&quot;&quot;&quot; Output<br>Testing Accuracy : 97.22222222222221 %<br>&quot;&quot;&quot;</pre><p>Hmm, Not bad, so our testing accuracy is also close to 98% which seems to be perfect. Let’s visualize some of the predictions from the testing set and look at how accurately our model is recognizing the digits from 0–9.</p><pre><strong>import</strong> <strong>random</strong><br><strong>for</strong> j <strong>in</strong> range(15):<br>    i=random.randint(0,n_samples)<br>    pl.gray()<br>    pl.matshow(digits.images[i])<br>    pl.show()<br>    img=digits.images[i].reshape((64,1)).T<br>    img = sc.transform(img)<br>    img=img.T<br>    predicted_digit=predict_L_layer(img,parameters)<br>    print(&#39;Predicted digit is : &#39;+str(predicted_digit))<br>    print(&#39;True digit is: &#39;+ str(y[i]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*r8Ml5Wr-t2WRSqbnGyiz2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*OvUhDIFrHLch_Y9XKOZ3Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*30pc47s25UXct6evneeKtA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*KELbqfEWe_kVrPOY1c6_0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*S1g5X5AQjkW0gUpBa5GJJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*wNp6pLXzJs3M6CBDl9Fg9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*sP0QqTdgtGvDgYPbV9mtZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*8kIaVHP77WB7_8gEvieUZA.png" /></figure><p>Great, we have implemented and tested our machine learning model which recognizes even the low-quality blurry images with great accuracy. This is a difficult task because even humans might not recognize some of the blurry images some times and now our machine learning model is giving promising results.</p><p>Some of the implementations tips while training neural networks are:</p><ol><li>We will not get these results on the very first attempt, in fact when I trained this model for the first time, it’s nowhere near to the current model. So don’t lose hope and try different values of learning rate, number of layers, number of hidden units in each layer etc.</li><li>In this article, we have used 3 hidden layers with 60,10, 10 units in each hidden layer. It’s important to know how to choose this number of hidden layers and the number of units in each hidden layer.</li><li>Always monitor the cost function. If your cost function is not decreasing with iterations, then there is surely a problem with your model.</li></ol><p>In fact, all these issues and implementation tips can be covered under “Hyperparameter Tuning and Optimization” topic which will be discussed in upcoming articles. There are a lot of Hyperparameters that are necessary to tune such as learning rate (α), number of units in each hidden layer, number of hidden layers, regularization parameter (λ) etc.</p><p>This is one of the simplest application of deep-learning and we shall be covering many more in upcoming articles.</p><p>Thank you.</p><p>Full code can be found <a href="https://github.com/pavankalyan1997/Machine-learning-without-any-libraries/blob/master/9.%20Deep%20Neural%20Network/Digit%20Recognition%20from%200-9%20from%20Scratch%20using%20Deep%20Neural%20network.ipynb">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e6bcf1dbd3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/machine-learning-algorithms-from-scratch/digit-recognition-from-0-9-using-deep-neural-network-from-scratch-8e6bcf1dbd3">Digit Recognition from 0–9 using Deep Neural Network from scratch</a> was originally published in <a href="https://medium.com/machine-learning-algorithms-from-scratch">Machine Learning Algorithms from scratch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K-means Clustering from Scratch in Python]]></title>
            <link>https://medium.com/machine-learning-algorithms-from-scratch/k-means-clustering-from-scratch-in-python-1675d38eee42?source=rss----2f01df7a83b0---4</link>
            <guid isPermaLink="false">https://medium.com/p/1675d38eee42</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[scratch-programming]]></category>
            <dc:creator><![CDATA[pavan kalyan urandur]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 21:03:08 GMT</pubDate>
            <atom:updated>2018-12-28T15:12:13.874Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*Dvt1V_YylUPX5_LluzLx9A.png" /></figure><p>In this article, we shall be covering the role of unsupervised learning algorithms, their applications, and K-means clustering approach. On a brief note, Machine learning algorithms can be classified as Supervised and unsupervised learning. In the supervised learning, there will be the data set with input features and the target variable. The aim of the algorithm is to learn the dataset, find the hidden patterns in it and predict the target variable. The target variable can be continuous as in the case of <strong>Regression </strong>or discrete as in the case of <strong>Classification</strong>. Examples of Regression problems include housing price prediction, Stock market prediction, Air humidity, and temperature prediction. Examples of classification problems include Cancer prediction(either benign or malignant), email spam classification etc. Our article demonstrates the next category of machine learning which is unsupervised learning.</p><p>So now one of the other areas of machine learning is unsupervised learning, where we will have the data, but we don’t have any target variable as in the case of supervised learning. So the goal here is to observe the hidden patterns among the data and group them into clusters i.e the data points which have some shared properties will fall into one cluster or one alike group. So where is clustering used? Ever observed the google news where each category like sports, politics, movies, science have 1000’s of news articles? This is clustering. The algorithm groups the news articles which have common features in them into separate groups to form a cluster. Other examples of clustering include social network analysis, market segmentation, recommendation systems etc.</p><p>One of the basic clustering algorithms is K-means clustering algorithm which we are going to discuss and implement from scratch in this article. Let’s look at the final aim of the clustering from the two sample images and a practical example. The first image is the plot of the data set with features x1 and x2. You can see that data is unclustered, so we can’t conclude anything just by looking into the plot. The second image is obtained after performing Clustering, we can observe that data points which are close to each other are grouped as a cluster. In practical situations, this is can be treated as a case where we are dealing with customer data and are asked to segment the market based on customer’s income and the number of previous transactions. Suppose here x1 feature is the annual income and x2 feature is the number of transactions, based on these features we can cluster the data and segment them into three categories like customers with low annual income but a high number of transactions (like orange cluster), customers with medium income and a medium number of transactions (like green cluster), customers with high income but low number of transactions (like blue cluster). Based on these segments, the marketing team of the company can redefine their marketing strategies to get more transactions by recommending the products, which each cluster’s customer might buy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*YsAr1naAwlsZZSr3VD4t9A.png" /><figcaption>Before Clustering</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*PVpkK1-V9AIwtiLHasIppA.png" /><figcaption>After Clustering</figcaption></figure><p><strong>K-Means Clustering Intuition:</strong></p><p>So far we have discussed the goal of clustering and a practical application, now it’s time to dive into K-means clustering implementation and algorithm. As the name itself suggests, this algorithm will regroup <strong>n </strong>data points into <strong>K </strong>number of clusters. So given a large amount of data, we need to cluster this data into K clusters.</p><p>But the problem is how to choose the number of clusters? Most of the times we will know what type of clusters we need to use, for example in the case of Google news, we already knew that say 4 types of clusters (sports, politics, science, movies) exist and news articles should be clustered into any of them. But in the case of market segmentation problem, as the goal is to cluster the type of customers, we don’t know how many types of customers are present in the dataset ( like rich, poor, who loves shopping, who doesn’t love shopping etc) i.e we don’t know exactly how many number of clusters that need to chosen and we can’t randomly assume the number of clusters based on some ground rules (like people with annual income greater than ‘x’ amount should be one cluster and lower than ‘x’ should fall into another).</p><p>We shall discuss the solution to this problem, later in the article but now let’s assume that we are going to segment our data into 3 clusters. Some of the mathematical terms involved in K-means clustering are centroids, euclidian distance. On a quick note centroid of a data is the average or mean of the data and euclidian distance is the distance between two points in the coordinate plane. Given two points A(x1,y1) and B(x2,y2), the euclidian distance between these two points is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*T8YEErGM1R7H__pIW32ixw.png" /></figure><p>While implementing, we won’t consider the square root as we can compare the square of the distances and arrive at conclusions.</p><p><strong>Algorithm:</strong></p><p>Now let’s start talking about the implementation steps. We shall be using either cluster centers or centroids words to describe the cluster centers.</p><p>Step 1:</p><p>Randomly initialize the cluster centers of each cluster from the data points. In fact, random initialization is not an efficient way to start with, as sometimes it leads to increased numbers of required clustering iterations to reach convergence, a greater overall runtime, and a less-efficient algorithm overall. So there are many techniques to solve this problem like K-means++ etc. Let’s also discuss one of the approaches to solve the problem of random initialization later in the article. So let’s assume K=3, so we choose randomly 3 data points and assume them as centroids.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*G-SF8Y8IAvPJEQaTKIQLSA.png" /><figcaption>Random Initialization of Centroids</figcaption></figure><p>Here three cluster centers or centroids with the green, orange, and blue triangle markers are chosen randomly. Again this is not an efficient method to choose the initial cluster centers.</p><p>Step 2:</p><p>2a. For each data point, compute the euclidian distance from all the centroids (3 in this case) and assign the cluster based on the minimal distance to all the centroids. In our example, we need to take each black dot, compute its euclidian distance from all the centroids (green, orange and blue), and finally color the black dot to the color of the closest centroid.</p><p>2b. Adjust the centroid of each cluster by taking the average of all the data points which belong to that cluster on the basis of the computations performed in step 2a. In our example, as we have assigned all the data points to one of the clusters, we need to calculate the mean of all the individual clusters and move the centroid to calculated mean.</p><p>Repeat this process till clusters are well separated or convergence is achieved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*Ke7f0CvFKCFXijCgVNsX0g.png" /><figcaption>2.a Assign the centroids to each data point based on computed euclidian distances</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*3Jtr8-EfKHztRx1V-wjF9A.png" /><figcaption>2b. Adjust the centroids by taking the average of all data points which belong to that cluster</figcaption></figure><p>Repeat these steps until convergence is achieved:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*5ebsmSaH5UfhLLo4QKcxIg.png" /><figcaption>Step 2.a</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*NrpSqxWkrQJLEEs7_ZB-aQ.png" /><figcaption>step 2.b</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*vGhNh_8saG9AqLUbKP5s3w.png" /><figcaption>step 2.a</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*FfKg0FDUtxmNkfKw1SaRXw.png" /><figcaption>step 2.b</figcaption></figure><p><strong>Implementation from scratch:</strong></p><p>Now as we are familiar with intuition, let’s implement the algorithm in python from scratch. We need numpy, pandas and matplotlib libraries to improve the computational complexity and visualization of the results. The dataset which we are going to use is ‘Mall_Customers.csv’ and the link to the dataset can be found in the GitHub page. (Note: The dataset has been downloaded from SuperDataScience website).</p><p>Let’s read the dataset and get the data examples</p><pre>dataset=pd.read_csv(&#39;Mall_Customers.csv&#39;)<br>dataset.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*MdD77mZQpOHVWcdqtHdGDg.png" /></figure><p>For visualization convenience, we are going to take Annual Income and Spending score as our data.</p><pre>X = dataset.iloc[:, [3, 4]].values</pre><p>Now X is two matrix of shape (200,2).</p><p>Next step is to choose the number of iterations which might guarantee convergence. We need to try many possibilities to find optimum number of iterations required for convergence. There is no need to choose a very large number because say at 100th iteration, if the centroids arrived to their true location or best possible location, even after performing 1000 extra iterations, the algorithm will give same results. So for convenience let’s start by choosing number of iterations as 100.</p><pre>m=X.shape[0] #number of training examples<br>n=X.shape[1] #number of features. Here n=2</pre><pre>n_iter=100</pre><p>Next step is to choose number of clusters K. Let’s take 5 as K and as it has been mentioned earlier we are going to see a method later in the article, which will find us the optimum number of clusters K.</p><pre>K=5 # number of clusters</pre><p>We are ready to implement our Kmeans Clustering steps. Let’s proceed:</p><p>Step 1: Initialize the centroids randomly from the data points:</p><pre>Centroids=np.array([]).reshape(n,0) </pre><p>Centroids is a n x K dimentional matrix, where each column will be a centroid for one cluster.</p><pre>for i in range(K):<br>    rand=rd.randint(0,m-1)<br>    Centroids=np.c_[Centroids,X[rand]]</pre><p>Step 2.a: For each training example compute the euclidian distance from the centroid and assign the cluster based on the minimal distance</p><p>The output of our algorithm should be a dictionary with cluster number as Keys and the data points which belong to that cluster as values. So let’s initialize the dictionary.</p><pre>Output={}</pre><p>We find the euclidian distance from each point to all the centroids and store in a m X K matrix. So every row in EuclidianDistance matrix will have distances of that particular data point from all the centroids. Next, we shall find the minimum distance and store the index of the column in a vector C.</p><pre>EuclidianDistance=np.array([]).reshape(m,0)<br>for k in range(K):<br>       tempDist=np.sum((X-Centroids[:,k])**2,axis=1)<br>        EuclidianDistance=np.c_[EuclidianDistance,tempDist]<br>C=np.argmin(EuclidianDistance,axis=1)+1</pre><p>Step 2.b: We need to regroup the data points based on the cluster index C and store in the Output dictionary and also compute the mean of separated clusters and assign it as new centroids. Y is a temporary dictionary which stores the solution for one particular iteration.</p><pre>Y={}<br>for k in range(K):<br>    Y[k+1]=np.array([]).reshape(2,0)<br>for i in range(m):<br>    Y[C[i]]=np.c_[Y[C[i]],X[i]]<br>     <br>for k in range(K):<br>    Y[k+1]=Y[k+1].T<br>    <br>for k in range(K):<br>     Centroids[:,k]=np.mean(Y[k+1],axis=0)</pre><p>Now we need to repeat step 2 till convergence is achieved. In other words, we loop over n_iter and repeat the step 2.a and 2.b as shown:</p><pre>for i in range(n_iter):<br>     #step 2.a<br>      EuclidianDistance=np.array([]).reshape(m,0)<br>      for k in range(K):<br>          tempDist=np.sum((X-Centroids[:,k])**2,axis=1)<br>          EuclidianDistance=np.c_[EuclidianDistance,tempDist]<br>      C=np.argmin(EuclidianDistance,axis=1)+1<br>     #step 2.b<br>      Y={}<br>      for k in range(K):<br>          Y[k+1]=np.array([]).reshape(2,0)<br>      for i in range(m):<br>          Y[C[i]]=np.c_[Y[C[i]],X[i]]<br>     <br>      for k in range(K):<br>          Y[k+1]=Y[k+1].T<br>    <br>      for k in range(K):<br>          Centroids[:,k]=np.mean(Y[k+1],axis=0)<br>      Output=Y</pre><p>Now it’s time to visualize the algorithm and notice how the original data is clustered. To start with, let’s scatter the original unclustered data first.</p><pre>plt.scatter(X[:,0],X[:,1],c=&#39;black&#39;,label=&#39;unclustered data&#39;)<br>plt.xlabel(&#39;Income&#39;)<br>plt.ylabel(&#39;Number of transactions&#39;)<br>plt.legend()<br>plt.title(&#39;Plot of data points&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*7lUdqEorixmpLjcmsC0MXw.png" /></figure><p>Now let’s plot the clustered data:</p><pre>color=[&#39;red&#39;,&#39;blue&#39;,&#39;green&#39;,&#39;cyan&#39;,&#39;magenta&#39;]<br>labels=[&#39;cluster1&#39;,&#39;cluster2&#39;,&#39;cluster3&#39;,&#39;cluster4&#39;,&#39;cluster5&#39;]<br>for k in range(K):<br>    plt.scatter(Output[k+1][:,0],Output[k+1][:,1],c=color[k],label=labels[k])<br>plt.scatter(Centroids[0,:],Centroids[1,:],s=300,c=&#39;yellow&#39;,label=&#39;Centroids&#39;)<br>plt.xlabel(&#39;Income&#39;)<br>plt.ylabel(&#39;Number of transactions&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*sZhlLqgJBnzWH56LiILrDA.png" /></figure><p>Wow, it’s so beautiful and informative too! Our data has become a clustered data from unclustered raw data. We can observe that there are five categories of clusters which are :</p><ol><li>Customers with low income but a High number of transactions (For these type may be the company can recommend products with low price) — Red cluster</li><li>Customers with low income and a low number of transactions (Maybe these type of customers are too busy saving their money) — Cyan Cluster</li><li>Customers with medium income and a medium number of transactions — Green Cluster</li><li>Customers with High income and a low number of transactions — Magenta Cluster</li><li>Customers with High income and a High number of transactions — Blue cluster.</li></ol><p>So the company can divide their customers into 5 classes and design different strategies for a different type of customers to increase their sales. So far so good right, but how do we arrive at the conclusion that there should be 5 number of clusters? To find out how, let’s visualize the clustered data with different clusters starting from 1 to 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*xMYMj80vtG28PBnyvfBXTA.png" /></figure><p>Now we see a lot of plots showing the clustered data with a different number of clusters. So the question is what is the best value for K? Suppose we have <strong>n </strong>data points, and we choose <strong>n</strong> clusters i.e every data point is a cluster, is it a good model? No, obviously not and the converse is also not a good model i.e one cluster for <strong>n </strong>data points. So how do we find the appropriate K? The answer lies in the fact that for every data point, it’s cluster center should be the nearest or in other words, “<strong>Sum of squares of distances of every data point from its corresponding cluster centroid should be as minimum as possible”. </strong>This statement will again contradict with the fact that if all data points are treated as individual clusters, then the sum of squares of distances will be 0. So to counter this fact, we use a method called ELBOW method to find the appropriate number of clusters. The parameter which will be taken into consideration is <strong>Sum of squares of distances of every data point from its corresponding cluster centroid </strong>which is called <strong>WCSS</strong> ( <strong>Within-Cluster Sums of Squares).</strong></p><p>Steps involved in ELBOW method are:</p><ol><li>Perform K means clustering on different values of K ranging from 1 to any upper limit. Here we are taking the upper limit as 10.</li><li>For each K, calculate WCSS</li><li>Plot the value for WCSS with the number of clusters K.</li><li>The location of a bend (knee) in the plot is generally considered as an indicator of the appropriate number of clusters. i.e the point after which WCSS doesn’t decrease more rapidly is the appropriate value of K.</li></ol><p>So let’s implement this and see how K=5 is the best.</p><p>Note: I have converted the algorithm into an object-oriented manner. Kmeans is the name of the class, fit method will perform the Kmeans Clustering, and predict will return the Output dictionary and Centroid matrix. The actual code can be found at Github link.</p><pre>WCSS_array=np.array([])<br>for K in range(1,11):<br>    kmeans=Kmeans(X,K)<br>    kmeans.fit(n_iter)<br>    Output,Centroids=kmeans.predict()<br>    wcss=0<br>    for k in range(K):<br>        wcss+=np.sum((Output[k+1]-Centroids[k,:])**2)<br>    WCSS_array=np.append(WCSS_array,wcss)</pre><p>Now let’s plot the WCSS_array with clusters from 1 to 10.</p><pre>K_array=np.arange(1,11,1)<br>plt.plot(K_array,WCSS_array)<br>plt.xlabel(&#39;Number of Clusters&#39;)<br>plt.ylabel(&#39;within-cluster sums of squares (WCSS)&#39;)<br>plt.title(&#39;Elbow method to determine optimum number of clusters&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*5yCKQYcJLl0CYxziybFD_A.png" /></figure><p>Now if we observe the point after which there isn’t a sudden change in WCSS is K=5. So we choose K=5 as an appropriate number of clusters. So for any given dataset, we need to find the appropriate number of clusters first and start making predictions and alterations to the conclusions.</p><p>So are we done yet? No. we have one more problem left which is random initialization. This is certainly a problem because if two initial centroids are very near, it would take a lot of iterations for the algorithm to converge and so something need to be done in order to make sure that initial centroids are far apart from each other. This is Described as KMeans++ algorithm, where only the initialization of the centroids will change, rest everything is similar to conventional KMeans.</p><p>The objective of the KMeans++ initialization is that chosen centroids should be far from one another. The first cluster center is chosen uniformly at random from the data points that are being clustered, after which each subsequent cluster center is chosen from the remaining data points with probability proportional to its squared distance from the point’s closest existing cluster center. This will be understood well if we represent it as steps.</p><p>Steps involved in Kmeans++ initialization are:</p><ol><li>Randomly select the first cluster center from the data points and append it to the centroid matrix.</li><li>Loop over the number of Centroids that need to be chosen (K):</li><li>For each data point calculate the euclidian distance square from already chosen centroids and append the minimum distance to a Distance array.</li><li>Calculate the probabilities of choosing the particular data point as the next centroid by dividing the Distance array elements with the sum of Distance array. Let’s call this probability distribution as PD.</li><li>Calculate the cumulative probability distribution from this PD distribution. We knew that the cumulative probability distribution ranges from 0 to 1.</li><li>Select a random number between 0 to 1, get the index (i) of the cumulative probability distribution which is just greater than the chosen random number and assign the data point corresponding to the selected index (i).</li><li>Repeat the process until we have K number of cluster centers.</li></ol><p>Here is a one-dimensional example. Our observations are [0, 1, 2, 3, 4]. Let the first center, c1 , be 0. The probability that the next cluster center, c2, is x is proportional to ||c1-x||^2. So, P(c2 = 1) = 1a, P(c2 = 2) = 4a, P(c2 = 3) = 9a, P(c2 = 4) = 16a, where a = 1/(1+4+9+16).</p><p>Suppose c2=4. Then, P(c3 = 1) = 1a, P(c3 = 2) = 4a, P(c3 = 3) = 1a, where a = 1/(1+4+1).</p><p>Example probs=[0.1,0.2,0.3,0.4], cumprobs=[0.1,0.3,0.6,1.0]. Let’s choose a random number r between 0 and 1. if chosen r&lt;cumprobs[0], then this event has probability 0.1, else if chosen r&lt;cumprobs[1], then this event has probability 0.3 and so the data point corresponding to the index 1 is chosen as next cluster center.</p><p>More about Kmeans++ can be found at <a href="https://en.wikipedia.org/wiki/K-means%2B%2B">Wikipedia-link</a> and the above example has been referenced from <a href="https://stackoverflow.com/questions/5466323/how-exactly-does-k-means-work">StackOverFlow-question</a>.</p><p>Now let’s implement this initialization and compare the results with random initialization.</p><ol><li>Randomly select the first cluster center from the data points and append it to the centroid matrix.</li></ol><pre>i=rd.randint(0,X.shape[0])<br>Centroid=np.array([X[i]])</pre><p>2. Let’s put everything into for loop later.</p><p>3. For each data point calculate the euclidian distance square from already chosen centroids and append the minimum distance to a Distance array.</p><pre>D=np.array([]) <br>for x in X:<br>    D=np.append(D,np.min(np.sum((x-Centroid)**2)))</pre><p>4. Calculate the probabilities of choosing the particular data point as the next centroid by dividing the Distance array elements with the sum of Distance array. Let’s call this probability distribution as P.</p><pre>prob=D/np.sum(D)</pre><p>We can’t just select the data point with the highest probability as the cluster center because there might be a chance that the selected cluster will coincide with the previous cluster centers.</p><p>5. Calculate the cumulative probability distribution from this PD distribution. We knew that the cumulative probability distribution ranges from 0 to 1.</p><pre>cummulative_prob=np.cumsum(prob)</pre><p>6. Select a random number between 0 to 1, get the index (i) of the cumulative probability distribution which is just greater than the chosen random number and assign the data point corresponding to the selected index (i).</p><pre>r=rd.random()<br>i=0<br>for j,p in enumerate(cummulative_prob):<br>    if r&lt;p:<br>       i=j<br>       break<br>Centroid=np.append(Centroid,[X[i]],axis=0)</pre><p>Putting together in a for loop for K clusters:</p><pre>i=rd.randint(0,X.shape[0])<br>Centroid=np.array([X[i]])<br>K=5<br>for k in range(1,K):<br>    D=np.array([]) <br>    for x in X:<br>        D=np.append(D,np.min(np.sum((x-Centroid)**2)))<br>    prob=D/np.sum(D)<br>    cummulative_prob=np.cumsum(prob)<br>    r=rd.random()<br>    i=0<br>    for j,p in enumerate(cummulative_prob):<br>        if r&lt;p:<br>            i=j<br>            break<br>    Centroid=np.append(Centroid,[X[i]],axis=0)</pre><p>The final part of the article is to visualize the results and see how Kmeans++ solves the problem of random initialization.</p><p>Centroid_rand is the randomly chosen cluster centers and Centroid is the ones obtained from Kmeans++.</p><pre>for i in range(K):<br>    rand=rd.randint(0,m-1)<br>    Centroids_rand=np.c_[Centroids_rand,X[rand]]<br>plt.scatter(X[:,0],X[:,1])<br>plt.scatter(Centroid_temp[:,0],Centroid_temp[:,1],s=200,color=&#39;yellow&#39;)<br>plt.scatter(Centroids_rand[0,:],Centroids_rand[1,:],s=300,color=&#39;black&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*H45uHWgr3uucgyh-SmYrJg.png" /></figure><p>Yellow dots are the cluster centers chosen using Kmeans++ algorithm and Black dots are cluster centers chosen using Random initialization.</p><p>It is clear that yellow dots are spread wide over the plot and black dots are much closer. This will definitely show an impact in improving the computational complexity of the Kmeans clustering algorithm.</p><p>Finally, let’s implement all these steps using the sklearn library so that we can compare the results:</p><pre><em>#lets implement the same algorithm using sklearn libraries</em><br><em># Using the elbow method to find the optimal number of clusters</em><br><strong>from</strong> <strong>sklearn.cluster</strong> <strong>import</strong> KMeans<br>wcss = []<br><strong>for</strong> i <strong>in</strong> range(1, 11):<br>    kmeans = KMeans(n_clusters = i, init = &#39;k-means++&#39;, random_state = 42)<br>    kmeans.fit(X)<br>    wcss.append(kmeans.inertia_)<br>plt.plot(range(1, 11), wcss)<br>plt.title(&#39;The Elbow Method&#39;)<br>plt.xlabel(&#39;Number of clusters&#39;)<br>plt.ylabel(&#39;WCSS&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*mvhLyWh5ruzqZCFWC1Ir3g.png" /></figure><p>We can observe that the curve is much smoother than our implemented curve.</p><p>Centroid Initialization in sklearn happens using k-means++ parameter as shown.</p><pre><em># Fitting K-Means to the dataset</em><br>kmeans = KMeans(n_clusters = 5, init = &#39;k-means++&#39;, random_state = 42)<br>y_kmeans = kmeans.fit_predict(X)<br></pre><pre><em># Visualising the clusters</em><br>plt.scatter(X[y_kmeans == 0, 0], X[y_kmeans == 0, 1], s = 100, c = &#39;red&#39;, label = &#39;Cluster 1&#39;)<br>plt.scatter(X[y_kmeans == 1, 0], X[y_kmeans == 1, 1], s = 100, c = &#39;blue&#39;, label = &#39;Cluster 2&#39;)<br>plt.scatter(X[y_kmeans == 2, 0], X[y_kmeans == 2, 1], s = 100, c = &#39;green&#39;, label = &#39;Cluster 3&#39;)<br>plt.scatter(X[y_kmeans == 3, 0], X[y_kmeans == 3, 1], s = 100, c = &#39;cyan&#39;, label = &#39;Cluster 4&#39;)<br>plt.scatter(X[y_kmeans == 4, 0], X[y_kmeans == 4, 1], s = 100, c = &#39;magenta&#39;, label = &#39;Cluster 5&#39;)<br>plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s = 300, c = &#39;yellow&#39;, label = &#39;Centroids&#39;)<br>plt.title(&#39;Clusters of customers&#39;)<br>plt.xlabel(&#39;Annual Income (k$)&#39;)<br>plt.ylabel(&#39;Spending Score (1-100)&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*ITlC4T3LSJeMrP0iU60kGg.png" /></figure><p>As a conclusion, in this article,</p><ol><li>We have learned K-means Clustering from scratch and implemented the algorithm in python.</li><li>Solved the problem of choosing the number of clusters based on the Elbow method.</li><li>Solved the problem of random initialization using KMeans++ algorithm.</li></ol><p><strong>So what’s next:</strong></p><ol><li>You can try with the different number of iterations and see how convergence is affected.</li><li>There can be many other ways to find the appropriate number of clusters not just by Elbow method.</li><li>There can be many other ways to solve the random initialization method not just by Kmeans++.</li></ol><p>We shall discuss Hierarchical clustering in the upcoming article.</p><p>All the code samples are present in this <a href="https://github.com/pavankalyan1997/Machine-learning-without-any-libraries/tree/master/2.Clustering/1.K_Means_Clustering">GitHub</a> link.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1675d38eee42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/machine-learning-algorithms-from-scratch/k-means-clustering-from-scratch-in-python-1675d38eee42">K-means Clustering from Scratch in Python</a> was originally published in <a href="https://medium.com/machine-learning-algorithms-from-scratch">Machine Learning Algorithms from scratch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Naive Bayes classification from Scratch in Python]]></title>
            <link>https://medium.com/machine-learning-algorithms-from-scratch/naive-bayes-classification-from-scratch-in-python-e3a48bf5f91a?source=rss----2f01df7a83b0---4</link>
            <guid isPermaLink="false">https://medium.com/p/e3a48bf5f91a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[scratch-programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginner-coding]]></category>
            <dc:creator><![CDATA[pavan kalyan urandur]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 16:16:10 GMT</pubDate>
            <atom:updated>2018-12-12T13:40:55.979Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*NSCq0Wfbf683kI8dIgbVfw.jpeg" /></figure><p>In machine learning, Naive Bayes Classifier belongs to the category of <em>Probabilistic Classifiers</em>. A probabilistic classifier can predict given observation by using a probability distribution over a set of classes and based on that distribution it will predict the most likely class that the observation should belong to.</p><p>Naive Bayes classification is a probabilistic approach to classify the data set based on the famous and well known Bayes Theorem of probability. Key terms in Naive Bayes classification are Prior probability, Posterior Probability, Likelihood probability, and Evidence probability.</p><p>While performing Naive Bayes classification, the commonly used Probability distributions are Gaussian distribution, Multinomial Distribution, Bernoulli distribution etc. In this article, we shall be using Gaussian distribution as it can be related to most of the real-time data (as per the central limit theorem if the thing you measure has a lot of (small, independent) causes (or components), the measure will behave like a normal distribution) and it is very easy to implement because we only need mean and standard deviation of the data. We can even call this implementation as Gaussian Naive Bayes Classification.</p><p>Let’s have a look into the mathematical formula behind Naive Bayes theorem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*UDr3_xCLwD1kP6A2mWv8KA.png" /></figure><ol><li><strong>P(c/x)</strong> represents what is the probability of class c given the input features x. This term is called Posterior probability function which we are interested in finding out. Example: What is the probability that the given input vector or feature <strong>x</strong> belong to class <strong>c.</strong></li><li><strong>P(x/c) </strong>represents what is the probability that the same set of features <strong>x </strong>can contribute to the particular class <strong>c </strong>or the number that is the probability of some observed parameters (here <strong>x</strong>) given a set of outcomes (here <strong>c</strong>) is regarded as the likelihood of the set of parameter values (here <strong>c</strong>) given the observed outcomes. This term is called the Likelihood function.</li><li><strong>P(c) </strong>represents what part of the input examples belongs to class <strong>c </strong>to the total number of input examples in the dataset. Example: What is the ratio of the total number of input examples present in the data set which belong to class <strong>c </strong>to the total number of input examples. For the binary classification problem (with classes c1,c2), <strong>P(c1)=1-P(c2). </strong>This term is called Prior probability function.</li></ol><p>In fact, the numerator of the Bayes theorem is enough to predict the classes because the denominator is independent of any class <strong>c.</strong></p><p>Now the given a dataset, it is easy to find the Prior Probability function <strong>P(c)</strong>. Our interest shifts only towards the likelihood function as we need to predict the class (posterior probability) based on the product of likelihood and prior functions.</p><p>Let’s dive deeper into likely hood term <strong>P(x/c)</strong> and draw some important assumptions and conclusions. Let’s assume there are <strong>n </strong>features in the dataset so overall <strong>x </strong>can be written as <strong>x=[X1, X2, X3……Xn] </strong>concatenated by columns where X1, X2, X3…. Xn represents individual features present in the dataset. An important assumption among these columns is that “<strong><em>ALL THESE COLUMNS ARE INDEPENDENT</em></strong>”.</p><p>Now as <strong>x </strong>is a combination of individual features X1, X2, X3 ….. Xn, we can write P(x/c) in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*iuLad-Li3t-QjPy-lao94Q.png" /></figure><p>From the basic probability, we know that if two events A and B are disjoint or independent, then,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*0qnOMsKPR0RLMxaE7BS63Q.png" /></figure><p>So our likelihood function is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*1JqJDZtdVjTskcvf3Nyitg.png" /></figure><p>The numerator of the Bayes theorem is called Joint probability distribution and we aren’t interested in finding the value of P(x) as it is independent of class labels <strong>c</strong>. All together posterior probability in terms of the joint probability distribution (neglecting denominator P(x)) is written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*yeih0GehKub-FXz2o_qsyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*dOZplb3FPFwaZ03sQyBlCw.png" /></figure><p>Now to calculate each feature’s likelihood function, we use a Gaussian distribution which is parameterized by mean and sigma(standard deviation).</p><p>We can also compute the likelihood function of the entire dataset but the parameters will be mean vector and covariance vector.</p><p>So for a particular feature say X1, likelihood density for class <strong>c </strong>is modeled as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*g9f1ravjII3737wELTPqIg.png" /></figure><p>Here X1 is the vector of features with class label <strong>c.</strong></p><p>Finally putting all together, steps involved in Naive Bayes classification for two class problem with class labels as 0 and 1 are :</p><ol><li>Divide the training dataset into subgroups based on class labels. Example: For a two-class problem with labels 0 and 1, all the training examples which belong to class 0 are separated and class 1 are separated.</li><li>Compute the mean and standard deviations of each feature present in the separated dataset individually. Example: For a feature X1 from the original training set, there will be 2 set of mean and standard deviations for class 0 and class 1.</li><li>For each testing example Xtest, we calculate the two joint probabilities i.e P(c=0/Xtest) and P(c=1/Xtest). Finally, Xtest is assigned to the maximum of both joint probabilities.</li></ol><p>Now let’s implement above steps in python. We are going to use numpy, pandas and matplotlib libraries. The dataset we are going to use is “Social_network_ads.csv” .</p><p>Let’s load the data set and see the first five rows:</p><pre><em>#reading dataset</em><br>Data=pd.read_csv(&#39;Social_Network_Ads.csv&#39;)<br>Data.head(10)</pre><pre>&quot;&quot;&quot;output<br>    User ID  Gender  Age  EstimatedSalary  Purchased<br>0  15624510    Male   19            19000          0<br>1  15810944    Male   35            20000          0<br>2  15668575  Female   26            43000          0<br>3  15603246  Female   27            57000          0<br>4  15804002    Male   19            76000          0<br>5  15728773    Male   27            58000          0<br>6  15598044  Female   27            84000          0<br>7  15694829  Female   32           150000          1<br>8  15600575    Male   25            33000          0<br>9  15727311  Female   35            65000          0</pre><pre>&quot;&quot;&quot;</pre><p>We take Age and EstimatedSalary as independent features and Purchased as dependent features (class labels are 0 and 1). Now let’s divide the training and testing set sizes. We shall take 75% of the original data as training data and 25% as testing data.</p><pre><em>#training and testing set size</em><br>train_size=int(0.75*Data.shape[0])<br>test_size=int(0.25*Data.shape[0])<br>print(&quot;Training set size : &quot;+ str(train_size))<br>print(&quot;Testing set size : &quot;+str(test_size))</pre><pre>&quot;&quot;&quot;output<br>Training set size : 300<br>Testing set size : 100 &quot;&quot;&quot;</pre><p>Shuffle the dataset and extract the required features.</p><pre><em>#Getting features from dataset</em><br>Data=Data.sample(frac=1)<br>X=Data.iloc[:,[2, 3]].values<br>y=Data.iloc[:,4].values<br>X=X.astype(float)</pre><p>It’s useful to perform feature scaling as well because the scales of two features age and salary are entirely different. Feature scaling class is already implemented and you can get the code from Github link.</p><pre><em>#feature scaling</em><br><strong>from</strong> <strong>FeatureScaling</strong> <strong>import</strong> FeatureScaling<br>fs=FeatureScaling(X,y)<br>X=fs.fit_transform_X()</pre><p>Now split the data into training and testing set.</p><pre><em>#training set split</em><br>X_train=X[0:train_size,:]<br>y_train=y[0:train_size]</pre><pre><em>#testing set split</em> <br>X_test=X[train_size:,:] <br>y_test=y[train_size:]</pre><p>We are ready with our data and let’s proceed to the task of classifying the testing data using Naive Bayes algorithm. Before that let’s visualize the training set.</p><pre><em>#visualize the training set </em><br><strong>from</strong> <strong>matplotlib.colors</strong> <strong>import</strong> ListedColormap<br>X_set, y_set = X_train, y_train<br><strong>for</strong> i, j <strong>in</strong> enumerate(np.unique(y_set)):<br>    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],<br>                c = ListedColormap((&#39;red&#39;, &#39;green&#39;))(i), label = j,marker=&#39;.&#39;)<br>plt.title(&#39;Training set&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;Estimated Salary&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*X83liCL_KLS2QvvlFRiQ6g.png" /></figure><p>The objective is to find a decision boundary which separates red and green dots. So let’s implement each step:</p><ol><li>generate dataset function will take X_train and y_train as parameters and split the data set based on class labels 0 and 1. Split data is stored in a dictionary named class_data_dic which is indexed by 0 and 1, i.e class_data_dic[0] fetches all the input training examples which have labels 0 and class_data_dic[1] fetches all the input training examples which have labels 1.</li></ol><pre><strong>def</strong> generate_data(class_data_dic,X_train,y_train):<br>            first_one=<strong>True</strong><br>            first_zero=<strong>True</strong><br>            <strong>for</strong> i <strong>in</strong> range(y_train.shape[0]):<br>                X_temp=X_train[i,:].reshape(X_train[i,:].shape[0],1)<br>                <strong>if</strong> y_train[i]==1:<br>                    <strong>if</strong> first_one==<strong>True</strong>:<br>                        class_data_dic[1]=X_temp<br>                        first_one=<strong>False</strong><br>                    <strong>else</strong>: <br>                        class_data_dic[1]=<br>                        np.append(class_data_dic[1],X_temp,axis=1)                                      <br>                <strong>elif</strong> y_train[i]==0:<br>                    <strong>if</strong> first_zero==<strong>True</strong>:<br>                        class_data_dic[0]=X_temp<br>                        first_zero=<strong>False</strong><br>                    <strong>else</strong>:<br>                        class_data_dic[0]=<br>                        np.append(class_data_dic[0],X_temp,axis=1)                     <br>            <strong>return</strong> class_data_dic</pre><p>2. Let’s compute the mean and standard deviation of each individual split vectors.</p><pre>mean_0=np.mean(class_data_dic[0],axis=0)<br>mean_1=np.mean(class_data_dic[1],axis=0)        std_0=np.std(class_data_dic[0],axis=0)       std_1=np.std(class_data_dic[1],axis=0)</pre><p>let’s also implement some helper functions</p><p>a. likelihood function returns the Gaussian distribution vector for particular X</p><pre><strong>def</strong> likelyhood(x,mean,sigma):<br>                <strong>return</strong> np.exp(-(x-mean)**2/(2*sigma**2))*(1/(np.sqrt(2*np.pi)*sigma))</pre><p>b. posterior function calculates the joint probability and returns the product.</p><pre><strong>def</strong> posterior(X,X_train_class,mean_,std_):<br>            product=np.prod(likelyhood(X,mean_,std_),axis=1)<br>            product=product*  (X_train_class.shape[0]/X_train.shape[0])<br>            <strong>return</strong> product</pre><p>3. Now the only step left is to take each testing example and predict the class based on all the above-implemented functions. This is vectorized as shown. p_1 represents the joint probability that X_test belongs to class 1 and p_0 represents the joint probability that X_test belongs to class 0.</p><pre>p_1=posterior(X_test,class_data_dic[1],mean_1,std_1)   p_0=posterior(X_test,class_data_dic[0],mean_0,std_0)<br>y_pred=1*(p_1&gt;p_0)</pre><p>Finally, y_pred contains the predicted class labels for the test set. Now let’s evaluate the y_pred and y_test and compare the predicted results with actual results. For this, we seek the help of Confusion matrix where each element in the Confusion matrix is True negative, False negative, False positive and True positive. You can read more about confusion matrix from <a href="https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/">here</a>.</p><pre><em>#getting the confusion matrix</em><br>tp=len([i <strong>for</strong> i <strong>in</strong> range(0,y_test.shape[0]) <strong>if</strong> y_test[i]==0 <strong>and</strong> y_pred[i]==0])<br>tn=len([i <strong>for</strong> i <strong>in</strong> range(0,y_test.shape[0]) <strong>if</strong> y_test[i]==0 <strong>and</strong> y_pred[i]==1])<br>fp=len([i <strong>for</strong> i <strong>in</strong> range(0,y_test.shape[0]) <strong>if</strong> y_test[i]==1 <strong>and</strong> y_pred[i]==0])<br>fn=len([i <strong>for</strong> i <strong>in</strong> range(0,y_test.shape[0]) <strong>if</strong> y_test[i]==1 <strong>and</strong> y_pred[i]==1])<br>confusion_matrix=np.array([[tp,tn],[fp,fn]])<br>print(confusion_matrix)</pre><pre>&quot;&quot;&quot;output<br>[[62  6]<br> [ 8 24]]<br>&quot;&quot;&quot;</pre><p>So what does that indicate? The confusion matrix is a metric to evaluate the performance of the classification model. Accuracy can’t be only metric to evaluate because sometimes the data set might be skewed, in that case even if all the elements of y_pred are 0, there is a chance that accuracy of that model will be 99%. So confusion matrix provides a clear-cut understanding of how our model is behaving. Here out of 100 testing examples, 62 examples have the class label 0 in both y_pred and y_test, 24 examples have the class label 1 in both y_pred and y_test. So overall accuracy of our model will be (62+24)/100= 86/100=0.86. On the other hand, there are 6 examples where y_pred=0 and y_test=1 i.e False Negative which is dangerous in some situations and 8 examples where y_pred=1 and y_test=0 i.e False Positive.</p><p>So our model seems to be working nicely for this data set. Let’s visualize the actual decision boundary and understand that Naive Bayes is an excellent non-linear classifier.</p><p>Decision Boundary for the Training and Testset.</p><pre><em># Visualising the Training set results</em><br><strong>from</strong> <strong>matplotlib.colors</strong> <strong>import</strong> ListedColormap<br>X_set, y_set = X_train, y_train<br>X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.1),<br>                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.1))<br>plt.contourf(X1, X2, nb.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),<br>             alpha = 0.75, cmap = ListedColormap((&#39;orange&#39;, &#39;green&#39;)))<br>plt.xlim(X1.min(), X1.max())<br>plt.ylim(X2.min(), X2.max())<br><strong>for</strong> i, j <strong>in</strong> enumerate(np.unique(y_set)):<br>    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],<br>                c = ListedColormap((&#39;red&#39;, &#39;green&#39;))(i), label = j,marker=&#39;.&#39;)<br>plt.title(&#39;Naive Bayes Classification our implementation(Training set)&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;Estimated Salary&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*s2DT1X15-NgKzl0rGWYjrw.png" /></figure><pre><em># Visualising the Test set results</em><br><strong>from</strong> <strong>matplotlib.colors</strong> <strong>import</strong> ListedColormap<br>X_set, y_set = X_test, y_test<br>X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.1),<br>                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.1))<br>plt.contourf(X1, X2, nb.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),<br>             alpha = 0.75, cmap = ListedColormap((&#39;orange&#39;, &#39;green&#39;)))<br>plt.xlim(X1.min(), X1.max())<br>plt.ylim(X2.min(), X2.max())<br><strong>for</strong> i, j <strong>in</strong> enumerate(np.unique(y_set)):<br>    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],<br>                c = ListedColormap((&#39;red&#39;, &#39;green&#39;))(i), label = j,marker=&#39;.&#39;)<br>plt.title(&#39;Naive Bayes Classification our implementation(Test set)&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;Estimated Salary&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*dohgwz_R1BOiayp_ResCQw.png" /></figure><p>Wow, this looks interesting as our model has classified the testing set with a perfect non-linear decision boundary.</p><p>Before wrapping up the article let’s also perform the same using the sklearn library. Implementing the algorithm is perfectly advisable if you are learning the math behind it and for more understanding. However, for practical problems, it is always recommended to use already existing libraries as they provide a more optimized implementation.</p><pre><em>#lets do the same with sk learn and compare</em><br><em># Fitting Naive Bayes to the Training set</em><br><strong>from</strong> <strong>sklearn.naive_bayes</strong> <strong>import</strong> GaussianNB<br>classifier = GaussianNB()<br>classifier.fit(X_train, y_train)</pre><pre><em># Predicting the Test set results</em><br>y_pred = classifier.predict(X_test)</pre><pre><strong>from</strong> <strong>sklearn.metrics</strong> <strong>import</strong> confusion_matrix<br>cm=confusion_matrix(y_test,y_pred)</pre><pre>&quot;&quot;&quot;output<br>[[63 6<br>6 25]]&quot;&quot;&quot;</pre><pre><em># Visualising the Training set results</em><br><strong>from</strong> <strong>matplotlib.colors</strong> <strong>import</strong> ListedColormap<br>X_set, y_set = X_train, y_train<br>X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.1),<br>                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.1))<br>plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),<br>             alpha = 0.75, cmap = ListedColormap((&#39;orange&#39;, &#39;green&#39;)))<br>plt.xlim(X1.min(), X1.max())<br>plt.ylim(X2.min(), X2.max())<br><strong>for</strong> i, j <strong>in</strong> enumerate(np.unique(y_set)):<br>    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],<br>                c = ListedColormap((&#39;red&#39;, &#39;green&#39;))(i), label = j,marker=&#39;.&#39;)<br>plt.title(&#39;Naive Bayes Classification scikit-learn(Training set)&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;Estimated Salary&#39;)<br>plt.legend()<br>plt.show()</pre><pre><em># Visualising the Test set results</em><br><strong>from</strong> <strong>matplotlib.colors</strong> <strong>import</strong> ListedColormap<br>X_set, y_set = X_test, y_test<br>X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.1),<br>                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.1))<br>plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),<br>             alpha = 0.75, cmap = ListedColormap((&#39;orange&#39;, &#39;green&#39;)))<br>plt.xlim(X1.min(), X1.max())<br>plt.ylim(X2.min(), X2.max())<br><strong>for</strong> i, j <strong>in</strong> enumerate(np.unique(y_set)):<br>    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],<br>                c = ListedColormap((&#39;red&#39;, &#39;green&#39;))(i), label = j,marker=&#39;.&#39;)<br>plt.title(&#39;Naive Bayes Classification scikit-learn (Test set)&#39;)<br>plt.xlabel(&#39;Age&#39;)<br>plt.ylabel(&#39;Estimated Salary&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*s2DT1X15-NgKzl0rGWYjrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*dohgwz_R1BOiayp_ResCQw.png" /></figure><p>Full implementation code can be found <a href="https://github.com/pavankalyan1997/Machine-learning-without-any-libraries/tree/master/5.Naive%20Bayes%20Classification">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3a48bf5f91a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/machine-learning-algorithms-from-scratch/naive-bayes-classification-from-scratch-in-python-e3a48bf5f91a">Naive Bayes classification from Scratch in Python</a> was originally published in <a href="https://medium.com/machine-learning-algorithms-from-scratch">Machine Learning Algorithms from scratch</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>