<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tinniam V Ganesh on Medium]]></title>
        <description><![CDATA[Stories by Tinniam V Ganesh on Medium]]></description>
        <link>https://medium.com/@tvganesh_85?source=rss-291dd97ccb01------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*d3mY85XKhBEq4N1m.jpg</url>
            <title>Stories by Tinniam V Ganesh on Medium</title>
            <link>https://medium.com/@tvganesh_85?source=rss-291dd97ccb01------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:46:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tvganesh_85/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep Learning from first principles in Python, R and Octave — Part 1]]></title>
            <link>https://medium.com/@tvganesh_85/deep-learning-from-first-principles-in-python-r-and-octave-part-1-7aa8eed786ce?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/7aa8eed786ce</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Thu, 27 Dec 2018 04:33:27 GMT</pubDate>
            <atom:updated>2018-12-27T04:37:47.040Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Learning from first principles in Python, R and Octave — Part 1</h3><p><em>“You don’t perceive objects as they are. You perceive them as you are.”</em><br><em>“Your interpretation of physical objects has everything to do with the historical trajectory of your brain — and little to do with the objects themselves.”</em><br><em>“The brain generates its own reality, even before it receives information coming in from the eyes and the other senses. This is known as the internal model”</em></p><pre>David Eagleman - The Brain: The Story of You</pre><p>This is the first in the series of posts, I intend to write on Deep Learning. This post is inspired by the <a href="https://www.coursera.org/specializations/deep-learning">Deep Learning Specialization</a> by Prof Andrew Ng on Coursera and <a href="https://www.coursera.org/learn/neural-networks">Neural Networks for Machine Learning</a> by Prof Geoffrey Hinton also on Coursera. In this post I implement Logistic regression with a 2 layer Neural Network i.e. a Neural Network that just has an input layer and an output layer and with no hidden layer.<em>I am certain that any self-respecting Deep Learning/Neural Network would consider a Neural Network without hidden layers as no Neural Network at all!</em></p><p>This 2 layer network is implemented in Python, R and Octave languages. I have included Octave, into the mix, as Octave is a close cousin of Matlab. These implementations in Python, R and Octave are equivalent vectorized implementations. So, if you are familiar in any one of the languages, you should be able to look at the corresponding code in the other two. You can download this R Markdown file and Octave code from <a href="https://github.com/tvganesh/DeepLearning-Part1">DeepLearning -Part 1</a></p><p>To start with, Logistic Regression is performed using sklearn’s logistic regression package for the cancer data set also from sklearn. This is shown below</p><h3>1. Logistic Regression</h3><pre>import numpy as np<br>import pandas as pd<br>import os<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.datasets import make_classification, make_blobs</pre><pre>from sklearn.metrics import confusion_matrix<br>from matplotlib.colors import ListedColormap<br>from sklearn.datasets import load_breast_cancer<br># Load the cancer data<br>(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)<br>X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,<br>                                                   random_state = 0)<br># Call the Logisitic Regression function<br>clf = LogisticRegression().fit(X_train, y_train)<br>print(&#39;Accuracy of Logistic regression classifier on training set: {:.2f}&#39;<br>     .format(clf.score(X_train, y_train)))<br>print(&#39;Accuracy of Logistic regression classifier on test set: {:.2f}&#39;<br>     .format(clf.score(X_test, y_test)))</pre><pre>## Accuracy of Logistic regression classifier on training set: 0.96<br>## Accuracy of Logistic regression classifier on test set: 0.96</pre><p>To check on other classification algorithms, check my post <a href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/">Practical Machine Learning with R and Python — Part 2</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*PALShflCInTj2nJL.png" /></figure><p>Checkout my book ‘Deep Learning from first principles: Second Edition — In vectorized Python, R and Octave’. My book starts with the implementation of a simple 2-layer Neural Network and works its way to a generic L-Layer Deep Learning Network, with all the bells and whistles. The derivations have been discussed in detail. The code has been extensively commented and included in its entirety in the Appendix sections. My book is available on Amazon as <a href="https://www.amazon.com/dp/1791596177">paperback </a>($14.99) and in <a href="https://www.amazon.com/dp/B07LBG542L">kindle version</a>($9.99/Rs449).</p><p>You may also like my companion book “Practical Machine Learning with R and Python:Second Edition- Machine Learning in stereo” available in Amazon in <a href="https://www.amazon.com/dp/1983035661">paperback</a>($10.99) and <a href="https://www.amazon.com/dp/B07DFKSCWZ">Kindle</a>($7.99/Rs449) versions. This book is ideal for a quick reference of the various ML functions and associated measurements in both R and Python which are essential to delve deep into Deep Learning.</p><h3>2. Logistic Regression as a 2 layer Neural Network</h3><p>In the following section Logistic Regression is implemented as a 2 layer Neural Network in Python, R and Octave. The same cancer data set from sklearn will be used to train and test the Neural Network in Python, R and Octave. This can be represented diagrammatically as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*Am0MEHN-6dFHctPC" /></figure><p>The cancer data set has 30 input features, and the target variable ‘output’ is either 0 or 1. Hence the sigmoid activation function will be used in the output layer for classification.</p><p>This simple 2 layer Neural Network is shown below<br>At the input layer there are 30 features and the corresponding weights of these inputs which are initialized to small random values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/0*8PGRMS0tOkd0EiUU" /></figure><p>where ‘b’ is the bias term</p><p>The Activation function is the sigmoid function which is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/0*sTWRL-dzufCHrkhY" /></figure><p>The Loss, when the sigmoid function is used in the output layer, is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/0*kBmIg76-LxIWp9Ne" /></figure><p>(1)</p><h3>Gradient Descent</h3><h3>Forward propagation</h3><p>In forward propagation cycle of the Neural Network the output Z and the output of activation function, the sigmoid function, is first computed. Then using the output ‘y’ for the given features, the ‘Loss’ is computed using equation (1) above.</p><h3>Backward propagation</h3><p>The backward propagation cycle determines how the ‘Loss’ is impacted for small variations from the previous layers upto the input layer. In other words, backward propagation computes the changes in the weights at the input layer, which will minimize the loss. Several cycles of gradient descent are performed in the path of steepest descent to find the local minima. In other words the set of weights and biases, at the input layer, which will result in the lowest loss is computed by gradient descent. The weights at the input layer are decreased by a parameter known as the ‘learning rate’. Too big a ‘learning rate’ can overshoot the local minima, and too small a ‘learning rate’ can take a long time to reach the local minima. This is done for ‘m’ training examples.</p><p><strong>Chain rule of differentiation</strong><br>Let y=f(u)<br>and u=g(x) then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/0*0WZRc80SaA_xTJCy" /></figure><p><strong>Derivative of sigmoid</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/111/0*0zXR3mBUCWBHwZKK" /></figure><p>Let</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/82/0*ZwnLpAnA_a_ChnLb" /></figure><p>then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/57/0*ce56Q_LHTE8eQtIc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/0*Tr-V5K23TNvx4KCW" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/0*gUpvRZ24dt9zlmQs" /></figure><p>Using the chain rule of differentiation we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/0*fymBSWE1rDjM3_Hr" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/0*DgkY7aGGlnOmTDNp" /></figure><p>Therefore</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/0*g-YYmx5LxjCw16u8" /></figure><p>-(2)</p><p>The 3 equations for the 2 layer Neural Network representation of Logistic Regression are</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/0*o0BqegzQsG1lYYRP" /></figure><p>-(a)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/0*Yrlox1SHxj_zg4WP" /></figure><p>-(b)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*9Ypykc7mtlPPkbC5" /></figure><p>-©</p><p>The back propagation step requires the computation of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/51/0*7hg1Ge83s67KbTuj" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/46/0*wxJ1Cl0Qye8LZMg8" /></figure><p>. In the case of regression it would be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/53/0*SyoJrPiTjUY00AaS" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/0*yFEYZFb5IDT1OTci" /></figure><p>where dE is the Mean Squared Error function.<br>Computing the derivatives for back propagation we have</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/0*c9fItwGbUfC1bxPY" /></figure><p>-(d)<br>because</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/124/0*-iN4lE3KsL6VaVod" /></figure><p>Also from equation (2) we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/124/0*IUsPShjBZtkJXyB0" /></figure><p>– (e)<br>By chain rule</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/0*hy-D0qwYAsY7e4QS" /></figure><p>therefore substituting the results of (d) &amp; (e) we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*wD8hIXwgh0aOLc_2" /></figure><p>(f)<br>Finally</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/0*__WMIsnpIAB0OUUv" /></figure><p>-(g)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/90/0*5m0H2QLN1__Opw5E" /></figure><p>– (h)<br>and from (f) we have</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/0*uWjTfQQiE-7dkjXG" /></figure><p>Therefore (g) reduces to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/0*sE_vAnMOhnz7ttHD" /></figure><p>-(i)<br>Also</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/0*PYkU-zqJB5MG8Zed" /></figure><p>-(j)<br>Since</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/74/0*4wuWM5yDkVYl2jzL" /></figure><p>and using (f) in (j)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/0*BR3F4HvcizpYX_ek" /></figure><p>The gradient computes the weights at the input layer and the corresponding bias by using the values<br>of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/24/0*MdZLPdpzlrVHwX8i" /></figure><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/15/0*C5AFLIsp1G_mYJ2X" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/0*Yk5E0nM3Fgns5ORw" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/0*OHA3vftLYN6V9lMR" /></figure><p>I found the computation graph representation in the book Deep Learning: Ian Goodfellow, Yoshua Bengio, Aaron Courville, very useful to visualize and also compute the backward propagation. For the 2 layer Neural Network of Logistic Regression the computation graph is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*hGq_KqDu0ZcH4RoR" /></figure><h3>3. Neural Network for Logistic Regression -Python code (vectorized)</h3><pre>import numpy as np<br>import pandas as pd<br>import os<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split</pre><pre># Define the sigmoid function<br>def sigmoid(z):  <br>    a=1/(1+np.exp(-z))    <br>    return a</pre><pre># Initialize<br>def initialize(dim):<br>    w = np.zeros(dim).reshape(dim,1)<br>    b = 0   <br>    return w</pre><pre># Compute the loss<br>def computeLoss(numTraining,Y,A):<br>    loss=-1/numTraining *np.sum(Y*np.log(A) + (1-Y)*(np.log(1-A)))<br>    return(loss)</pre><pre># Execute the forward propagation<br>def forwardPropagation(w,b,X,Y):<br>    # Compute Z<br>    Z=np.dot(w.T,X)+b<br>    # Determine the number of training samples<br>    numTraining=float(len(X))<br>    # Compute the output of the sigmoid activation function <br>    A=sigmoid(Z)<br>    #Compute the loss<br>    loss = computeLoss(numTraining,Y,A)<br>    # Compute the gradients dZ, dw and db<br>    dZ=A-Y<br>    dw=1/numTraining*np.dot(X,dZ.T)<br>    db=1/numTraining*np.sum(dZ)<br>    <br>    # Return the results as a dictionary<br>    gradients = {&quot;dw&quot;: dw,<br>             &quot;db&quot;: db}<br>    loss = np.squeeze(loss)<br>    return gradients,loss</pre><pre># Compute Gradient Descent    <br>def gradientDescent(w, b, X, Y, numIerations, learningRate):<br>    losses=[]<br>    idx =[]<br>    # Iterate <br>    for i in range(numIerations):<br>        gradients,loss=forwardPropagation(w,b,X,Y)<br>        #Get the derivates<br>        dw = gradients[&quot;dw&quot;]<br>        db = gradients[&quot;db&quot;]<br>        w = w-learningRate*dw<br>        b = b-learningRate*db</pre><pre>        # Store the loss<br>        if i % 100 == 0:<br>            idx.append(i)<br>            losses.append(loss)      <br>        # Set params and grads<br>        params = {&quot;w&quot;: w,<br>                  &quot;b&quot;: b}  <br>        grads = {&quot;dw&quot;: dw,<br>                 &quot;db&quot;: db}<br>    <br>    return params, grads, losses,idx</pre><pre># Predict the output for a training set <br>def predict(w,b,X):<br>    size=X.shape[1]<br>    yPredicted=np.zeros((1,size))<br>    Z=np.dot(w.T,X)<br>    # Compute the sigmoid<br>    A=sigmoid(Z)<br>    for i in range(A.shape[1]):<br>        #If the value is &gt; 0.5 then set as 1<br>        if(A[0][i] &gt; 0.5):<br>            yPredicted[0][i]=1<br>        else:<br>        # Else set as 0<br>            yPredicted[0][i]=0</pre><pre>    return yPredicted</pre><pre>#Normalize the data   <br>def normalize(x):<br>    x_norm = None<br>    x_norm = np.linalg.norm(x,axis=1,keepdims=True)<br>    x= x/x_norm<br>    return x</pre><pre>   <br># Run the 2 layer Neural Network on the cancer data set</pre><pre>from sklearn.datasets import load_breast_cancer<br># Load the cancer data<br>(X_cancer, y_cancer) = load_breast_cancer(return_X_y = True)<br># Create train and test sets<br>X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,<br>                                                   random_state = 0)<br># Normalize the data for better performance<br>X_train1=normalize(X_train)<br></pre><pre># Create weight vectors of zeros. The size is the number of features in the data set=30<br>w=np.zeros((X_train.shape[1],1))<br>#w=np.zeros((30,1))<br>b=0</pre><pre>#Normalize the training data so that gradient descent performs better<br>X_train1=normalize(X_train)<br>#Transpose X_train so that we have a matrix as (features, numSamples)<br>X_train2=X_train1.T</pre><pre># Reshape to remove the rank 1 array and then transpose<br>y_train1=y_train.reshape(len(y_train),1)<br>y_train2=y_train1.T</pre><pre># Run gradient descent for 4000 times and compute the weights<br>parameters, grads, costs,idx = gradientDescent(w, b, X_train2, y_train2, numIerations=4000, learningRate=0.75)<br>w = parameters[&quot;w&quot;]<br>b = parameters[&quot;b&quot;]<br>   </pre><pre># Normalize X_test<br>X_test1=normalize(X_test)<br>#Transpose X_train so that we have a matrix as (features, numSamples)<br>X_test2=X_test1.T</pre><pre>#Reshape y_test<br>y_test1=y_test.reshape(len(y_test),1)<br>y_test2=y_test1.T</pre><pre># Predict the values for <br>yPredictionTest = predict(w, b, X_test2)<br>yPredictionTrain = predict(w, b, X_train2)</pre><pre># Print the accuracy<br>print(&quot;train accuracy: {} %&quot;.format(100 - np.mean(np.abs(yPredictionTrain - y_train2)) * 100))<br>print(&quot;test accuracy: {} %&quot;.format(100 - np.mean(np.abs(yPredictionTest - y_test)) * 100))</pre><pre># Plot the Costs vs the number of iterations<br>fig1=plt.plot(idx,costs)<br>fig1=plt.title(&quot;Gradient descent-Cost vs No of iterations&quot;)<br>fig1=plt.xlabel(&quot;No of iterations&quot;)<br>fig1=plt.ylabel(&quot;Cost&quot;)<br>fig1.figure.savefig(&quot;fig1&quot;, bbox_inches=&#39;tight&#39;)</pre><pre>## train accuracy: 90.3755868545 %<br>## test accuracy: 89.5104895105 %</pre><p><strong>Note</strong>: It can be seen that the Accuracy on the training and test set is 90.37% and 89.51%. This is comparatively poorer than the 96% which the logistic regression of sklearn achieves! But this is mainly because of the absence of hidden layers which is the real power of neural networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/0*LZWh87wa7lEQI-QK.png" /></figure><h3>4. Neural Network for Logistic Regression -R code (vectorized)</h3><pre>source(&quot;RFunctions-1.R&quot;)</pre><pre># Define the sigmoid function<br>sigmoid &lt;- function(z){<br>    a &lt;- 1/(1+ exp(-z))<br>    a<br>}</pre><pre># Compute the loss<br>computeLoss &lt;- function(numTraining,Y,A){<br>    loss &lt;- -1/numTraining* sum(Y*log(A) + (1-Y)*log(1-A))<br>    return(loss)<br>}</pre><pre># Compute forward propagation<br>forwardPropagation &lt;- function(w,b,X,Y){<br>    # Compute Z<br>    Z &lt;- t(w) %*% X +b<br>    #Set the number of samples<br>    numTraining &lt;- ncol(X)<br>    # Compute the activation function<br>    A=sigmoid(Z) <br>    <br>    #Compute the loss<br>    loss &lt;- computeLoss(numTraining,Y,A)<br>    <br>    # Compute the gradients dZ, dw and db<br>    dZ&lt;-A-Y<br>    dw&lt;-1/numTraining * X %*% t(dZ)<br>    db&lt;-1/numTraining*sum(dZ)<br>    <br>    fwdProp &lt;- list(&quot;loss&quot; = loss, &quot;dw&quot; = dw, &quot;db&quot; = db)<br>    return(fwdProp)<br>}</pre><pre># Perform one cycle of Gradient descent<br>gradientDescent &lt;- function(w, b, X, Y, numIerations, learningRate){<br>    losses &lt;- NULL<br>    idx &lt;- NULL<br>    # Loop through the number of iterations<br>    for(i in 1:numIerations){<br>        fwdProp &lt;-forwardPropagation(w,b,X,Y)<br>        #Get the derivatives<br>        dw &lt;- fwdProp$dw<br>        db &lt;- fwdProp$db<br>        #Perform gradient descent<br>        w = w-learningRate*dw<br>        b = b-learningRate*db<br>        l &lt;- fwdProp$loss<br>        # Stoe the loss<br>        if(i %% 100 == 0){<br>            idx &lt;- c(idx,i)<br>            losses &lt;- c(losses,l)  <br>        }<br>    }<br>    <br>    # Return the weights and losses<br>    gradDescnt &lt;- list(&quot;w&quot;=w,&quot;b&quot;=b,&quot;dw&quot;=dw,&quot;db&quot;=db,&quot;losses&quot;=losses,&quot;idx&quot;=idx)<br>   <br>    return(gradDescnt)<br>}</pre><pre># Compute the predicted value for input<br>predict &lt;- function(w,b,X){<br>    m=dim(X)[2]<br>    # Create a ector of 0&#39;s<br>    yPredicted=matrix(rep(0,m),nrow=1,ncol=m)<br>    Z &lt;- t(w) %*% X +b<br>    # Compute sigmoid<br>    A=sigmoid(Z)<br>    for(i in 1:dim(A)[2]){<br>        # If A &gt; 0.5 set value as 1<br>        if(A[1,i] &gt; 0.5)<br>        yPredicted[1,i]=1<br>       else<br>        # Else set as 0<br>        yPredicted[1,i]=0<br>    }</pre><pre>    return(yPredicted)<br>}</pre><pre># Normalize the matrix<br>normalize &lt;- function(x){<br>    #Create the norm of the matrix.Perform the Frobenius norm of the matrix <br>    n&lt;-as.matrix(sqrt(rowSums(x^2)))<br>    #Sweep by rows by norm. Note &#39;1&#39; in the function which performing on every row<br>    normalized&lt;-sweep(x, 1, n, FUN=&quot;/&quot;)<br>    return(normalized)<br>}</pre><pre># Run the 2 layer Neural Network on the cancer data set<br># Read the data (from sklearn)<br>cancer &lt;- read.csv(&quot;cancer.csv&quot;)<br># Rename the target variable<br>names(cancer) &lt;- c(seq(1,30),&quot;output&quot;)<br># Split as training and test sets<br>train_idx &lt;- trainTestSplit(cancer,trainPercent=75,seed=5)<br>train &lt;- cancer[train_idx, ]<br>test &lt;- cancer[-train_idx, ]</pre><pre># Set the features<br>X_train &lt;-train[,1:30]<br>y_train &lt;- train[,31]<br>X_test &lt;- test[,1:30]<br>y_test &lt;- test[,31]<br># Create a matrix of 0&#39;s with the number of features<br>w &lt;-matrix(rep(0,dim(X_train)[2]))<br>b &lt;-0<br>X_train1 &lt;- normalize(X_train)<br>X_train2=t(X_train1)</pre><pre># Reshape  then transpose<br>y_train1=as.matrix(y_train)<br>y_train2=t(y_train1)</pre><pre># Perform gradient descent<br>gradDescent= gradientDescent(w, b, X_train2, y_train2, numIerations=3000, learningRate=0.77)<br></pre><pre># Normalize X_test<br>X_test1=normalize(X_test)<br>#Transpose X_train so that we have a matrix as (features, numSamples)<br>X_test2=t(X_test1)</pre><pre>#Reshape y_test and take transpose<br>y_test1=as.matrix(y_test)<br>y_test2=t(y_test1)</pre><pre># Use the values of the weights generated from Gradient Descent<br>yPredictionTest = predict(gradDescent$w, gradDescent$b, X_test2)<br>yPredictionTrain = predict(gradDescent$w, gradDescent$b, X_train2)</pre><pre>sprintf(&quot;Train accuracy: %f&quot;,(100 - mean(abs(yPredictionTrain - y_train2)) * 100))</pre><pre>## [1] &quot;Train accuracy: 90.845070&quot;</pre><pre>sprintf(&quot;test accuracy: %f&quot;,(100 - mean(abs(yPredictionTest - y_test)) * 100))</pre><pre>## [1] &quot;test accuracy: 87.323944&quot;</pre><pre>df &lt;-data.frame(gradDescent$idx, gradDescent$losses)<br>names(df) &lt;- c(&quot;iterations&quot;,&quot;losses&quot;)<br>ggplot(df,aes(x=iterations,y=losses)) + geom_point() + geom_line(col=&quot;blue&quot;) +<br>    ggtitle(&quot;Gradient Descent - Losses vs No of Iterations&quot;) +<br>    xlab(&quot;No of iterations&quot;) + ylab(&quot;Losses&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*zBHKlUtjygfrumUt" /></figure><h3>4. Neural Network for Logistic Regression -Octave code (vectorized)</h3><p>1;<br> # Define sigmoid function<br> function a = sigmoid(z)<br> a = 1 ./ (1+ exp(-z));<br> end<br> # Compute the loss<br> function loss=computeLoss(numtraining,Y,A)<br> loss = -1/numtraining * sum((Y .* log(A)) + (1-Y) .* log(1-A));<br> end</p><p># Perform forward propagation<br> function [loss,dw,db,dZ] = forwardPropagation(w,b,X,Y)<br> % Compute Z<br> Z = w&#39; * X + b;<br> numtraining = size(X)(1,2);<br> # Compute sigmoid<br> A = sigmoid(Z);</p><p>#Compute loss. Note this is element wise product<br> loss =computeLoss(numtraining,Y,A);<br> # Compute the gradients dZ, dw and db<br> dZ = A-Y;<br> dw = 1/numtraining* X * dZ&#39;;<br> db =1/numtraining*sum(dZ);</p><p>end</p><p># Compute Gradient Descent<br> function [w,b,dw,db,losses,index]=gradientDescent(w, b, X, Y, numIerations, learningRate)<br> #Initialize losses and idx<br> losses=[];<br> index=[];<br> # Loop through the number of iterations<br> for i=1:numIerations,<br> [loss,dw,db,dZ] = forwardPropagation(w,b,X,Y);<br> # Perform Gradient descent<br> w = w - learningRate*dw;<br> b = b - learningRate*db;<br> if(mod(i,100) ==0)<br> # Append index and loss<br> index = [index i];<br> losses = [losses loss];<br> endif</p><p>end<br>end</p><p># Determine the predicted value for dataset<br> function yPredicted = predict(w,b,X)<br> m = size(X)(1,2);<br> yPredicted=zeros(1,m);<br> # Compute Z<br> Z = w&#39; * X + b;<br> # Compute sigmoid<br> A = sigmoid(Z);<br> for i=1:size(X)(1,2),<br> # Set predicted as 1 if A &gt; 0,5<br> if(A(1,i) &gt;= 0.5)<br> yPredicted(1,i)=1;<br> else<br> yPredicted(1,i)=0;<br> endif<br> end<br> end</p><p># Normalize by dividing each value by the sum of squares<br> function normalized = normalize(x)<br> # Compute Frobenius norm. Square the elements, sum rows and then find square root<br> a = sqrt(sum(x .^ 2,2));<br> # Perform element wise division<br> normalized = x ./ a;<br> end</p><p># Split into train and test sets<br> function [X_train,y_train,X_test,y_test] = trainTestSplit(dataset,trainPercent)<br> # Create a random index<br> ix = randperm(length(dataset));<br> # Split into training<br> trainSize = floor(trainPercent/100 * length(dataset));<br> train=dataset(ix(1:trainSize),:);<br> # And test<br> test=dataset(ix(trainSize+1:length(dataset)),:);<br> X_train = train(:,1:30);<br> y_train = train(:,31);<br> X_test = test(:,1:30);<br> y_test = test(:,31);<br> end</p><p>cancer=csvread(&quot;cancer.csv&quot;);<br> [X_train,y_train,X_test,y_test] = trainTestSplit(cancer,75);<br> w=zeros(size(X_train)(1,2),1);<br> b=0;<br> X_train1=normalize(X_train);<br> X_train2=X_train1&#39;;<br> y_train1=y_train&#39;;<br> [w1,b1,dw,db,losses,idx]=gradientDescent(w, b, X_train2, y_train1, numIerations=3000, learningRate=0.75);<br> # Normalize X_test<br> X_test1=normalize(X_test);<br> #Transpose X_train so that we have a matrix as (features, numSamples)<br> X_test2=X_test1&#39;;<br> y_test1=y_test&#39;;<br> # Use the values of the weights generated from Gradient Descent<br> yPredictionTest = predict(w1, b1, X_test2);<br> yPredictionTrain = predict(w1, b1, X_train2);</p><p>trainAccuracy=100-mean(abs(yPredictionTrain - y_train1))*100<br> testAccuracy=100- mean(abs(yPredictionTest - y_test1))*100<br> trainAccuracy = 90.845<br> testAccuracy = 89.510<br> graphics_toolkit(&#39;gnuplot&#39;)<br> plot(idx,losses);<br> title (&#39;Gradient descent- Cost vs No of iterations&#39;);<br> xlabel (&quot;No of iterations&quot;);<br> ylabel (&quot;Cost&quot;);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*QRscRNTc0CP_jJFS" /></figure><p><strong>Conclusion</strong><br>This post starts with a simple 2 layer Neural Network implementation of Logistic Regression. Clearly the performance of this simple Neural Network is comparatively poor to the highly optimized sklearn’s Logistic Regression. This is because the above neural network did not have any hidden layers. Deep Learning &amp; Neural Networks achieve extraordinary performance because of the presence of deep hidden layers</p><p>The Deep Learning journey has begun… Don’t miss the bus!<br>Stay tuned for more interesting posts in Deep Learning!!</p><p>References<br>1. <a href="https://www.coursera.org/specializations/deep-learning">Deep Learning Specialization</a><br>2. <a href="https://www.coursera.org/learn/neural-networks">Neural Networks for Machine Learning</a><br>3. <a href="http://www.deeplearningbook.org/">Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville</a><br>4. <a href="https://gigadom.wordpress.com/2017/01/21/neural-networks-the-mechanics-of-backpropagation/">Neural Networks: The mechanics of backpropagation</a><br>5. <a href="https://www.coursera.org/learn/machine-learning">Machine Learning</a></p><p>Also see<br>1. <a href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>2. <a href="https://gigadom.wordpress.com/2014/01/03/simplifying-machine-learning-bias-variance-regularization-and-odd-facts-part-4/">Simplifying Machine Learning: Bias, Variance, regularization and odd facts — Part 4</a><br>3. <a href="https://gigadom.wordpress.com/2017/12/16/the-3rd-paperback-edition-of-my-books-on-cricket-on-amazon/">The 3rd paperback &amp; kindle editions of my books on Cricket, now on Amazon</a><br>4. <a href="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/">Practical Machine Learning with R and Python — Part 4</a><br>5. <a href="https://gigadom.wordpress.com/2016/06/23/introducing-qcsimulator-a-5-qubit-quantum-computing-simulator-in-r/">Introducing QCSimulator: A 5-qubit quantum computing simulator in R</a><br>6. <a href="https://gigadom.wordpress.com/2014/07/27/a-bluemix-recipe-with-mongodb-and-node-js/">A Bluemix recipe with MongoDB and Node.js</a><br>7. <a href="https://gigadom.wordpress.com/2017/09/11/my-travels-through-the-realms-of-data-science-machine-learning-deep-learning-and-ai/">My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)</a></p><p>To see all posts check <a href="https://gigadom.wordpress.com/aa-2/">Index of posts</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7aa8eed786ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My book ‘Practical Machine Learning with R and Python’ on Amazon]]></title>
            <link>https://medium.com/@tvganesh_85/my-book-practical-machine-learning-with-r-and-python-on-amazon-664fa4971576?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/664fa4971576</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Tue, 26 Dec 2017 10:46:25 GMT</pubDate>
            <atom:updated>2017-12-26T10:46:25.641Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*bfVBx2zxD8mPzoUs.png" /></figure><p>My book ‘Practical Machine Learning with R and Python — Machine Learning in stereo’ is now available in both paperback ($9.99) and kindle ($6.97/Rs449) versions. In this book I implement some of the most common, but important Machine Learning algorithms in R and equivalent Python code. This is almost like listening to parallel channels of music in stereo!<br> 1. <a href="https://www.amazon.com/dp/1973443503">Practical machine with R and Python — Machine Learning in Stereo</a> (Paperback)<br> 2. <a href="https://www.amazon.in/dp/B077WFS87Z">Practical machine with R and Python — Machine Learning in Stereo</a> (Kindle)<br> This book is ideal both for beginners and the experts in R and/or Python. Those starting their journey into datascience and ML will find the first 3 chapters useful, as they touch upon the most important programming constructs in R and Python and also deal with equivalent statements in R and Python. Those who are expert in either of the languages, R or Python, will find the equivalent code ideal for brushing up on the other language. And finally,those who are proficient in both languages, can use the R and Python implementations to internalize the ML algorithms better.</p><p>Here is a look at the topics covered</p><p><strong>Table of Contents</strong><br> Essential R …………………………………….. 7<br> Essential Python for Datascience ……………….. 54<br> R vs Python ……………………………………. 77<br> Regression of a continuous variable ………………. 96<br> Classification and Cross Validation ……………….113<br> Regression techniques and regularization …………. 134<br> SVMs, Decision Trees and Validation curves …………175<br> Splines, GAMs, Random Forests and Boosting …………202<br> PCA, K-Means and Hierarchical Clustering …………. 234</p><p>Pick up your copy today!!<br> Hope you have a great time learning as I did while implementing these algorithms!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=664fa4971576" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My book ‘Practical Machine Learning with R and Python’ on Amazon]]></title>
            <link>https://medium.com/@tvganesh_85/my-book-practical-machine-learning-with-r-and-python-on-amazon-497d5f1bd8bb?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/497d5f1bd8bb</guid>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Wed, 06 Dec 2017 04:52:24 GMT</pubDate>
            <atom:updated>2017-12-06T04:52:24.326Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*sFgm5AMmUgCn-LVp." /></figure><p>My book ‘Practical Machine Learning with R and Python — Machine Learning in stereo’ is now available in both paperback ($9.99) and kindle ($6.97/Rs449) versions. In this book I implement some of the most common, but important Machine Learning algorithms in R and equivalent Python code. This is almost like listening to parallel channels of music in stereo!<br> 1. <a href="https://www.amazon.com/dp/1973443503">Practical machine with R and Python — Machine Learning in Stereo</a> (Paperback)<br> 2. <a href="https://www.amazon.in/dp/B077WFS87Z">Practical machine with R and Python — Machine Learning in Stereo</a> (Kindle)<br> This book is ideal both for beginners and the experts in R and/or Python. Those starting their journey into datascience and ML will find the first 3 chapters useful, as they touch upon the most important programming constructs in R and Python and also deal with equivalent statements in R and Python. Those who are expert in either of the languages, R or Python, will find the equivalent code ideal for brushing up on the other language. And finally,those who are proficient in both languages, can use the R and Python implementations to internalize the ML algorithms better.</p><p>Here is a look at the topics covered</p><p><strong>Table of Contents</strong><br> Essential R …………………………………….. 7<br> Essential Python for Datascience ……………….. 54<br> R vs Python ……………………………………. 77<br> Regression of a continuous variable ………………. 96<br> Classification and Cross Validation ……………….113<br> Regression techniques and regularization …………. 134<br> SVMs, Decision Trees and Validation curves …………175<br> Splines, GAMs, Random Forests and Boosting …………202<br> PCA, K-Means and Hierarchical Clustering …………. 234</p><p>Pick up your copy today!!<br> Hope you have a great time learning as I did while implementing these algorithms!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=497d5f1bd8bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks: On Perceptrons and Sigmoid Neurons]]></title>
            <link>https://medium.com/@tvganesh_85/neural-networks-on-perceptrons-and-sigmoid-neurons-5f2bb0e6a464?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f2bb0e6a464</guid>
            <category><![CDATA[perceptron]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[sigmoid]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Mon, 23 Jan 2017 04:01:03 GMT</pubDate>
            <atom:updated>2017-01-23T04:01:03.499Z</atom:updated>
            <content:encoded><![CDATA[<p>Neural Networks had their beginnings in 1943 when Warren McCulloch, a neurophysiologist, and a young mathematician, Walter Pitts, wrote a paper on how neurons might work. Much later in 1958, Frank Rosenblatt, a neuro-biologist proposed the Perceptron. The Perceptron is a computer model or computerized machine which is devised to represent or simulate the ability of the brain to recognize and discriminate. In machine learning, the perceptron is an algorithm for supervised learning of binary classifiers</p><p>Initially it was believed that Perceptrons were capable of many things including “the ability to walk, talk, see, write, reproduce itself and be conscious of its existence.”</p><p>However, a subsequent paper by Marvin Minky and Seymour Papert of MIT, titled “Perceptrons” proved that the Perceptron was truly limited in its functionality. Specifically they showed that the Perceptron was incapable of producing XOR functionality. The Perceptron is only capable of classification where the data points are linearly separable.</p><p>This post implements the simple learning algorithm of the ‘Linear Perceptron’ and the ‘Sigmoid Perceptron’. The implementation has been done in Octave. This implementation is based on “<a href="https://www.coursera.org/learn/neural-networks">Neural networks for Machine Learning</a>” course by Prof Geoffrey Hinton at Coursera</p><p><strong>Perceptron learning procedure</strong><br>z = ∑wixi + b<br>where wi is the ith weight and xi is the ith feature</p><p>For every training case compute the activation output zi</p><ul><li>If the output classifies correctly, leave the weights alone</li><li>If the output classifies a ‘0’ as a ‘1’, then subtract the the feature from the weight</li><li>If the output classifies a ‘0’ as a ‘1’, then add the feature to the weight</li></ul><p>This simple neural network is represented below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*-z65-Ecov3aMak6p." /></figure><p><strong>Sigmoid neuron learning procedure</strong><br>zi = sigmoid(∑wixi + b)<br>where sigmoid is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/0*g5PHs_TP9mGNQXzt." /></figure><p>Hence</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/0*CS5Swp9nj-u3GE0I." /></figure><p>For every training case compute the activation output zi</p><ul><li>If the output classifies correctly, leave the weights alone</li><li>If the output incorrectly classifies a ‘0’ as a ‘1’ i.e.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/111/0*JId6hdWMhK_lLTcJ." /></figure><ul><li>, then subtract the feature from the weight</li><li>If the output incorrectly classifies a ‘1’ as ‘0’ i.e., i.e</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/111/0*jKjTO-EWOF0sVO0z." /></figure><ul><li>, then add the feature to the weight</li><li>Iterate till errors &lt;= 1</li></ul><p>This is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*FklPczLROP6RKqcG." /></figure><p>I have implemented the learning algorithm of the Perceptron and Sigmoid Neuron in Octave. The code is available at Github at <a href="https://github.com/tvganesh/perceptron">Perceptron</a>.</p><ol><li><strong>Perceptron execution</strong></li></ol><p>I performed the tests on 2 different datasets</p><p><strong>Data 1</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/0*vkYlrRQjV7I85kck." /></figure><p><strong>Data 2</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*QQA7QEdFydqsN8y-." /></figure><p><strong>2. Sigmoid Perceptron execution</strong><br><strong>Data 1 &amp; Data 2</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/0*TWoHODWCNS_dgsQY." /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/0*6zUKiBeg1ZgaWvdo." /></figure><p>It can be seen that the Perceptron does work for simple linearly separable data</p><p>For more posts see <a href="https://gigadom.wordpress.com/">https://gigadom.wordpress.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f2bb0e6a464" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing cricketr! : An R package to analyze performances of cricketers]]></title>
            <link>https://medium.com/@tvganesh_85/introducing-cricketr-an-r-package-to-analyze-performances-of-cricketers-c993db6b5aa2?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/c993db6b5aa2</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[cricket]]></category>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Thu, 13 Aug 2015 09:26:40 GMT</pubDate>
            <atom:updated>2015-08-13T09:30:32.167Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Yet all experience is an arch wherethro’<br>Gleams that untravell’d world whose margin fades<br>For ever and forever when I move.<br>How dull it is to pause, to make an end,<br>To rust unburnish’d, not to shine in use!</em><br>Ulysses by Alfred Tennyson</p><p>Introduction</p><p>This is an initial post in which I introduce a cricketing package <strong>‘cricketr’</strong> which I have created. This package was a natural culmination to my earlier posts on cricket and my completing 9 modules of Data Science Specialization, from John Hopkins University at Coursera. The thought of creating this package struck me some time back, and I have finally been able to bring this to fruition.</p><p>So here it is. My R package <strong>‘cricketr!!!’</strong></p><p>This package uses the statistics info available in <a href="http://stats.espncricinfo.com/ci/engine/stats/index.html">ESPN Cricinfo Statsguru</a>. The current version of this package only uses data from test cricket. I plan to develop functionality for One-day and Twenty20 cricket later.</p><p>You should be able to install the package from GitHub and use many of the functions available in the package. Please be mindful of <a href="http://www.espncricinfo.com/ci/content/site/company/terms_use.html">ESPN Cricinfo Terms of Use</a></p><p>(Note: This page is also hosted as a GitHub page at <a href="http://tvganesh.github.io/cricketer/cricketer.html">cricketr</a> and also at RPubs as <a href="http://rpubs.com/tvganesh/91202">cricketr: A R package for analyzing performances of cricketers</a></p><p>You can download this analysis as a PDF file from <a href="https://drive.google.com/file/d/0Bxz66bp5PxPBd0JEUEg3eVlEQTQ/view?usp=sharing">Introducing cricketr</a></p><p>The <strong>cricketr</strong> package</p><p>The cricketr package has several functions that perform several different analyses on both batsman and bowlers. The package has functions that plot percentage frequency runs or wickets, runs likelihood for a batsman, relative run/strike rates of batsman and relative performance/economy rate for bowlers are available.</p><p>Other interesting functions include batting performance moving average, forecast and a function to check whether the batsman/bowler is in in-form or out-of-form.</p><p>The data for a particular player can be obtained with the getPlayerData() function from the package. To do this you will need to go to <a href="http://www.espncricinfo.com/ci/content/player/index.html">ESPN CricInfo Player</a> and type in the name of the player for e.g Ricky Ponting, Sachin Tendulkar etc. This will bring up a page which have the profile number for the player e.g. for Sachin Tendulkar this would be <a href="http://www.espncricinfo.com/india/content/player/35320.html">http://www.espncricinfo.com/india/content/player/35320.html</a>. Hence, Sachin’s profile is 35320. This can be used to get the data for Tendulkar as shown below</p><p>The cricketr package can be installed from GitHub with</p><pre>library(devtools)<br>install_github(&quot;tvganesh/cricketr&quot;)<br>library(cricketr)<br>tendulkar &lt;- getPlayerData(35320,dir=&quot;..&quot;,file=&quot;tendulkar.csv&quot;,type=&quot;batting&quot;,homeOrAway=c(1,2),<br>                           result=c(1,2,4))</pre><p><strong>Important Note</strong> This needs to be done only once for a player. This function stores the player’s data in a CSV file (for e.g. tendulkar.csv as above) which can then be reused for all other functions. Once we have the data for the players many analyses can be done. This post will use the stored CSV file obtained with a prior getPlayerData for all subsequent analyses</p><h2>Sachin Tendulkar’s performance — Basic Analyses</h2><p>The 3 plots below provide the following for Tendulkar</p><ol><li>Frequency percentage of runs in each run range over the whole career</li><li>Mean Strike Rate for runs scored in the given range</li><li>A histogram of runs frequency percentages in runs ranges</li></ol><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>batsmanRunsFreqPerf(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)<br>batsmanMeanStrikeRate(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)<br>batsmanRunsRanges(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*vFQPXtEar-qFSxu-.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>More analyses</h2><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>batsman4s(&quot;./tendulkar.csv&quot;,&quot;Tendulkar&quot;)<br>batsman6s(&quot;./tendulkar.csv&quot;,&quot;Tendulkar&quot;)<br>batsmanDismissals(&quot;./tendulkar.csv&quot;,&quot;Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*1LH0HfVHZyRdyueS.png" /></figure><h2>3D scatter plot and prediction plane</h2><p>The plots below show the 3D scatter plot of Sachin’s Runs versus Balls Faced and Minutes at crease. A linear regression model is then fitted between Runs and Balls Faced + Minutes at crease</p><pre>battingPerf3d(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*Fof7c_LGwFnderfd.png" /></figure><h2>Average runs at different venues</h2><p>The plot below gives the average runs scored by Tendulkar at different grounds. The plot also displays the number of innings at each ground as a label at x-axis. It can be seen Tendulkar did great in Colombo (SSC), Melbourne ifor matches overseas and Mumbai, Mohali and Bangalore at home</p><pre>batsmanAvgRunsGround(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*DLahfx11caPekILR.png" /></figure><h2>Average runs against different opposing teams</h2><p>This plot computes the average runs scored by Tendulkar against different countries. The x-axis also gives the number of innings against each team</p><pre>batsmanAvgRunsOpposition(&quot;./tendulkar.csv&quot;,&quot;Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*eX82lvuzaFmMmwgK.png" /></figure><h2>Highest Runs Likelihood</h2><p>The plot below shows the Runs Likelihood for a batsman. For this the performance of Sachin is plotted as a 3D scatter plot with Runs versus Balls Faced + Minutes at crease using. K-Means. The centroids of 3 clusters are computed and plotted. In this plot. Sachin Tendulkar’s highest tendencies are computed and plotted using K-Means</p><pre>batsmanRunsLikelihood(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*iVttrnUaCxAr_dLr.png" /></figure><pre>## Summary of  Sachin Tendulkar &#39;s runs scoring likelihood<br>## **************************************************<br>## <br>## There is a 16.51 % likelihood that Sachin Tendulkar  will make  139 Runs in  251 balls over 353  Minutes <br>## There is a 58.41 % likelihood that Sachin Tendulkar  will make  16 Runs in  31 balls over  44  Minutes <br>## There is a 25.08 % likelihood that Sachin Tendulkar  will make  66 Runs in  122 balls over 167  Minutes</pre><p>A look at the Top 4 batsman — Tendulkar, Kallis, Ponting and Sangakkara</p><p>The batsmen with the most hundreds in test cricket are</p><ol><li>Sachin Tendulkar :<strong>Average:53.78,100’s — 51, 50’s — 68</strong></li><li>Jacques Kallis : <strong>Average: 55.47, 100’s — 45, 50’s — 58</strong></li><li>Ricky Ponting : <strong>Average: 51.85, 100’s — 41 , 50’s — 62</strong></li><li>Kumara Sangakarra: <strong>Average: 58.04 ,100’s — 38 , 50’s — 52</strong></li></ol><p>in that order.</p><p>The following plots take a closer at their performances. The box plots show the mean (red line) and median (blue line). The two ends of the boxplot display the 25th and 75th percentile.</p><h2>Box Histogram Plot</h2><p>This plot shows a combined boxplot of the Runs ranges and a histogram of the Runs Frequency. The calculated Mean differ from the stated means possibly because of data cleaning. Also not sure how the means were arrived at ESPN Cricinfo for e.g. when considering not out..</p><pre>batsmanPerfBoxHist(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*wEH26RdAze-vParo.png" /></figure><pre>batsmanPerfBoxHist(&quot;./kallis.csv&quot;,&quot;Jacques Kallis&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*witqc2rrYdrsGF1s.png" /></figure><pre>batsmanPerfBoxHist(&quot;./ponting.csv&quot;,&quot;Ricky Ponting&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*sTC3mHdAhWdrFGGp.png" /></figure><pre>batsmanPerfBoxHist(&quot;./sangakkara.csv&quot;,&quot;K Sangakkara&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*NSwu6H5-Di1Q5-3o.png" /></figure><h2>Contribution to won and lost matches</h2><p>The plot below shows the contribution of Tendulkar, Kallis, Ponting and Sangakarra in matches won and lost. The plots show the range of runs scored as a boxplot (25th &amp; 75th percentile) and the mean scored. The total matches won and lost are also printed in the plot.</p><p>All the players have scored more in the matches they won than the matches they lost. Ricky Ponting is the only batsman who seems to have more matches won to his credit than others. This could also be because he was a member of strong Australian team</p><pre>par(mfrow=c(2,2))<br>par(mar=c(4,4,2,2))<br>batsmanContributionWonLost(&quot;35320&quot;,&quot;Sachin Tendulkar&quot;)<br>batsmanContributionWonLost(&quot;45789&quot;,&quot;Jacques Kallis&quot;)<br>batsmanContributionWonLost(&quot;7133&quot;,&quot;Ricky Ponting&quot;)<br>batsmanContributionWonLost(&quot;50710&quot;,&quot;K Sangakarra&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*fSMA8enPEX8vn9_f.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Performance at home and overseas</h2><p>From the plot below it can be seen</p><p>Tendulkar has more matches overseas than at home and his performance is consistent in all venues at home or abroad. Ponting has lesser innings than Tendulkar and has an equally good performance at home and overseas.Kallis and Sangakkara’s performance abroad is lower than the performance at home.</p><pre>par(mfrow=c(2,2))<br>par(mar=c(4,4,2,2))<br>batsmanPerfHomeAway(&quot;35320&quot;,&quot;Tendulkar&quot;)<br>batsmanPerfHomeAway(&quot;45789&quot;,&quot;Kallis&quot;)<br>batsmanPerfHomeAway(&quot;7133&quot;,&quot;Ponting&quot;)<br>batsmanPerfHomeAway(&quot;50710&quot;,&quot;Sangakarra&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*JGH-P06a1Tono56O.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Relative Mean Strike Rate plot</h2><p>The plot below compares the Mean Strike Rate of the batsman for each of the runs ranges of 10 and plots them. The plot indicate the following Range 0–50 Runs — Ponting leads followed by Tendulkar Range 50 -100 Runs — Ponting followed by Sangakkara Range 100–150 — Ponting and then Tendulkar</p><pre>frames &lt;- list(&quot;./tendulkar.csv&quot;,&quot;./kallis.csv&quot;,&quot;ponting.csv&quot;,&quot;sangakkara.csv&quot;)<br>names &lt;- list(&quot;Tendulkar&quot;,&quot;Kallis&quot;,&quot;Ponting&quot;,&quot;Sangakkara&quot;)<br>relativeBatsmanSR(frames,names)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*3ExzQjbVNl9E1GdB.png" /></figure><h2>Relative Runs Frequency plot</h2><p>The plot below gives the relative Runs Frequency Percetages for each 10 run bucket. The plot below show</p><p>Sangakkara leads followed by Ponting</p><pre>frames &lt;- list(&quot;./tendulkar.csv&quot;,&quot;./kallis.csv&quot;,&quot;ponting.csv&quot;,&quot;sangakkara.csv&quot;)<br>names &lt;- list(&quot;Tendulkar&quot;,&quot;Kallis&quot;,&quot;Ponting&quot;,&quot;Sangakkara&quot;)<br>relativeRunsFreqPerf(frames,names)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*-ID5ILGNwsiwC53X.png" /></figure><h2>Moving Average of runs in career</h2><p>Take a look at the Moving Average across the career of the Top 4. Clearly . Kallis and Sangakkara have a few more years of great batting ahead. They seem to average on 50. . Tendulkar and Ponting definitely show a slump in the later years</p><pre>par(mfrow=c(2,2))<br>par(mar=c(4,4,2,2))<br>batsmanMovingAverage(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)<br>batsmanMovingAverage(&quot;./kallis.csv&quot;,&quot;Jacques Kallis&quot;)<br>batsmanMovingAverage(&quot;./ponting.csv&quot;,&quot;Ricky Ponting&quot;)<br>batsmanMovingAverage(&quot;./sangakkara.csv&quot;,&quot;K Sangakkara&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*0padt7lRTeSrsMrz.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><p>Future Runs forecast</p><p>Here are plots that forecast how the batsman will perform in future. In this case 90% of the career runs trend is uses as the training set. the remaining 10% is the test set.</p><p>A Holt-Winters forecating model is used to forecast future performance based on the 90% training set. The forecated runs trend is plotted. The test set is also plotted to see how close the forecast and the actual matches</p><p>Take a look at the runs forecasted for the batsman below.</p><ul><li>Tendulkar’s forecasted performance seems to tally with his actual performance with an average of 50</li><li>Kallis the forecasted runs are higher than the actual runs he scored</li><li>Ponting seems to have a good run in the future</li><li>Sangakkara has a decent run in the future averaging 50 runs</li></ul><pre>par(mfrow=c(2,2))<br>par(mar=c(4,4,2,2))<br>batsmanPerfForecast(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)<br>batsmanPerfForecast(&quot;./kallis.csv&quot;,&quot;Jacques Kallis&quot;)<br>batsmanPerfForecast(&quot;./ponting.csv&quot;,&quot;Ricky Ponting&quot;)<br>batsmanPerfForecast(&quot;./sangakkara.csv&quot;,&quot;K Sangakkara&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*TNQDYt2CtiQS0whv.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Check Batsman In-Form or Out-of-Form</h2><p>The below computation uses Null Hypothesis testing and p-value to determine if the batsman is in-form or out-of-form. For this 90% of the career runs is chosen as the population and the mean computed. The last 10% is chosen to be the sample set and the sample Mean and the sample Standard Deviation are caculated.</p><p>The Null Hypothesis (H0) assumes that the batsman continues to stay in-form where the sample mean is within 95% confidence interval of population mean The Alternative (Ha) assumes that the batsman is out of form the sample mean is beyond the 95% confidence interval of the population mean.</p><p>A significance value of 0.05 is chosen and p-value us computed If p-value &gt;= .05 — Batsman In-Form If p-value &lt; 0.05 — Batsman Out-of-Form</p><p><strong>Note</strong> Ideally the p-value should be done for a population that follows the Normal Distribution. But the runs population is usually left skewed. So some correction may be needed. I will revisit this later</p><p>This is done for the Top 4 batsman</p><pre>checkBatsmanInForm(&quot;./tendulkar.csv&quot;,&quot;Sachin Tendulkar&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 294  Mean of population: 50.48 <br>## Sample size: 33  Mean of sample: 32.42 SD of sample: 29.8 <br>## <br>## Null hypothesis H0 : Sachin Tendulkar &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : Sachin Tendulkar &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;Sachin Tendulkar &#39;s Form Status: Out-of-Form because the p value: 0.000713  is less than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><pre>checkBatsmanInForm(&quot;./kallis.csv&quot;,&quot;Jacques Kallis&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 240  Mean of population: 47.5 <br>## Sample size: 27  Mean of sample: 47.11 SD of sample: 59.19 <br>## <br>## Null hypothesis H0 : Jacques Kallis &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : Jacques Kallis &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;Jacques Kallis &#39;s Form Status: In-Form because the p value: 0.48647  is greater than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><pre>checkBatsmanInForm(&quot;./ponting.csv&quot;,&quot;Ricky Ponting&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 251  Mean of population: 47.5 <br>## Sample size: 28  Mean of sample: 36.25 SD of sample: 48.11 <br>## <br>## Null hypothesis H0 : Ricky Ponting &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : Ricky Ponting &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;Ricky Ponting &#39;s Form Status: In-Form because the p value: 0.113115  is greater than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><pre>checkBatsmanInForm(&quot;./sangakkara.csv&quot;,&quot;K Sangakkara&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 193  Mean of population: 51.92 <br>## Sample size: 22  Mean of sample: 71.73 SD of sample: 82.87 <br>## <br>## Null hypothesis H0 : K Sangakkara &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : K Sangakkara &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;K Sangakkara &#39;s Form Status: In-Form because the p value: 0.862862  is greater than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><p>3D plot of Runs vs Balls Faced and Minutes at Crease</p><p>The plot is a scatter plot of Runs vs Balls faced and Minutes at Crease. A prediction plane is fitted</p><pre>par(mfrow=c(1,2))<br>par(mar=c(4,4,2,2))<br>battingPerf3d(&quot;./tendulkar.csv&quot;,&quot;Tendulkar&quot;)<br>battingPerf3d(&quot;./kallis.csv&quot;,&quot;Kallis&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*GIKGw1QET_Wsa0I4.png" /></figure><pre>par(mfrow=c(1,2))<br>par(mar=c(4,4,2,2))<br>battingPerf3d(&quot;./ponting.csv&quot;,&quot;Ponting&quot;)<br>battingPerf3d(&quot;./sangakkara.csv&quot;,&quot;Sangakkara&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*v1hUMVodtp_OLoPl.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><p>Predicting Runs given Balls Faced and Minutes at Crease</p><p>A multi-variate regression plane is fitted between Runs and Balls faced +Minutes at crease. A sample sequence of Balls Faced(BF) and Minutes at crease (Mins) is setup as shown below. The fitted model is used to predict the runs for these values</p><pre>BF &lt;- seq( 10, 400,length=15)<br>Mins &lt;- seq(30,600,length=15)<br>newDF &lt;- data.frame(BF,Mins)<br>tendulkar &lt;- batsmanRunsPredict(&quot;./tendulkar.csv&quot;,&quot;Tendulkar&quot;,newdataframe=newDF)<br>kallis &lt;- batsmanRunsPredict(&quot;./kallis.csv&quot;,&quot;Kallis&quot;,newdataframe=newDF)<br>ponting &lt;- batsmanRunsPredict(&quot;./ponting.csv&quot;,&quot;Ponting&quot;,newdataframe=newDF)<br>sangakkara &lt;- batsmanRunsPredict(&quot;./sangakkara.csv&quot;,&quot;Sangakkara&quot;,newdataframe=newDF)</pre><p>The fitted model is then used to predict the runs that the batsmen will score for a given Balls faced and Minutes at crease. It can be seen Ponting has the will score the highest for a given Balls Faced and Minutes at crease.</p><p>Ponting is followed by Tendulkar who has Sangakkara close on his heels and finally we have Kallis. This is intuitive as we have already seen that Ponting has a highest strike rate.</p><pre>batsmen &lt;-cbind(round(tendulkar$Runs),round(kallis$Runs),round(ponting$Runs),round(sangakkara$Runs))<br>colnames(batsmen) &lt;- c(&quot;Tendulkar&quot;,&quot;Kallis&quot;,&quot;Ponting&quot;,&quot;Sangakkara&quot;)<br>newDF &lt;- data.frame(round(newDF$BF),round(newDF$Mins))<br>colnames(newDF) &lt;- c(&quot;BallsFaced&quot;,&quot;MinsAtCrease&quot;)<br>predictedRuns &lt;- cbind(newDF,batsmen)<br>predictedRuns</pre><pre>##    BallsFaced MinsAtCrease Tendulkar Kallis Ponting Sangakkara<br>## 1          10           30         7      6       9          2<br>## 2          38           71        23     20      25         18<br>## 3          66          111        39     34      42         34<br>## 4          94          152        54     48      59         50<br>## 5         121          193        70     62      76         66<br>## 6         149          234        86     76      93         82<br>## 7         177          274       102     90     110         98<br>## 8         205          315       118    104     127        114<br>## 9         233          356       134    118     144        130<br>## 10        261          396       150    132     161        146<br>## 11        289          437       165    146     178        162<br>## 12        316          478       181    159     194        178<br>## 13        344          519       197    173     211        194<br>## 14        372          559       213    187     228        210<br>## 15        400          600       229    201     245        226</pre><p>Analysis of Top 3 wicket takers</p><p>The top 3 wicket takes in test history are<br>1. M Muralitharan:Wickets: 800, Average = 22.72, Economy Rate — 2.47<br>2. Shane Warne: Wickets: 708, Average = 25.41, Economy Rate — 2.65<br>3. Anil Kumble: Wickets: 619, Average = 29.65, Economy Rate — 2.69</p><p>How do Anil Kumble, Shane Warne and M Muralitharan compare with one another with respect to wickets taken and the Economy Rate. The next set of plots compute and plot precisely these analyses.</p><h2>Wicket Frequency Plot</h2><p>This plot below computes the percentage frequency of number of wickets taken for e.g 1 wicket x%, 2 wickets y% etc and plots them as a continuous line</p><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>bowlerWktsFreqPercent(&quot;./kumble.csv&quot;,&quot;Anil Kumble&quot;)<br>bowlerWktsFreqPercent(&quot;./warne.csv&quot;,&quot;Shane Warne&quot;)<br>bowlerWktsFreqPercent(&quot;./murali.csv&quot;,&quot;M Muralitharan&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*X0niZMNsJbCxhb-z.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Wickets Runs plot</h2><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>bowlerWktsRunsPlot(&quot;./kumble.csv&quot;,&quot;Kumble&quot;)<br>bowlerWktsRunsPlot(&quot;./warne.csv&quot;,&quot;Warne&quot;)<br>bowlerWktsRunsPlot(&quot;./murali.csv&quot;,&quot;Muralitharan&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*-Xp1LRG1tR7fSBiJ.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Average wickets at different venues</h2><p>The plot gives the average wickets taken by Muralitharan at different venues. Muralitharan has taken an average of 8 and 6 wickets at Oval &amp; Wellington respectively in 2 different innings. His best performances are at Kandy and Colombo (SSC)</p><pre>bowlerAvgWktsGround(&quot;./murali.csv&quot;,&quot;Muralitharan&quot;)</pre><h2>Average wickets against different opposition</h2><p>The plot gives the average wickets taken by Muralitharan against different countries. The x-axis also includes the number of innings against each team</p><pre>bowlerAvgWktsOpposition(&quot;./murali.csv&quot;,&quot;Muralitharan&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*gLs7Sv6p1Vuv7UJf.png" /></figure><h2>Relative Wickets Frequency Percentage</h2><p>The Relative Wickets Percentage plot shows that M Muralitharan has a large percentage of wickets in the 3–8 wicket range</p><pre>frames &lt;- list(&quot;./kumble.csv&quot;,&quot;./murali.csv&quot;,&quot;warne.csv&quot;)<br>names &lt;- list(&quot;Anil KUmble&quot;,&quot;M Muralitharan&quot;,&quot;Shane Warne&quot;)<br>relativeBowlingPerf(frames,names)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*T5JnJzU-I7WaOrZr.png" /></figure><p>Relative Economy Rate against wickets taken</p><p>Clearly from the plot below it can be seen that Muralitharan has the best Economy Rate among the three</p><pre>frames &lt;- list(&quot;./kumble.csv&quot;,&quot;./murali.csv&quot;,&quot;warne.csv&quot;)<br>names &lt;- list(&quot;Anil KUmble&quot;,&quot;M Muralitharan&quot;,&quot;Shane Warne&quot;)<br>relativeBowlingER(frames,names)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*dzyhoQT7xY4tSgNG.png" /></figure><h2>Wickets taken moving average</h2><p>From th eplot below it can be see 1. Shane Warne’s performance at the time of his retirement was still at a peak of 3 wickets 2. M Muralitharan seems to have become ineffective over time with his peak years being 2004–2006 3. Anil Kumble also seems to slump down and become less effective.</p><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>bowlerMovingAverage(&quot;./kumble.csv&quot;,&quot;Anil Kumble&quot;)<br>bowlerMovingAverage(&quot;./warne.csv&quot;,&quot;Shane Warne&quot;)<br>bowlerMovingAverage(&quot;./murali.csv&quot;,&quot;M Muralitharan&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*KMgpwL_6SsLANmKs.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Future Wickets forecast</h2><p>Here are plots that forecast how the bowler will perform in future. In this case 90% of the career wickets trend is used as the training set. the remaining 10% is the test set.</p><p>A Holt-Winters forecating model is used to forecast future performance based on the 90% training set. The forecated wickets trend is plotted. The test set is also plotted to see how close the forecast and the actual matches</p><p>Take a look at the wickets forecasted for the bowlers below. — Shane Warne and Muralitharan have a fairly consistent forecast — Kumble forecast shows a small dip</p><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>bowlerPerfForecast(&quot;./kumble.csv&quot;,&quot;Anil Kumble&quot;)<br>bowlerPerfForecast(&quot;./warne.csv&quot;,&quot;Shane Warne&quot;)<br>bowlerPerfForecast(&quot;./murali.csv&quot;,&quot;M Muralitharan&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*7_nCyJK_X4rcG21y.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Contribution to matches won and lost</h2><p>The plot below is extremely interesting<br>1. Kumble wickets range from 2 to 4 wickets in matches wons with a mean of 3<br>2. Warne wickets in won matches range from 1 to 4 with more matches won. Clearly there are other bowlers contributing to the wins, possibly the pacers<br>3. Muralitharan wickets range in winning matches is more than the other 2 and ranges ranges 3 to 5 and clearly had a hand (pun unintended) in Sri Lanka’s wins</p><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>bowlerContributionWonLost(30176,&quot;Anil Kumble&quot;)<br>bowlerContributionWonLost(8166,&quot;Shane Warne&quot;)<br>bowlerContributionWonLost(49636,&quot;M Muralitharan&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*vAVkb-X5C2MI2m1Q.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><p>Performance home and overseas</p><p>From the plot below it can be seen that Kumble &amp; Warne have played more matches overseas than Muralitharan. Both Kumble and Warne show an average of 2 wickers overseas, Murali on the other hand has an average of 2.5 wickets overseas but a slightly less number of matches than Kumble &amp; Warne</p><pre>par(mfrow=c(1,3))<br>par(mar=c(4,4,2,2))<br>bowlerPerfHomeAway(30176,&quot;Kumble&quot;)<br>bowlerPerfHomeAway(8166,&quot;Warne&quot;)<br>bowlerPerfHomeAway(49636,&quot;Murali&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*DJ4-lJ-Hp1Gp0wtz.png" /></figure><pre>dev.off()</pre><pre>## null device <br>##           1</pre><h2>Check for bowler in-form/out-of-form</h2><p>The below computation uses Null Hypothesis testing and p-value to determine if the bowler is in-form or out-of-form. For this 90% of the career wickets is chosen as the population and the mean computed. The last 10% is chosen to be the sample set and the sample Mean and the sample Standard Deviation are caculated.</p><p>The Null Hypothesis (H0) assumes that the bowler continues to stay in-form where the sample mean is within 95% confidence interval of population mean The Alternative (Ha) assumes that the bowler is out of form the sample mean is beyond the 95% confidence interval of the population mean.</p><p>A significance value of 0.05 is chosen and p-value us computed If p-value &gt;= .05 — Batsman In-Form If p-value &lt; 0.05 — Batsman Out-of-Form</p><p><strong>Note</strong> Ideally the p-value should be done for a population that follows the Normal Distribution. But the runs population is usually left skewed. So some correction may be needed. I will revisit this later</p><p><strong>Note:</strong> The check for the form status of the bowlers indicate 1. That both Kumble and Muralitharan were out of form. This also shows in the moving average plot 2. Warne is still in great form and could have continued for a few more years. Too bad we didn’t see the magic later</p><pre>checkBowlerInForm(&quot;./kumble.csv&quot;,&quot;Anil Kumble&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 212  Mean of population: 2.69 <br>## Sample size: 24  Mean of sample: 2.04 SD of sample: 1.55 <br>## <br>## Null hypothesis H0 : Anil Kumble &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : Anil Kumble &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;Anil Kumble &#39;s Form Status: Out-of-Form because the p value: 0.02549  is less than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><pre>checkBowlerInForm(&quot;./warne.csv&quot;,&quot;Shane Warne&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 240  Mean of population: 2.55 <br>## Sample size: 27  Mean of sample: 2.56 SD of sample: 1.8 <br>## <br>## Null hypothesis H0 : Shane Warne &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : Shane Warne &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;Shane Warne &#39;s Form Status: In-Form because the p value: 0.511409  is greater than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><pre>checkBowlerInForm(&quot;./murali.csv&quot;,&quot;M Muralitharan&quot;)</pre><pre>## *******************************************************************************************<br>## <br>## Population size: 207  Mean of population: 3.55 <br>## Sample size: 23  Mean of sample: 2.87 SD of sample: 1.74 <br>## <br>## Null hypothesis H0 : M Muralitharan &#39;s sample average is within 95% confidence interval <br>##         of population average<br>## Alternative hypothesis Ha : M Muralitharan &#39;s sample average is below the 95% confidence<br>##         interval of population average<br>## <br>## [1] &quot;M Muralitharan &#39;s Form Status: Out-of-Form because the p value: 0.036828  is less than alpha=  0.05&quot;<br>## *******************************************************************************************</pre><pre>dev.off()</pre><pre>## null device <br>##           1</pre><p>Key Findings</p><p>The plots above capture some of the capabilities and features of my <strong>cricketr</strong> package. Feel free to install the package and try it out. Please do keep in mind ESPN Cricinfo’s Terms of Use.<br>Here are the main findings from the analysis above</p><h2>Analysis of Top 4 batsman</h2><p>The analysis of the Top 4 test batsman Tendulkar, Kallis, Ponting and Sangakkara show the folliwing</p><ol><li>Sangakkara has the highest average, followed by Tendulkar, Kallis and then Ponting.</li><li>Ponting has the highest strike rate followed by Tendulkar,Sangakkara and then Kallis</li><li>The predicted runs for a given Balls faced and Minutes at crease is highest for Ponting, followed by Tendulkar, Sangakkara and Kallis</li><li>The moving average for Tendulkar and Ponting shows a downward trend while Kallis and Sangakkara retired too soon</li><li>Tendulkar was out of form about the time of retirement while the rest were in-form. But this result has to be taken along with the moving average plot. Ponting was clearly on the way out.</li><li>The home and overseas performance indicate that Tendulkar is the clear leader. He has the highest number of matches played overseas and his performance has been consistent. He is followed by Ponting, Kallis and finally Sangakkara</li></ol><h2>Analysis of Top 3 legs spinners</h2><p>The analysis of Anil Kumble, Shane Warne and M Muralitharan show the following</p><ol><li>Muralitharan has the highest wickets and best economy rate followed by Warne and Kumble</li><li>Muralitharan has higher wickets frequency percentage between 3 to 8 wickets</li><li>Muralitharan has the best Economy Rate for wickets between 2 to 7</li><li>The moving average plot shows that the time was up for Kumble and Muralitharan but Warne had a few years ahead</li><li>The check for form status shows that Muralitharan and Kumble time was over while Warne still in great form</li><li>Kumble’s has more matches abroad than the other 2, yet Kumble averages of 3 wickets at home and 2 wickets overseas liek Warne . Murali has played few matches but has an average of 4 wickets at home and 3 wickets overseas.</li></ol><p>Final thoughts</p><p>Here are my final thoughts</p><h2>Batting</h2><p>Among the 4 batsman Tendulkar, Kallis, Ponting and Sangakkara the clear leader is Tendulkar for the following reasons</p><ol><li>Tendulkar has the highest test centuries and runs of all time.Tendulkar’s average is 2nd to Sangakkara, Tendulkar’s predicted runs for a given Balls faced and Minutes at Crease is 2nd and is behind Ponting. Also Tendulkar’s performance at home and overseas are consistent throughtout despite the fact that he has a highest number of overseas matches</li><li>Ponting takes the 2nd spot with the 2nd highest number of centuries, 1st in Strike Rate and 2nd in home and away performance.</li><li>The 3rd spot goes to Sangakkara, with the highest average, 3rd highest number of centuries, reasonable run frequency percentage in different run ranges. However he has a fewer number of matches overseas and his performance overseas is significantly lower than at home</li><li>Kallis has the 2nd highest number of centuries but his performance overseas and strike rate are behind others</li><li>Finally Kallis and Sangakkara had a few good years of batting still left in them (pity they retired!) while Tendulkar and Ponting’s time was up</li></ol><h2>Bowling</h2><p>Muralitharan leads the way followed closely by Warne and finally Kumble. The reasons are</p><ol><li>Muralitharan has the highest number of test wickets with the best Wickets percentage and the best Economy Rate. Murali on average gas taken 4 wickets at home and 3 wickets overseas</li><li>Warne follows Murali in the highest wickets taken, however Warne has less matches overseas than Murali and average 3 wickets home and 2 wickets overseas</li><li>Kumble has the 3rd highest wickets, with 3 wickets on an average at home and 2 wickets overseas. However Kumble has played more matches overseas than the other two. In that respect his performance is great. Also Kumble has played less matches at home otherwise his numbers would have looked even better.</li><li>Also while Kumble and Muralitharan’s career was on the decline , Warne was going great and had a couple of years ahead.</li></ol><p>You can download this analysis at <a href="https://drive.google.com/file/d/0Bxz66bp5PxPBd0JEUEg3eVlEQTQ/view?usp=sharing">Introducing cricketr</a></p><p>Hope you have fun using the cricketr package as I had in developing it.</p><p>From my blog <a href="https://gigadom.wordpress.com/">https://gigadom.wordpress.com/</a></p><p>Do take a look at my follow up post</p><p><a href="https://gigadom.wordpress.com/2015/07/06/taking-cricketr-for-a-spin-part-1/">Taking cricketr for a spin — Part 1</a></p><p>Also see<br>1. <a href="https://gigadom.wordpress.com/2015/02/06/analyzing-crickets-batting-legends-through-the-mirage-with-r/">Analyzing cricket’s batting legends — Through the mirage with R</a><br>2. <a href="https://gigadom.wordpress.com/2015/02/23/masters-of-spin-unraveling-the-web-with-r/">Masters of spin: Unraveling the web with R</a><br>3. <a href="https://gigadom.wordpress.com/2015/03/24/mirror-mirror-the-best-batsman-of-them-all/">Mirror,mirror …best batsman of them all</a></p><p>You may also like<br>1. <a href="https://gigadom.wordpress.com/2015/01/16/a-crime-map-of-india-in-r-crime-against-women/">A crime map of India in R: Crimes against women</a><br>2. <a href="https://gigadom.wordpress.com/2014/10/11/whats-up-watson-using-ibm-watsons-qaapi-with-bluemix-nodeexpress-part-1/">What’s up Watson? Using IBM Watson’s QAAPI with Bluemix, NodeExpress — Part 1</a><br>3. <a href="https://gigadom.wordpress.com/2014/11/05/bend-it-like-bluemix-mongodb-using-auto-scaling-part-2/">Bend it like Bluemix, MongoDB with autoscaling — Part 2</a><br>4. <a href="https://gigadom.wordpress.com/2014/12/12/informed-choices-through-machine-learning-analyzing-kohli-tendulkar-and-dravid/">Informed choices through Machine Learning : Analyzing Kohli, Tendulkar and Dravid</a><br>5. <a href="https://gigadom.wordpress.com/2015/03/31/thinking-web-scale-1-map-reduce-bring-compute-to-data/">Thinking Web Scale (TWS-3): Map-Reduce — Bring compute to data</a><br>6. <a href="https://gigadom.wordpress.com/2012/05/11/deblurring-with-opencv-weiner-filter-reloaded/">Deblurring with OpenCV:Weiner filter reloaded</a><br>7. <a href="https://gigadom.wordpress.com/2013/06/26/fun-simulation-of-a-chain-in-android/">Fun simulation of a Chain in Android</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c993db6b5aa2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimal Cloud Computing]]></title>
            <link>https://medium.com/@tvganesh_85/optimal-cloud-computing-b43c6e12080d?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/b43c6e12080d</guid>
            <category><![CDATA[instance]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[optimal]]></category>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Wed, 25 Feb 2015 03:47:57 GMT</pubDate>
            <atom:updated>2015-02-25T03:47:57.765Z</atom:updated>
            <content:encoded><![CDATA[<p>Optimal Cloud Computing</p><p>The murmur of cloud computing today, is bound to build up to crescendo in the years to come, simply because it makes a sound business sense. Cloud computing is a new paradigm in the world of computing. The cloud essentially creates an illusion of infinite computing resources that are available on demand to the user who only pays based on the usage. While on the surface it appears extremely simple and straightforward, making an optimal use of the cloud is no trivial task.</p><p>Prior to deploying on the cloud the enterprise has to decide the CPU, memory and bandwidth usage of the application. For e.g. the Amazon EC2 provides several variants of CPUs based on different pricing schemes namely $0.085/hr, $0.34/hr or $0.68/hr for small, large or extra large CPU instances. There are different pricing schemes for memory and bandwidth usage as well.</p><p>While the technological challenge of deploying the cloud is a separate endeavor in itself, the business considerations needed for deciding the cloud computing resources, optimally, is a separate and an equally important endeavor. This article focuses on the business considerations needed for making an optimal choice of resources while deploying on the cloud.</p><p>Since the enterprise is free to choose different CPUs which typically consists of CPU processors with different clock speeds or multi core CPUs for extra large instance the choice is really complicated.</p><p>The designer needs to consider how his application scales up with respect to increasing, decreasing or burst demands in traffic. To estimate the kind of resources that would be needed would require a good understanding of how the application scales with respect to increasing traffic. Ideally it will be remarkable if the application can scale linearly with increasing traffic. The key parameters that need to be considered for application performance is application latency and throughput versus the instance type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*WD7otG67101djxC2.jpg" /></figure><p>Also another consideration is to choose is the kind of resources types that need to be added. Ideally it would make more sense to add small CPU instances which can be added incrementally rather than adding extra large CPU instances which only handle part of the traffic. If we choose the large instance which is only partially used but has to be instantiated, nevertheless, to handle the extra traffic then it could result it wasting of precious resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*8ZiBUT6TJpLzLfnG.jpg" /></figure><p>A prime consideration is the choice of CPU resource type and the need to understand how the CPU loads up with increasing traffic with respect to latency and throughput. Once the CPU type, small, medium, large or extra large is chosen the designer needs to monitor how the loading of the CPU resource performs with increasing traffic.<br>Hence regardless of the choice there will be 3 windows of performance to consider</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*3SqMEFOe3rx8PtZm.jpg" /></figure><p><strong>Window of Optimality</strong>: In the optimal window the cost of cloud computing resources, for handling the incoming traffic versus revenue for the enterprise is truly profitable. In the optimal window the application will be capable of scaling extremely well to increasing traffic thus resulting in excellent revenue for the enterprise.</p><p>b) <strong>Window of Diminishing Returns</strong>: In this window the addition of extra resources at additional cost will not result in a proportional increase in scalability. In fact the increasing cost of adding additional resource will offset the revenue to the enterprise as the application will not scale appropriately and will result in diminishing returns.</p><p>c) <strong>Window of Loss</strong>: This is the window, in which no enterprise should not find itself in. In this window the cost of adding the extra resources will be larger than the revenue to the enterprise as an inordinate amount of resources will have to be added for small incremental increase in scalability. This will be the result of a poorly designed application. In this situation the enterprise must go back to the drawing room and re-architect the application.</p><p>Hence cloud computing, while truly alluring for the enterprise, it is a path that must be tread very carefully by the enterprise.</p><p>From my blog <a href="https://gigadom.wordpress.com/">Giga thoughts …</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b43c6e12080d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking baby steps in Lisp]]></title>
            <link>https://medium.com/@tvganesh_85/taking-baby-steps-in-lisp-8f010b273ed8?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f010b273ed8</guid>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Sun, 08 Jun 2014 10:47:56 GMT</pubDate>
            <atom:updated>2014-06-08T10:47:56.041Z</atom:updated>
            <content:encoded><![CDATA[<p>Lisp can be both fascinating and frustrating. Fascinating, because you can write compact code to solve really complex problems. Frustrating, because you can easily get lost in its maze of parentheses. I, for one, have been truly smitten by Lisp. My initial encounter with Lisp did not yield much success as I tried to come to terms with its strange syntax. The books I read on the Lisp language typically gloss over the exotic features of Lisp like writing Lisp code to solve the Towers of Hanoi or the Eight Queens problem. They talk about functions returning functions, back quotes and macros that can make your head spin.</p><p>I found this approach extremely difficult to digest the language. So I decided to view Lisp through the eyes of any other regular programming language like C, C++,, Java, Perl, Python or Ruby. I was keen on being able to do regular things with Lisp before I try out its unique features. So I decided to investigate Lisp from this view point and learn how to make Lisp do mundane things like an assignment, conditional, loop, array, input and output etc.</p><p>This post is centered on this fact.</p><p><strong>Assignment statement</strong></p><p>The most fundamental requirement for any language is to perform an assignment. For e.g. these are assignment statements in Lisp and its equivalent in C for e.g.</p><p>$ (setf x 5) -&gt; $ x = 5</p><p>$ (setf x (+ (* y 2) (* z 8)) -&gt; $x = 2y + 8z</p><p><strong>Conditional statement</strong></p><p>There are a couple of forms of conditional statement in Lisp. The most basic is the ‘if’ statement which is special case. You can do if-then-else without the possibility of if-then-else if-else if — else</p><p>if (condition) statement else-statement</p><p>In Lisp this is written as</p><p>$(setf x 5)</p><p>$ (if (= x 5)</p><p>(setf x (+ x 5))</p><p>(setf (- x 6)))</p><p>10</p><p>In C this equivalent to</p><p>$ x = 5</p><p>$ if (x == 5)</p><p>x = x + 5;</p><p>else</p><p>x = x -6;</p><p>However Lisp allows the if-then-else if — else if –else through the use of the COND statement</p><p>So we could write</p><p>$ (setf x 10)</p><p>$ (cond ((&lt; x 5) (setf x (+ x 8)) (setf y (* 2 y)))</p><p>((= x 10) (setf x (* x 2)))</p><p>(t (setf x 8)))</p><p>20</p><p>The above statement in C would be</p><p>$ x = 2</p><p>$ y = 10</p><p>$ if (x &lt; 5)</p><p>{</p><p>x = x + 8;</p><p>y = 2 * y;</p><p>}</p><p>else if (x == 10)</p><p>{</p><p>x = x * 2;</p><p>}</p><p>else</p><p>x = 8;</p><p><strong>Loops</strong></p><p>Lisp has many forms of loops dotimes, dolist, do , loop for etc. I found the following most intuitive and best to get started with</p><p>$ (setf x 5)</p><p>$ (let ((i 0))</p><p>(loop</p><p>(setf y (* x i))</p><p>(when (&gt; i 10) (return))</p><p>(print i) (prin1 y)</p><p>(incf i</p><p>)))</p><p>In C this could be written as</p><p>$ x = 5</p><p>(for i = 0; i &lt; 10; i++)</p><p>{</p><p>y = x * i</p><p>printf(“%d %d\n”,i,y);</p><p>}</p><p>Another easy looping construct in C is</p><p>(loop for x from 2 to 10 by 3</p><p>do (print x))</p><p>In C this would be</p><p>(for x=2; x &lt; 10; x = x+3)</p><p>print x;</p><p><strong>Arrays</strong></p><p>To create an array of 10 elements with initial value of 20</p><p>(setf numarray (make-array 10 :initial-element 20))</p><p>#(20 20 20 20 20 20 20 20 20 20)</p><p>To read an array element it is</p><p>$ (aref numarray 3) — — — &gt; numarray[3]</p><p>For e.g.</p><p>(setf x (* 2 (aref numarray 4))) — — — — &gt; x = numarray[4] * 2</p><p><strong>Functions</strong></p><p>(defun square (x)</p><p>(* x x))</p><p>This is the same as</p><p>int square (x)</p><p>{</p><p>return (x * x)</p><p>}</p><p>While in C you would invoke the function as</p><p>y = square (8)</p><p>In Lisp you would write as</p><p>(setf y (square 8))</p><p>Note: In Lisp the function is invoked as (function arg1 arg2… argn) instead of (function (arg1 arg2 … argn))</p><p><strong>Structures</strong></p><p>a) Create a global variable *db*</p><p>(defvar *db* nil)</p><p>b) Make a function to add an employee</p><p>$(defun make-emp (name age title)</p><p>(list :name name :age age :title title))</p><p>$(add-emp (make-emp “ganesh” 49 “manager”))</p><p>$(add-emp (make-emp “manish” 50 “gm”))</p><p>$(add-emp (make-emp “ram” 46 “vp”))</p><p>$ (dump-db)</p><p>For a more complete and excellent post on managing a simple DB looks at <a href="http://www.gigamonkeys.com/book/practical-a-simple-database.html">Practical Common Lisp</a> by Peter Siebel</p><p><strong>Reading and writing to standard output</strong></p><p>To write to standard output you can use</p><p>(print “This is a test”) or</p><p>(print ‘(This is a test))</p><p>To read from standard input use</p><p>(let ((temp 0))</p><p>(print ‘(Enter temp))</p><p>(setf temp (read))</p><p>(print (append ‘(the temp is) (list temp))))</p><p><strong>Reading and writing to a file</strong></p><p>The typical way to do this is to use</p><p>a) Read</p><p>(with-open-file (stream “C:\\acl82express\\lisp\\count.cl”)</p><p>(do ((line (read-line stream nil)</p><p>(read-line stream nil)))</p><p>((null line))</p><p>(print line)))</p><p>b) Write</p><p>(with-open-file (stream “C:\\acl82express\\lisp\\test.txt”</p><p>:direction :output</p><p>:if-exists :supersede)</p><p>(write-line “test” stream)</p><p>nil)</p><p>I found the following construct a lot easier</p><p>(let ((in (open “C:\\acl82express\\lisp\\count.cl” :if-does-not-exist nil)))</p><p>(when in</p><p>(loop for line = (read-line in nil)</p><p>while line do (format t “~a~%” line))</p><p>(close in)))</p><p>With the above you can get started on Lisp. However with just the above constructs the code one writes will be very “non-Lispy”. Anyway this is definitely a start.</p><p>[polldaddy rating=”7295683&quot;]</p><p><a href="https://plus.google.com/103077316191161424665/?rel=author">Find me on Google+</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f010b273ed8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The mind of a programmer]]></title>
            <link>https://medium.com/@tvganesh_85/the-mind-of-a-programmer-3eea2e6e9dd5?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eea2e6e9dd5</guid>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Sat, 17 May 2014 16:01:52 GMT</pubDate>
            <atom:updated>2014-05-17T16:12:18.738Z</atom:updated>
            <content:encoded><![CDATA[<p>Here is a short essay on the minds of the programmer and programming in general. Programming has been variously described as a science, an art, as black magic, as the work of craftsman etc. It is true, programming can be any or all of that described above. Programming in my opinion is going to become increasingly important in the years ahead. I would certainly advocate some knowledge and grasp of programming. There are many books that claim to teach programming anywhere between 3 to 21 days etc. This is not true. Learning to program is just the beginning of a never ending process. Here is a great piece by Peter Norvig — <a href="http://norvig.com/21-days.html">Teach yourself programming in 10 years</a>.</p><p>Programming can be considered to be a language to express your thoughts on the solution to a problem. The ability to express in a programming language can vary between being simply pedestrian to being absolutely poetic! There are those who can wax eloquent in a programming language. In any case, programming is a means to an end, the end being the solution to a problem. Typically the solution to the problem, is expressed as an algorithm, which is then is coded through a programming language,. Programming can be a highly analytical and creative activity.</p><p>Programming is different from most other professions that I can think of. To get started all you need is a computer and an Integrated Development Environments (IDE) for e.g. Eclipse, which can be downloaded for free. The IDE can be used for writing code. There are no other associated costs.</p><p>Programming is also different from other professions in the sense, that you get your response immediately. For e.g. a painter can paint anything and imagine that he/she is the next Rembrandt or Picasso. A guitarist can create the most hideous sound and think he is Jimi Hendrix’s re-incarnation. Other professions like architects, civil engineers, scientists have to wait for several months to know whether they are in the right direction or not. It is not so with programming. You write code. When you compile it or execute it, the verdict is instantaneous. It is simply a “no go”, if you are wrong. There is no middle path. You are either right or you are wrong.</p><p>Having said that, I would like to look at the typical experiences of a programmer?</p><p><strong>Tears, sweat and frustration</strong>: In the beginning programming is usually very intimidating and frustrating. In the initial stages when you grapple with the quirky syntax of the language, and try to formulate your thoughts around the problem, you will hit many speed bumps. It can be exhausting, tiring and nerve racking. There are no shortcuts in learning how to program. You have to go through the grind, memorize certain phrases and hope that your program works. Once you have you arms around the syntax, you are on your way to actually writing code that achieves something. Here again you will run into all sorts of problems, like loops that never end, inexplicable program crashes and mysterious run time errors etc. The early stages can be difficult and quite unforgiving. This phase requires patience to get through.</p><p><strong>Feelings of megalomania</strong>: Someone with 5 to 7 years of programming experience knows most of the typical constructs by heart and will be able to quickly churn out programs, rather fast. This is a dangerous phase. Since you have been doing the same thing for a couple of years you are typically aware of the problems and can possibly tweak code to make it solve a slightly different problem. This is usually the stage when programmers start to experience a sense of megalomania. There are delusions of grandeur. You may remember the programmer shown in Golden Eye who keeps saying “I am invincible!” whenever he is able to solve a knotty problem. These programmers have the feeling that “Nothing is impossible”</p><p><strong>Programming is a great leveler. </strong>Programming can be a great boost to your ego. When you are able to visualize a problem, strategize the solution and actually get it to work, it does wonders to your ego. Programming can really boost your self-esteem. But you should not just stick to your comfort zone and write code in exactly the same language in exactly the same domain. It really helps to move to a different language, preferably a different paradigm — for example a move from procedural (C) to Object Oriented (Java, C++) or from object oriented to functional (Lisp, Haskell). Similarly moving from Web programming to protocol design or from data communication to app design will do wonders. The shift to a new programming paradigm and new technical domain will put you on even keel. All your knowledge and expertise will evaporate when you move to a new domain. Moving around in technology will keep you more grounded. You will realize that there is still so much more to learn. There is yet another universe.</p><p>In other words, programming keeps you honest!</p><p>My journey of 25+ years as a programming has helped me to learn technology in all its flavors. More importantly I was able to learn about myself. I have seen it all. Sweat, tears, frustration, fear, anger, pride and ecstasy.</p><p>.</p><p>A few years back, once you learned the basics, if your work did not involve coding, there was not much to do. But these days you can really do some fun things. You can imagine any app you want and actually start to realize it. Who knows, your app may be the next block buster! I am certain all of us have ideas which we want to implement. Programming allows you to just that!</p><p>Programming really makes you exercise your grey cells. Who knows we will soon hear that research has proved that programming helps prevent Alzheimer’s and Parkinson’s disease.:-)</p><p>In any case, learning to program is one good thing.</p><p>From my blog <a href="http://gigadom.wordpress.com/">Giga thoughts …</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eea2e6e9dd5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thought : Deliberate and deliberated]]></title>
            <link>https://medium.com/@tvganesh_85/thought-deliberate-and-deliberated-a7ea4bbe41f9?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7ea4bbe41f9</guid>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Thu, 27 Mar 2014 09:43:34 GMT</pubDate>
            <atom:updated>2014-03-27T09:43:34.920Z</atom:updated>
            <content:encoded><![CDATA[<p>Take a look at the face of the woman on the left. What do you see? It should be immediately apparent to you that the woman is on the verge of exploding with anger. Possibly also in the process of exchanging some choice words.</p><p>Now look at the following problem 17 x 34.</p><p>This bound to take some thought. You are immediately aware that this is multiplication problem. But in order to work it out mentally or with pen and paper, requires considerable mental effort.</p><p>The above 2 examples are from the book “Thinking, fast and slow” by Nobel laureate, Daniel Kahneman. Kahneman refers to the former mode of thinking as using System 1, which is fast, automatic and immediately apparent. The latter mode, which requires deliberate thought requires the use of System 2, and involves effortful mental activity.</p><p>While System 1 mode of thinking is both intuitive and fast and is our evolutionary response to quickly responding to many things. On the other hand System 2 is methodical is slow and has to be worked out,</p><p>System 1 &amp; System 2 are constantly at work with System 1 aiding any work done by System 2. This post is not a review about the book itself. However, I would like to use the above discussion for another related aspect of the way we conduct our lives. Here is another interesting example</p><p>A bat &amp; ball cost $. 1.10. The bat costs $1 more than the ball. How much does the ball cost?</p><p>If your answer was 10c then you were wrong. With a pen and paper or with a little conscious thought you will realize that the bat would need to cost $1.05 and the ball 5c.</p><p>So this is what I would like to focus on. We as human beings are lazy thinkers. We rush to conclusions without much thought. We fail to work System 2 appropriately, instead take the easy course of taking the immediate response of System 1.</p><p>Here are some examples where we would be much better off if we spend a little extra time pondering before we make a decision or come to a conclusion.</p><p>For e.g. when we learn a new skill for e.g. to play a musical instrument, painting or even taking a new role at your office. There are typically 3 stages in learning. Initially you will have a steep learning curve when you will be forced to exercise System 2. In the 2nd stage you would have acquired the skill and you will be able to go on autopilot without much work for System 2. Usually once people reach this stage they get used to comfort of the learned ease and just plateau at this point. If you really want to improve then in the 2nd stage if you make a conscious effort to learn where your shortcomings are you can move from the average to an expert.</p><p>Other instances where lazy thinking is in full force is when we jump to conclusions based on inbuilt biases and opinions. We never take a moment to reflect. We intensely dislike someone based on our System 1’s evaluation and we continue to dislike the person. If only we set our System 2 in motion and tried to understand why we hate somebody we may realize that there was no basis for the hatred.</p><p>While System 1 is there for a reason it does not make sense to use System 1 for all situations in our life. We need to get out of our lazy thinking style and take a more active role not only in our lives but also in our thought.</p><p>In other words, we need both deliberate and deliberated thought!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7ea4bbe41f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling out]]></title>
            <link>https://medium.com/@tvganesh_85/scaling-out-a778ff17991?source=rss-291dd97ccb01------2</link>
            <guid isPermaLink="false">https://medium.com/p/a778ff17991</guid>
            <dc:creator><![CDATA[Tinniam V Ganesh]]></dc:creator>
            <pubDate>Thu, 27 Mar 2014 09:39:31 GMT</pubDate>
            <atom:updated>2014-03-27T09:41:53.742Z</atom:updated>
            <content:encoded><![CDATA[<p>Web Applications have challenges that are unique to the domain of the web. The key differentiating fact between internet technologies and other technologies is the need to scale up to handle sudden and large increases in traffic. While telecommunication and data communication also have the need to handle high traffic these products can be dimensioned on some upper threshold. Typical web applications have the need to provide low latencies, handle large throughput and also be extremely scalable to changing demands.</p><p>The ability to scale seems trivial. It appears that one could just add more CPU horsepower and throw in memory and bandwidth in large measure to get the performance we need. But unfortunately this is not as simple as it appears. For one, adding more CPU horsepower may not necessarily result in better performance. A badly designed application will only improve marginally.</p><p>Some applications are “embarrassingly parallel”. There is no dependency of one task on another. For example if the task is to search for a particular string in documents or one that requires the conversion from AVI to MPEG format then this can be done in parallel. In a public cloud this could be achieved by running more instances.</p><p>However, most real world applications are more sequential than parallel. There is a lot of dependency of data between the modules of the application. When multiple instances have to run in a public cloud the design considerations can be quite daunting.</p><p>For example if we had an application with parts 1,2,3,4 as follows</p><p>Now let us further assume that this is a web application and thousands of requests come to it. For simplicity sake, if we assume that for each request there is counter that has to be incremented. How does one keep track of the total requests that are coming to application? The challenge is how to manage such a global counter when there are multiple instances. In a monolithic application this does not pose a problem. But with multiple instances handling web requests, each having its own copy of this counter the design becomes a challenge</p><p>Each instance has its own copy of the counter which it will update based on the requests that come to that instance through a load balancer. However, how does one compute the total number of requests that e come to all the instances</p><p>One possible solution is to use the memcached approach. Memcached was developed as solution by Danga Corporation for Livejournal. Memcached is a distributed caching mechanism that stores data in multiple participating servers. Memcached has some simple API calls like get(key) and set (key,value) . Memcached uses a consistent hashing mechanism which hashes the key to one of the servers among several servers. This method is known as the Distributed Hashing Table (DHT) by which it is able to distribute the keys to one of the servers. The Consistent Hashing technique is able to handle server crashes and new servers joining in the distributed cache. Since data is distributed among servers participating in the distributed cache when a server crashes all its data is distributed to the remaining servers. Similarly a server joining in the distributed cache also distributes some of the data to it. Memcached has been used in Facebook, Zynga and Livejournal.</p><p>From my blog <a href="http://gigadom.wordpress.com/">http://gigadom.wordpress.com/</a></p><p><a href="https://plus.google.com/103077316191161424665/?rel=author">Find me on Google+</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a778ff17991" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>