<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shailaja Gupta on Medium]]></title>
        <description><![CDATA[Stories by Shailaja Gupta on Medium]]></description>
        <link>https://medium.com/@shailajagupta?source=rss-673d053b01dd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-O_fWPGv8mjbPq7T.</url>
            <title>Stories by Shailaja Gupta on Medium</title>
            <link>https://medium.com/@shailajagupta?source=rss-673d053b01dd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:24:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shailajagupta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pros and cons of various Classification ML algorithms]]></title>
            <link>https://medium.com/data-science/pros-and-cons-of-various-classification-ml-algorithms-3b5bfb3c87d6?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b5bfb3c87d6</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[ai-and-data-science]]></category>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Fri, 28 Feb 2020 13:30:35 GMT</pubDate>
            <atom:updated>2020-06-23T04:45:39.702Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntdyE4PiFxIc34ikHBMkPg.jpeg" /><figcaption>Source: Pixabay</figcaption></figure><h3>Pros and cons of various Machine Learning algorithms</h3><p>There are many classification algorithms in machine learning. But ever wondered which algorithm should be used for what purpose and what kind of application. If yes, then please read the pros and cons of various machine learning algorithms used in classification. I have also listed down their use cases and applications.</p><h3>SVM (Support Vector Machine)</h3><p><strong>Pros</strong></p><ol><li><strong>Performs well in Higher dimension.</strong> In real world there are infinite dimensions (and not just 2D and 3D). For instance image data, gene data, medical data etc. has higher dimensions and SVM is useful in that. Basically when the number of features/columns are higher, SVM does well</li></ol><p>2. <strong>Best algorithm when classes are separable. </strong>(when instances of both the classes can be easily separated either by a straight line or non-linearly). To depict separable classes, lets take an example(here taking an example of linear separation, classes can also be non-linearly separable, by drawing a parabola for e.g. etc). In first graph you cannot tell easily whether X will be in class 1 or 2, but in case 2 you can easily tell that X is in class 2. Hence in second case classes are linearly separable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/0*mxdeoWu0p602Jj38.png" /></figure><p>First is non-separable class, second is separable class.</p><p>3. <strong>Outliers </strong>have less impact.</p><p>4. SVM is suited for extreme case binary classification.</p><p><strong>Cons:</strong></p><p>1. <strong>Slow: </strong>For larger dataset, it requires a large amount of time to process.</p><p>2. <strong>Poor performance with Overlapped classes</strong> : Does not perform well in case of overlapped classes.</p><p>3.<strong> Selecting appropriate hyperparameters is important: </strong>That will allow for sufficient generalization performance.</p><p>4. <strong>Selecting the appropriate kernel </strong>function can be tricky.</p><p><strong>Applications:</strong></p><p>Bag of words application(many features and columns), speech recognition data, classification of images(non-linear data), medical analytics(non linear data), text classification(many features)</p><h3>Naive Bayes</h3><p><strong>Pros</strong></p><ol><li><strong>Real time </strong>predictions: It is very fast and can be used in real time.</li></ol><p>2. <strong>Scalable </strong>with Large datasets</p><p>3. <strong>Insensitive to irrelevant features.</strong></p><p>4. <strong>Multi class prediction</strong> is effectively done in Naive Bayes</p><p>5. <strong>Good performance with high dimensional data</strong>(no. of features is large)</p><p><strong>Cons</strong></p><ol><li><strong>Independence of features does not hold: </strong>The fundamental Naive Bayes assumption is that each feature makes an independent and equal contribution to the outcome. However this condition is not met most of the times.</li></ol><p>2. <strong>Bad estimator: </strong>Probability outputs from predict_proba are not to be taken too seriously.</p><p>3. <strong>Training data should represent population well: </strong>If you have no occurrences of a class label and a certain attribute value together (e.g. class=”No”, shape=”Overcast “) then the posterior probability will be zero. So if the training data is not representative of the population, Naive bayes does not work well.(This problem is removed by smoothening techniques).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*yNXK_vC7X02AyPCS.png" /></figure><p><strong>Applications:</strong></p><p>Naive Bayes is used in Text classification/ Spam Filtering/ Sentiment Analysis. It is used in text classification (it can predict on multiple classes and doesn’t mind dealing with irrelevant features), Spam filtering (identify spam e-mail) and Sentiment Analysis (in social media analysis, to identify positive and negative sentiments), recommendation systems (what will the user buy next)</p><h3>Logistic Regression</h3><p><strong>Pros</strong></p><ol><li><strong>Simple</strong> to implement</li></ol><p>2.<strong> Effective</strong></p><p>3.<strong> Feature scaling not needed: </strong>Does not require input features to be scaled (can work with scaled features too, but doesn’t require scaling)</p><p>3.<strong> Tuning of hyperparameters not needed.</strong></p><p><strong>Cons</strong></p><ol><li><strong>Poor performance on non-linear data</strong>(image data for e.g)</li></ol><p>2. <strong>Poor performance with irrelevant and highly correlated features </strong>(use Boruta plot for removing similar or correlated features and irrelevant features).</p><p>3. <strong>Not very powerful</strong> algorithm and can be easily outperformed by other algorithms.</p><p>4. <strong>High reliance on proper presentation of data</strong>. All the important variables / features should be identified for it to work well.</p><p><strong>Applications:</strong></p><p>Any classification problem that is preferably binary (it can also perform multi class classification, but binary is preferred). For example you can use it if your output class has 2 outcomes; cancer detection problems, whether a student will pass/fail, default/no default in case of customer taking loan, whether a customer will churn or not, email is spam or not etc.</p><h3>Random Forest</h3><p><strong>Pros:</strong></p><p>1. Random forest can <strong>decorrelate trees</strong>. It picks the training sample and gives each tree a subset of the features(suppose training data was [1,2,3,4,5,6], so one tree will get subset of training data [1,2,3,2,6,6]. Note that size of training data remains same, both datas have length 6 and that feature ‘2’ and feature ‘6’ are repeated in the randomly sampled training data given to one tree. Each tree predicts according to the features it has. In this case tree 1 only has access to features 1,2,3 and 6 so it can predict based on these features. Some other tree will have access to features 1,4,5 say so it will predict according to those features. If features are highly correlated then that problem can be tackled in random forest.</p><p>2. <strong>Reduced error:</strong> Random forest is an ensemble of decision trees. For predicting the outcome of a particular row, random forest takes inputs from all the trees and then predicts the outcome. This ensures that the individual errors of trees are minimized and overall variance and error is reduced.</p><p>3. <strong>Good Performance on Imbalanced datasets</strong> : It can also handle errors in imbalanced data (one class is majority and other class is minority)</p><p>4.<strong> Handling of huge amount of data: </strong>It can handle huge amount of data with higher dimensionality of variables.</p><p>5.<strong> Good handling of missing data: </strong>It can handle missing data very well. So if there is large amount of missing data in your model, it will give good results.</p><p>6. <strong>Little impact of outliers: </strong>As the final outcome is taken by consulting many decision trees so certain data points which are outliers will not have a very big impact on Random Forest.</p><p>7. <strong>No problem of overfitting: </strong>In Random forest considers only a subset of features, and the final outcome depends on all the trees. So there is more generalization and less overfitting.</p><p>8. <strong>Useful to extract feature importance </strong>(we can use it for feature selection)</p><p><strong>Cons:</strong></p><p>1. <strong>Features </strong>need to have <strong>some predictive power </strong>else they won’t work.</p><p>2. <strong>Predictions of the trees need to be uncorrelated</strong>.</p><p>3. <strong>Appears as Black Box:</strong> It is tough to know what is happening. You can at best try different parameters and random seeds to change the outcomes and performance.</p><p><strong>Applications</strong>:</p><p>Credit card default, fraud customer/not, easy to identify patient’s disease or not, recommendation system for ecommerce sites.</p><p><strong>Decision Trees</strong></p><p><strong>Pros</strong></p><p>1. <strong>Normalization or scaling of data not needed</strong>.</p><p>2. <strong>Handling missing values</strong>: No considerable impact of missing values.</p><p>3. <strong>Easy to explain </strong>to non-technical team members.</p><p>4. <strong>Easy visualization</strong></p><p>5. <strong>Automatic</strong> <strong>Feature selection</strong> : Irrelevant features won’t affect decision trees.</p><p><strong>Cons</strong></p><p>1. <strong>Prone to overfitting.</strong></p><p>2. <strong>Sensitive to data. </strong>If data changes slightly, the outcomes can change to a very large extent.</p><p>3. <strong>Higher time required to train </strong>decision trees.</p><p><strong>Applications</strong>:</p><p>Identifying buyers for products, prediction of likelihood of default, which strategy can maximize profit, finding strategy for cost minimization, which features are most important to attract and retain customers (is it the frequency of shopping, is it the frequent discounts, is it the product mix etc), fault diagnosis in machines(keep measuring pressure, vibrations and other measures and predict before a fault occurs) etc.</p><h3>XGBoost</h3><p><strong>Pros</strong></p><p>1. <strong>Less feature engineering required </strong>(No need for scaling, normalizing data, can also handle missing values well)</p><p>2. <strong>Feature importance</strong> can be found out(it output importance of each feature, can be used for feature selection)</p><p>3. <strong>Fast </strong>to interpret</p><p>4. <strong>Outliers </strong>have minimal impact.</p><p>5. <strong>Handles large sized datasets</strong> well.</p><p>6. <strong>Good Execution</strong> speed</p><p>7. <strong>Good model performance</strong> (wins most of the Kaggle competitions)</p><p>8. <strong>Less prone to overfitting</strong></p><p><strong>Cons</strong></p><p>1. <strong>Difficult interpretation </strong>, visualization tough</p><p>2. <strong>Overfitting</strong> possible if parameters not tuned properly.</p><p>3. <strong>Harder to tune</strong> as there are too many hyperparameters.</p><p><strong>Applications</strong></p><p>Any classification problem. Specially useful if you have too many features and too large datasets, outliers are present, there are many missing values and you don’t want to do much feature engineering. It wins almost all competitions so this is an algo you must definitely have in mind while solving any classification problem.</p><h3>k-NN (K Nearest Neighbors)</h3><p><strong>Pros</strong></p><p>1. <strong>Simple </strong>to understand and impelment</p><p>2. <strong>No assumption about data </strong>(for e.g. in case of linear regression we assume dependent variable and independent variables are linearly related, in Naïve Bayes we assume features are independent of each other etc., but k-NN makes no assumptions about data)</p><p>3. <strong>Constantly evolving</strong> model: When it is exposed to new data, it changes to accommodate the new data points.</p><p>4. <strong>Multi-class</strong> problems can also be solved.</p><p>5. <strong>One Hyper Parameter:</strong> K-NN might take some time while selecting the first hyper parameter but after that rest of the parameters are aligned to it.</p><p><strong>Cons</strong></p><p>1. <strong>Slow</strong> for large datasets.</p><p>2. <strong>Curse of dimensionality</strong>: Does not work very well on datasets with large number of features.</p><p>3. <strong>Scaling</strong> of data absolute must.</p><p>4. <strong>Does not work well on Imbalanced data.</strong> So before using k-NN either undersamplemajority class or oversample minority class and have a balanced dataset.</p><p>5. Sensitive to <strong>outliers</strong>.</p><p>6. Can’t deal well with <strong>missing values</strong></p><p><strong>Applications:</strong></p><p>You can use it for any classification problem when dataset is smaller, and has lesser number of features so that computation time taken by k-NN is less. If you do not know the shape of the data and the way output and inputs are related (whether classes can be separated by a line or ellipse or parabola etc.), then you can use k-NN.</p><p><strong>Conclusion</strong></p><p>As the models came later , they became more and more advanced. So usually if you look at a model strictly from the performance perspective, then neural networks, XGBoost etc. are the best models since they are relatively recent. However , different models work better for different data.</p><p>For e.g. if features are highly independent then Naive Bayes will work great. If there are too many features and dataset is medium sized, SVM is good. If there is linear relationship between dependent and independent variables, then linear regression, logistic regression, SVM are good. If dataset is small and you don’t know the relationship between the dependent and independent variable, then you can use k-NN. Hence before you decide on which ML algorithm to use, you must know and analyse the data. If you are unable to zero in on one machine learning algorithm, then you can evaluate all the models and check their accuracies on training and test set and then finalize one model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b5bfb3c87d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pros-and-cons-of-various-classification-ml-algorithms-3b5bfb3c87d6">Pros and cons of various Classification ML algorithms</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to tackle any classification problem end to end & choose the right classification ML algorithm.]]></title>
            <link>https://medium.com/data-science/how-to-tackle-any-classification-problem-end-to-end-choose-the-right-classification-ml-algorithm-4d0becc6a295?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d0becc6a295</guid>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Fri, 28 Feb 2020 12:21:41 GMT</pubDate>
            <atom:updated>2020-02-28T15:32:34.901Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEVogcnMzoploVuXZ885HA.jpeg" /><figcaption>Source: Pixabay</figcaption></figure><p>We are aware of many machine learning algorithms now. However when I was learning ML , I was unable to find an article covering all the steps of machine learning including data preprocessing, choosing an ML algorithms and then narrowing down on one for various classification problems. Hence I wrote this article so that anyone can find all the resources at one place. I have also covered the pros and cons of various ML algorithms in the article so that you know when to use which type of algorithm. You can simply copy paste the code templates given in this article and apply to any classification problem at hand. Let us begin :)</p><h3><strong>A. Dataset download and preparation</strong></h3><ol><li>Download the dataset from <a href="https://www.kaggle.com/c/digit-recognizer/data">here</a>.</li><li>About dataset: MNIST is a very famous dataset. This dataset contains 784 columns representing the pixels of a digit (Every digit is made up of 256 pixels on the x-axis and 256 pixels on the y-axis. 256 x 256 pixels = 784 pixels). It has a column label which identifies the digit from the 784 pixels.</li><li>In the below article, we will classify a digit as 5 or not 5. We will thus deal with binary classification for the sake of simplicity. Also, it is seen that most of the classification problems are binary classification problems. Multi-class classification (classifying digits from 0 to 9) will be dealt with in another article.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fu0-fSFTJRRA2l7oMDpTUw.png" /></figure><h3>B. Data Pre-Processing before applying ML Models</h3><ol><li>Handling missing values</li><li>Handling non-numeric data (handling categorical, ordinal variables and strings)</li><li>Feature engineering and creating new features out of existing features.</li><li>Extracting only useful and relevant features: Feature selection</li><li>Remove features with missing values above a particular threshold</li><li>Splitting data into Training and test sets</li><li>Checking for Class imbalance and tackling it</li><li>Standard scaling of data</li></ol><h3><strong>Data Pre-processing Procedure</strong></h3><p>Since MNIST dataset does not require much pre-processing(no missing values, no string or categorical features/columns etc.),we have taken a simple dataset known as <strong>housing dataset and illustrated most of the data pre — processing techniques using that dataset</strong>. In the dataset, we need to predict whether someone will default or not based on the learnings from the past data. The profile of people who were given loans in the past is given and also whether they defaulted or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*ortA3r1Ar7IhOYsGJNVtXA.png" /><figcaption>Housing dataset</figcaption></figure><p>Full code for data pre-processing of housing dataset is given below:</p><p><a href="https://github.com/Shailaja-Gupta/Data-Preprocessing-for-medium/blob/master/Github%20Medium%20article/Data_preprocessing_for_Medium.ipynb">Shailaja-Gupta/Data-Preprocessing-for-medium</a></p><p><strong>B.1. Handling missing values </strong>(housing dataset)</p><p>If we have missing values, then it is tough to implement most of the ML algorithms . Hence we need to handle them first. One way to do that is using Imputer (Imputer works like fillna).</p><p>There can be different type of missing values. For e.g. we could have missing string data, missing categorical data and numerical data.Hence we have first segregated the numeric and non-numeric columns and then filled their NAN values using different methods. NAN values of numeric columns are filled for instance here using median and NAN values of string columns are filled using ‘most_frequent’.</p><pre>from sklearn.impute import SimpleImputer</pre><pre>cols_remove=[&#39;Name&#39;]  #col names which we wish to remove from model<br>X_train = X_train.drop(cols_remove, axis=1)<br>cols = X_train.columns #new coumns without the columns removed<br>X_train_numeric=X_train.select_dtypes(include=[&#39;int&#39;, &#39;float&#39;]).columns<br>X_train_non_numeric=X_train.select_dtypes(exclude=[&#39;int&#39;, &#39;float&#39;]).columns</pre><pre>#joined columns numeric and non numeric<br>new = X_train_numeric.tolist() + X_train_non_numeric.tolist()</pre><pre>from sklearn.impute import SimpleImputer<br>from sklearn.compose import ColumnTransformer<br>t = [(&#39;num&#39;, SimpleImputer(strategy=&#39;median&#39;), X_train_numeric),<br>(&#39;cat&#39;, SimpleImputer(strategy=&#39;most_frequent&#39;), X_train_non_numeric)]<br>transformer = ColumnTransformer(transformers=t, remainder=&#39;passthrough&#39;)<br>X_train = transformer.fit_transform(X_train) #numpy array</pre><pre>#DataFrame constructor with new columns names and added reindex for change by original order<br>X_train = pd.DataFrame(X_train, columns=new).reindex(cols, axis=1)</pre><p><strong>B.2. Handling non-numeric data </strong>(housing dataset)</p><p><strong>All </strong>your <strong>input data (X_train </strong>or <strong>X_test </strong>data<strong>) must be numeric </strong>data <strong>before </strong>you can do<strong> classification</strong>! However, from your experience, you must know by now, that this will not always be the case. You will have non-numeric data too most of the times. Non-numeric data can be of 3 types: categorical, ordinal and string data. In the dataset of housing, type of job is ordinal data, gender is categorical data,Name here is string data. The difference between these types of data is:</p><ul><li><strong>Categorical/Nominal data:</strong> Gender: Male and Female are neither worse nor better here. They are just categories of people’s genders.</li><li><strong>Ordinal data:</strong> Rating: There is a definite order among variables. Super skilled&gt;Skilled&gt;Unskilled.</li><li><strong>String data:</strong> Name:This data is similar to categorical data in the sense that names are just labels. However there is a difference. In the case of categorical variables, there are only fixed number of categories. For the name field on the other hand, there will be many names and none of them may be the same.</li></ul><p>Lets see how to deal with each of these data types:</p><p><strong>B.2.1. Handling categorical data </strong>(variables do not have any order or rank and specify different categories.):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*3jlX27brFopzLG-E4gzu2w.png" /><figcaption>We use only one variable M to specify whether a person is M/F. We use drop_first=True in get_dummies to do that.</figcaption></figure><pre>from sklearn.preprocessing import OneHotEncoder<br>onehotencoder = pd.get_dummies(X[&#39;Gender&#39;], drop_first=True)<br># Drop column non_numeric_col as it is now encoded<br>X = X.drop(&#39;Gender&#39;,axis = 1)<br># Join the encoded dataframe to X<br>X = X.join(onehotencoder)</pre><p><strong>B.2.2. Handling ordinal data</strong>: (Data where variables have a rank)</p><p>In the column ‘Type of job’, Super skilled&gt;Skilled&gt;Unskilled. We can tell the machine that one category is better than the other by assigning some numerical value to all these 3 labels. The higher the number the better it will be considered by ML Algorithms. Since we give 3 to Super skilled, 2 to skilled and 1 to unskilled; ML Algorithms will assume that Super skilled&gt; Skilled&gt; Unskilled , which is what we wanted.</p><pre>X[&#39;Type of job&#39;] = X[&#39;Type of job&#39;].map( {&#39;Super Skilled&#39;:3, &#39;Skilled&#39;:2, &#39;Unskilled&#39;:1})</pre><p><strong>B.2.3. Handling string data</strong></p><p>In the above example, we have string data ‘Name’. The string data can in no way be converted into numbers. Can we extract some useful information from this field Name? The answer here is No. (however if we had some prefix before the name Mr., Miss, Mrs etc. we could have extracted some information from the field and hence worked on it, but right now no info can be extracted). Hence we remove the field from our modelling X_train.</p><pre>#You can specify all the string columns you want to remove<br>cols_remove=[&#39;Name&#39;]<br>X = X_train.drop(cols_remove, axis=1)</pre><p><strong>B.3. Feature engineering</strong></p><p>Sometimes we can create new features from the given dataset or tweak existing features to extract more valuable information. All this requires domain knowledge. For example in the given dataset, we can create new features such as %Years employed and Credit income%. This gives an idea that how many years a person was actually working and what percentage of his income is credit. If he has more credit according to his income, then chances of default are more. If the person was working most of his life then, his chances of default are less.</p><pre>X[&#39;Years employed percent&#39;] = X[&#39;Years employed&#39;] / X[&#39;Age&#39;]<br>X[&#39;Credit Income percent&#39;] = X[&#39;Amt of credit&#39;] / X[&#39;Income&#39;]</pre><p><strong>B.4. Extracting only useful and relevant features: Feature selection</strong></p><p>Having redundant and useless features in our model reduces accuracy. Let us see how to do feature selection for any dataset with hundreds of variables. Feature selection will remove the problem of overfitting from your classification models.</p><p>Though there are many methods for feature selection or variable reduction, we shall depict only few very effective and popular ways for feature selection.</p><p><strong>B.4.1. If you need a specific number of features: </strong>SelectKBest class method.</p><p>It allows you to select top k most relevant features for your model(here we have taken k=5, so it will give us top 5 features).</p><pre>#apply SelectKBest to extract top 5 best features for housing df</pre><pre>bestfeatures = SelectKBest(score_func=chi2, k=5)<br>fit = bestfeatures.fit(X_train,y_train)<br>dfscores = pd.DataFrame(fit.scores_)<br>dfcolumns = pd.DataFrame(X_train.columns)<br>#concat two dataframes for better visualization <br>featureScores = pd.concat([dfcolumns,dfscores],axis=1)<br>featureScores.columns = [&#39;Features&#39;,&#39;Score&#39;]  #naming the dataframe columns<br>print(featureScores.nlargest(5,&#39;Score&#39;))  #print 5best features</pre><p><strong>B.4.2. If you just want the most relevant and uncorrelated features: Boruta plot</strong></p><p>In SelectKclass you need to specify that you want top k features. But sometimes, you do not know how many features you need. So you simply use Boruta plot. It will give you all the features which are important for your model. Also it will remove the highly correlated features, so that you will only have the most relevant features with you.</p><pre>from sklearn.ensemble import RandomForestClassifier<br>from boruta import BorutaPy</pre><pre># NOTE BorutaPy accepts numpy arrays only, if X_train and y_train #are pandas dataframes, then add .values attribute X_train.values in #that case<br>X_train = X_train.values<br>y_train = y_train.values</pre><pre># define random forest classifier, with utilising all cores and<br># sampling in proportion to y labels<br>from sklearn.ensemble import RandomForestClassifier<br>rf = RandomForestClassifier(n_jobs=-1, class_weight=&#39;balanced&#39;, max_depth=5)</pre><pre># define Boruta feature selection method<br>feat_selector = BorutaPy(rf, n_estimators=&#39;auto&#39;, verbose=2, random_state=1)</pre><pre># find all relevant features - 5 features should be selected<br>feat_selector.fit(X_train, y_train)</pre><pre># check selected features - first 5 features are selected<br>feat_selector.support_</pre><pre># check ranking of features<br>feat_selector.ranking_</pre><pre># call transform() on X to filter it down to selected features<br>X_filtered = feat_selector.transform(X_train)</pre><pre>#To get the new X_train now with selected features<br>X_train.columns[feat_selector.support_]</pre><p><a href="https://github.com/Shailaja-Gupta/Data-Preprocessing-for-medium/blob/master/Github%20MEDIUM%20boruta/Implementation%20of%20Boruta%20_%20Feature%20selection%20on%20MNIST%20dataset.ipynb">Shailaja-Gupta/Data-Preprocessing-for-medium</a></p><p>The full code for Boruta on MNIST dataset is mentioned above.</p><p><strong>B.5. Remove features with missing values above a particular threshold</strong></p><p>What if your column has 85% missing values? What should you do in that case? In that case it is better to drop the column. However sometimes business insists on keeping columns, then you can request for new data. What is the usually expected threshold above which we must not have missing values? Generally that threshold is expected at 25–30 %. Above this threshold, you should preferably drop the column, unless the customer insists on using it or if it is a very important parameter for analysis. In that case you can use imputer as depicted above.</p><pre>#Obtain the sum of missing values in all columns <br>df.isna().sum()</pre><pre>#Find the percentage of missing values in df:<br>result=df.isna().sum()/len(df)</pre><pre>#In df remove columns where missing values are more than a threshold<br>df = df.loc[:,result &gt; .8]</pre><p><strong>B.6. Splitting into Test and training data (</strong>MNIST Dataset<strong>)</strong></p><p>We will split the train_df into test and training set using train_test_split to test our model’s accuracy before making predictions on the test_df:</p><pre>X=train_df.drop(&#39;label&#39;, axis=1)<br>y=train_df[&#39;label&#39;]</pre><pre># implementing train-test-split<br>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train_full, y_test_full = train_test_split(X, y, test_size=0.20, random_state=66)</pre><pre>#Below part is only for MNIST dataset<br>#For digit 5, output label=1, for any other digit output label=0<br>y_train=((y_train_full==5).astype(int))  <br>y_test=((y_test_full==5).astype(int))</pre><p><strong>B.7. Checking for Class imbalance and tackling it</strong></p><p>Class imbalance means one class is dominating and there are very few instances of the other class. In MNIST dataset, there are only 3,060 instances where the digit is 5 out of around 40,000 total cases. That means our ML algorithm has exposure to very less training cases where the digit is 5.</p><pre>y_train.value_counts()<br>0    30540   #digit is not 5<br>1     3060   #digit is 5<br>Name: label, dtype: int64</pre><p>Class imbalance exists in our MNIST dataset as shown above.How to treat class imbalance?</p><ol><li>Increase the minority class instances(class 1 in this case, the digit is 5) by using oversampling</li><li>Reduce majority class instances(class 0 in this case, digit not 5) by using under-sampling.</li></ol><p>I personally prefer oversampling as it helps retain the variability of the data. <a href="https://arxiv.org/pdf/1106.1813.pdf">SMOTE algorithm</a>(Synthetic Minority Oversampling Technique) is a type of oversampling. At a high level, SMOTE:</p><ol><li>Randomly choose one of the k-nearest-neighbours and using it to create a similar, but randomly tweaked new observations.</li></ol><p>We are going to implement <a href="http://imbalanced-learn.org/en/stable/over_sampling.html#smote-variants">SMOTE in Python</a>.</p><pre>from imblearn.over_sampling import SMOTE<br>sm = SMOTE(random_state = 42)<br>X_train_oversampled, y_train_oversampled = sm.fit_sample(X_train, y_train)<br>X_train = pd.DataFrame(X_train, columns=X_train.columns)<br>y_train=y_train_oversampled</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*yhXGHgGa4d7-uVYsAibd4g.png" /></figure><pre>pd.Series(y_train).value_counts().sort_index() #values after SMOTE<br>0    30540<br>1    30540<br>dtype: int64</pre><p>After SMOTE implementation, both X_train and y_train have similar (in this case equal) number of values for both classes 0 and 1.</p><p><strong>B.8. Standard scaling of data</strong></p><pre>from sklearn.preprocessing import StandardScaler</pre><pre>sc = StandardScaler()<br>X_train = sc.fit_transform(X_train)<br>X_test=sc.transform(X_test)</pre><h3><strong>C. Choosing and Training a Binary Classifier</strong></h3><ol><li>Test all/many classifiers for classification on training data.</li><li>Select the one(s) that work best on training data and check their performance on test data</li><li>Try improvising those classifiers that work best by tuning their hyperparameters.</li></ol><p>Examples of classifiers which can be used for a classification project include Logistic Regression, K-Nearest Neighbors (KNN), SVM, Kernel SVM, Naive Bayes, Decision Tree classification, XGBoost and RandomForest classification to name a few. Treat these models as black-box models. You need not know the mathematics behind the models to implement them. Coding them is a simple plug and play according to the code given below.</p><p><strong>C.1. Testing the model on various classification ML algorithms.</strong></p><p>Let us train our model on Logistic Regression, K-Nearest Neighbors (KNN), SVM, Kernel SVM, Naive Bayes, Decision Tree classification, XGBoost and RandomForest classification one by one and test the performance on training and test data for each.</p><p><strong>C.1.1. Applying Logistic Regression on MNIST Dataset</strong></p><pre>from sklearn.linear_model import LogisticRegression<br>lr_clf = LogisticRegression(random_state = 0)<br>lr_clf.fit(X_train, y_train)</pre><p><strong>Logistic Regression performance on training and test data</strong></p><pre>print(&#39;Accuracy on training set:&#39;,lr_clf.score(X_train,y_train))<br>print(&#39;Accuracy on test set:&#39;,lr_clf.score(X_test,y_test))</pre><p>Accuracy on training set: 0.9622134905042568<br>Accuracy on test set: 0.9460714285714286</p><p><em>Please note that lr_clf.score(X_test, Y_test) is equivalent to your print(classificationreport(Y_test, Y_pred)). But you do not need to calculate Y_pred; that is done internally by the library when you implement .score method.</em></p><p><strong>C.1.2. Implementing Naive Bayes on MNIST Dataset</strong></p><pre>from sklearn.naive_bayes import GaussianNB<br>nb_clf= GaussianNB()<br>nb_clf.fit(X_train, y_train)</pre><p><strong>Naive Bayes performance on training and test data</strong></p><pre>print(&#39;Accuracy on training set:&#39;,nb_clf.score(X_train,y_train))<br>print(&#39;Accuracy on test set:&#39;,nb_clf.score(X_test,y_test))</pre><p>Accuracy on training set: 0.6189259986902423<br>Accuracy on test set: 0.31214285714285717</p><p><strong>C.1.3. Implementing RandomForest Regression on MNIST Dataset</strong></p><pre>from sklearn.ensemble import RandomForestRegressor<br>rf_clf = RandomForestRegressor(n_estimators = 10, random_state = 0)<br>rf_clf.fit(X_train, y_train)</pre><p><strong>Testing Random Forest Regression performance on training data</strong></p><pre>print(&#39;Accuracy on training set:&#39;,rf_clf.score(X_train,y_train))<br>print(&#39;Accuracy on test set:&#39;,rf_clf.score(X_test,y_test))</pre><p>Accuracy on training set: 0.9933483955468239<br>Accuracy on test set: 0.818514583915758</p><p><strong>C.1.4. Applying decision tree on MNIST dataset</strong></p><pre>from sklearn.tree import DecisionTreeClassifier<br>dec_clf = DecisionTreeClassifier()<br>dec_clf.fit(X_train, y_train)</pre><p><strong>Testing Decision Tree performance on training data</strong></p><pre>print(‘Accuracy on training set’,dec_clf.score(X_train, y_train))<br>print(&#39;Accuracy on test set:&#39;,dec_clf.score(X_test, y_test))</pre><p>Accuracy on training set 1.0<br>Accuracy on test set: 0.9645238095238096</p><p><strong>C.1.5. XGBoost:</strong></p><pre>&#39;&#39;&#39;Install xgboost following the instructions on this link:<br><a href="https://stackoverflow.com/questions/35139108/how-to-install-xgboost-in-anaconda-python-windows-platform">https://stackoverflow.com/questions/35139108/how-to-install-xgboost-in-anaconda-python-windows-platform</a>. &#39;&#39;&#39;</pre><pre>from xgboost import XGBClassifier<br>xgb_clf= XGBClassifier()<br>xgb_clf.fit(X_train, y_train)</pre><p><strong>XGBoost performance on training and test data</strong></p><pre>from sklearn.metrics import accuracy_score<br>print(&#39;Accuracy score on train_data: &#39;, accuracy_score(y_true = y_train, y_pred = xgb.predict(x_train).round()))<br>print(&#39;Accuracy score on test_data: &#39;, accuracy_score(y_true = y_test, y_pred = xgb.predict(x_test).round()))</pre><p>Accuracy score on train_data: 0.9498511904761905<br>Accuracy score on test_data: 0.9333333333333333</p><p>C.1.6. <strong>Trying ensembles</strong></p><p>Ensembles means that instead of using a single model, you take the the help of multiple models for prediction. For making each prediction, the ML ensemble algorithm takes every model’s output and suggestion into account and then makes a final prediction. If you take multiple models into account, then chances are that all of them make different errors. Hence when an ensemble takes multiple models’ predictions into consideration, one particular model’s error will have minimal effect on the overall result. Hence the overall prediction will be with minimal errors.</p><p><strong>C.1.6.1. Ensemble of SVM and Logistic Regression</strong></p><pre>from sklearn.svm import SVC<br>from sklearn.ensemble import VotingClassifier</pre><pre>voting_clf = VotingClassifier(<br> estimators=[(&#39;lr&#39;, lr_clf), (&#39;nb&#39;, nb_clf), (&#39;decision&#39;, decision_clf)],<br> voting=&#39;hard&#39;)<br>voting_clf.fit(X_train, y_train)</pre><p><strong>Performance of Ensemble method using SVM and Logistic Regression</strong></p><pre>voting_clf.score(X_train, y_train)<br>voting_clf.score(X_test, y_test)</pre><p>Accuracy on training set 0.9787164374590701<br>Accuracy on test set: 0.9401190476190476</p><h4><strong>C.1.6.2. </strong>Ensemble method using Logistic regression &amp; Decision trees</h4><pre>voting_clf2 = VotingClassifier(<br> estimators=[(&#39;lr&#39;, lr_clf),(&#39;decision&#39;, decision_clf)],<br> voting=&#39;hard&#39;)<br>voting_clf2.fit(X_train, y_train)</pre><p><strong>Performance of Ensemble method (Logistic regression &amp; decision trees)</strong></p><pre>voting_clf2.score(X_test, y_test)<br>voting_clf2.score(X_train, y_train)</pre><p>Accuracy on training set 0.9777380952380952<br>Accuracy on test set: 0.983513425016372</p><p><strong>D. Classification Models that don’t work too well on large datasets</strong></p><p>I am mentioning the below classification models since they are very useful for many classification problems that you will come across. They take too much time for implementation on MNIST dataset mainly due to the large size of MNIST. The performance of these algos below is too slow when the data is large, and hence it is not suggested to use them for MNIST.</p><ol><li>k-NN</li></ol><p>k-NN works well with a small number of input variables/features/columns, but struggles when the number of inputs is very large. So in case the input variables are too many (784 here in MNIST), k-NN will not perform as well as the other algorithms.</p><p>2. Kernel SVM: The Kernel Matrix KK is a n×n matrix where, n is number of instances in the data. So for MNIST dataset, after the oversampling we have 60,000x784 instances ~ 470.4k, the kernel matrix KK will have (60,000x784 instances)² ~ 2.2 x 10¹⁵ elements, and may take ~100G memories.</p><p>Basically, storing the kernel matrix requires memory that scales quadratically with the number of data points. Training time for traditional SVM algorithms also scales superlinearly with the number of data points. So, these algorithms aren’t feasible for large data sets.</p><p>Please find below how to implement these algorithms for other classification problems:</p><p><strong>1.1. Applying k-NN on any classification dataset</strong></p><pre>from sklearn.neighbors import KNeighborsClassifier<br>knn_clf = KNeighborsClassifier(n_neighbors = 5, metric = &#39;minkowski&#39;, p = 2)  <br>#You can change these hyperparameters like metric etc.<br>knn_clf.fit(X_train, y_train)</pre><p><strong>k-NN performance on training and test data of MNIST </strong>(time taken for output was close to 5 hours)</p><pre>print(&#39;Accuracy on training set:&#39;,knn_clf.score(X_train,y_train))<br>print(&#39;Accuracy on test set:&#39;,knn_clf.score(X_test,y_test))</pre><p>Accuracy on training set 0.9742857142857143<br>Accuracy on test set: 0.9907170923379175</p><p><strong>1.2. Kernel SVM on MNIST Dataset</strong></p><pre>from sklearn.svm import SVC<br>kernel_svm_clf = SVC(kernel = &#39;rbf&#39;, random_state = 0)<br>kernel_svm_clf.fit(X_train, y_train)</pre><p><strong>Kernel SVM performance on training and test data </strong>(time taken for output was close to 5 hours)</p><pre>print(&#39;Accuracy on training set:&#39;,kernel_svm_clf.score(X_train,y_train))<br>print(&#39;Accuracy on test set:&#39;,kernel_svm_clf.score(X_test,y_test))</pre><p>Accuracy on training set 1.0<br>Accuracy on test set: 0.9125</p><h3><strong>D. Finalizing a classification model based on performance</strong></h3><p>While finalising a model, we need to keep both the accuracy and time taken in mind. While k-NN gives better accuracy than decision trees and random forest, it takes ages to compute. Despite working on Google colab with a GPU, I had to wait for 5 hours before getting output from both SVM and k-NN, hence we will not be using those methods for this dataset. Both k-NN and SVM are well suited for smaller datasets. We will rather choose models that do almost as well as SVM and k-NN and in fraction of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pWeT5HmJlKxbifqQ0jfNUQ.png" /></figure><p>Based on the performances both time wise and accuracy wise, we shall go for ensemble of Logistic regression and decision trees. We chose to create the ensemble of LR and decision trees since both these models were individually doing the best. You can experiment with more ensembles too and check the accuracy of the models.</p><p>There are other parameters too such tuning the hyperparameters (hyperparameters are the parameters you input in each model, for e.g. in k-NN hyperparameters are specifying the k, for random forest, hyperparameters are specifying the number of trees etc.) inside each model which we could have done, but I will cover that in some other article.</p><p>If you are wondering about the pros and cons of various machine learning algorithms, and when to use which ML algorithm, their various applications, please check my article below.</p><p><a href="https://medium.com/@shailajagupta/pros-and-cons-of-various-classification-ml-algorithms-3b5bfb3c87d6">Pros and cons of various Classification ML algorithms</a></p><p>Happy reading :). Hope this helped :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d0becc6a295" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-tackle-any-classification-problem-end-to-end-choose-the-right-classification-ml-algorithm-4d0becc6a295">How to tackle any classification problem end to end &amp; choose the right classification ML algorithm.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science for Beginners]]></title>
            <link>https://medium.com/data-science/data-science-for-beginners-66e6c51ce066?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/66e6c51ce066</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[visualisation]]></category>
            <category><![CDATA[machine-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Tue, 28 Jan 2020 10:38:17 GMT</pubDate>
            <atom:updated>2020-02-14T14:44:33.431Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntdyE4PiFxIc34ikHBMkPg.jpeg" /><figcaption>Source: <a href="https://pixabay.com/photos/web-network-technology-developer-3963945/">Pixabay</a></figcaption></figure><p><strong>What all do we need for entering Data Science?</strong></p><ul><li>Query language like SQL</li><li>Programming language like R/Python</li><li>Visualisation tool like PowerBI/Qliksense/Qlikview/Tableau etc.</li><li>Basic Statistics for Machine Learning</li><li>Machine learning algorithms (make sure you try out use cases in the domain where you wish to expertise in, sales, finance, HR, ops etc. Use cases will be different for all)</li><li>Practise and implementation</li></ul><p><strong>a) Query Language</strong></p><p><strong>Types of query languages you can learn:</strong> SQL is hands down the best in the market and it is not going anywhere.</p><p>One more query language you can learn is elasticsearch. It is very much in use nowadays. I learnt it via a Udemy course. The difference between SQL and elasticsearch is that in elasticsearch, you have a dataframe where not every row has the same column values. For eg. a database car. Let’s say for some cars we have only model name, price and color info. For some, we may have color, model, price, number of models till date, parent company etc., for another type of car we may have info on only model and name etc. In SQL this is captured by putting NAN values where fields are missing. In elasticsearch, there is no concept of NAN.</p><p><strong>Sources of learning:</strong> You can learn from W3Schools/Tutorialspoint or anywhere actually since it will hardly take one week to learn SQL. If you don&#39;t have access to databases to practise, you can load any csv file as a database and practise.</p><p>Elasticsearch /Kibana can be learnt via the same websites. Also, you can take a Udemy course on the same since this is harder to learn compared to SQL.</p><p><strong>Environment: </strong>In case you are learning SQL, you can install MySQL/PostGRESQL and start the practice. For elasticsearch you can install Kibana and practice.</p><p><strong>Why is Query language important for Data Science?</strong></p><p>We have huge datasets in Data science that have millions of rows and millions of columns. For analysis, you don&#39;t need all the data. We need to extract the relevant data using a query language and then proceed with the analysis.</p><p><strong>Big Data:</strong> You can also learn big data languages or techniques such as Scala and Hadoop/MapReduce etc. However, that is good to have for most D.S. jobs and not a must-have. It is more like an icing on the cake. Big data is a part of Data engineering and usually involves coding mostly unlike Data science which is a blend of statistics, maths, coding and domain knowledge. Big data should be learnt after you are thorough with Data Science.</p><p><strong>b) Programming Language</strong></p><p><strong>Languages in vogue: </strong>There are 2 main languages: R and python. R is a language designed by statisticians and mathematicians. Python is a language of coders. Both are good. However, nowadays Python is more in vogue due to its large support eco-system from the programming world, better scalability and better integration with APIs and other codes for the complete product.</p><p><strong>Learning Source:</strong> I learnt Python via Coursera. The course name was “Python for data science.” You can also learn via tutorialspoint and W3schools. More than the course, I learnt via major and minor projects that we were supposed to do as part of our certificate.</p><p>I learnt R via Udemy. Advanced Course on R for data science. It is good to learn one language in-depth and have an overview of another language along with it. That is because in a team you will be working with many data scientists. Some will be comfortable with R, and some with python.</p><p><strong>Environment</strong>: Anaconda Navigator, Pycharm, Spyder or Jupyter notebooks.</p><p>An environment is a place where you code basically and implement your codes.</p><p>My personal favorite environment is Jupyter notebooks. Jupyter notebooks let you type in R/Python/HTML etc., so you can use the best libraries or techniques of a particular language and use it in Jupyter. Also if two people are using different languages, then you can easily collaborate using Jupyter notebooks.</p><p><strong>Queries resolution: </strong>Stackoverflow and Stackexchange are great places to ask queries about language issues. People usually reply within 5 minutes to 24 hours.</p><p><strong>c) Visualisation tools</strong></p><p><strong>Tools you can learn: </strong>Tableau/PowerBI/Qliksense/Qlikview are the most common tools. PowerBI and Tableau are most widely used.</p><p>PowerBI is almost like an advanced version of excel and very easy to learn. The only issue with excel is that it crashes and becomes slow with huge data. Also not too many options are available for amazing visualisations. All this is possible in PowerBI. Another good thing is that PowerBI is free. For Tableau, you can download a trial version of 30 days but in case of PowerBI, you can use it as long as you want (most of the features are available in the free version of PowerBI). PowerBI/Qliksense/Qlikview are not supported on Apple though since they are windows based products.</p><p><strong>Sources of learning:</strong> I have learnt PowerBI through Udemy. I selected the course based on ratings. Any course with a rating above 4.3 is cool. I learnt Tableau via the Tableau website.</p><p><strong>d) Statistics for Data Science:</strong></p><p><strong>Basic terms you should know: </strong>Standard deviation, mean, median, mode, skewness, hypothesis testing, central limit theorem, population versus sample, z score, confidence interval, p-value, statistical significance, critical value, proportion testing, two-tailed, one-tailed, Pareto principle, chi-square test, z test, t-test, normal distribution, Gaussian distribution etc.</p><p><strong>Sources of learning:</strong> Udemy course on Statistics for Data science by Kirill Eremko, Introduction to Statistical Learning in R or any basic book of stats.</p><p><strong>e) Machine learning algorithms:</strong></p><p><strong>Types of Algos mostly used:</strong> XGBoost, RandomForest, Deep Learning, Neural networks, Time series, Decision Trees, Clustering and classification algos.</p><p><strong>Sources of learning:</strong> Kirill Eremko course on Udemy for machine learning is amazing.<br>Otxts book is great for visualisation algos of ML. It is a free source and one of the best resources available online. <a href="https://l.facebook.com/l.php?u=https%3A%2F%2Fotexts.com%2Ffpp2%2Fgraphics-exercises.html%3Ffbclid%3DIwAR1iFo9tBeCa40HVy1kgplzZIITQZVmB46hEUcPrtfz4stXUYGLAA-fpmeA&amp;h=AT0293daVzIlZpMdgoec5WlQyxl5eP3cDKq3iQOLyJxcrR06-2SAG9wBjvF9lPKMYSpjN21uEsY0VyfshjuSGOHXsQH1MmiYJ__JmIZkOsfrX8dWMWKEZSOLp5YA460ZC2kMr69vixBrFlE_jr_M_A">https://otexts.com/fpp2/graphics-exercises.html</a></p><p>Introduction to statistical learning in R is good for basics of stats and applications in ML</p><p><strong>How to decide which algos to implement:</strong></p><p>First, know the uses case of your problem. What do you want to do? Is your output a continuous variable? (taking values 1,3,10 etc. basically any value) or is it binary kind of decision or do you want to club people and make some decisions for a bunch of folks?</p><p>For eg. for one project, I had to work on credit risk analysis in finance. You have to decide whether a person will be able to pay back a loan or not. So it is a binary decision. In this case, you need a classification algo because you want to either classify a person as a defaulter or a non-defaulter. So I use Logistic Regression (used when the output is binary yes/no etc), XGBoost or Decision trees. You can basically use any classification algo. Other classification algos which can be used here are SVN, Naive Bayes etc. (you can google classification algos for the complete list).</p><p>If in a project, I need to predict the value of a stock or house, then it can take any value starting from Rs.10 to Rs.1,000,000 or more, etc. (a continuous variable). Here I will use regression. Regression can be of many types, simple linear, multiple linear, polynomial, SVR etc. The right kind of regression technique can be found out by checking the r squared error (how far the predicted values lie from the actual).</p><p>Similarly, if I need to find out the target segments, I will need to use clustering algo. K means and hierarchical clustering are examples of clustering algos.</p><p>So first learn about the algorithms (at least one classification algo, one clustering algo, one regression algo to begin) and then see what algo you need for the problem at hand and then start the analysis.</p><p><strong>What all types of projects and algos should I learn:</strong></p><p>First, choose your domain. My domain, for instance, was Finance and marketing. So I can talk about those here. Though the list is endless, I will just write the most important algos here I learnt</p><p><strong>Marketing</strong>: Time series modeling (ARIMA modeling mainly) to predict future sales, volume and value. Time series modeling means that you have data for one thing, let&#39;s say sales data for a company for a period of time (could be months, years, days etc) and you predict sales for the company for the coming years/months/days.</p><p>Clustering algos to cluster target groups and design separate categories for each group.</p><p>Churn modeling to decide how many of the field staff will stay and how many will leave to manage workflow.</p><p><strong>Finance:</strong> Classification algos like Logistic Regression (Logistic Regression and linear or multiple regression are completely different, only the name is similar), SVN, Naive Bayes, XGBoost etc. for Credit Risk analysis to find out who will default.</p><p>Regression for asset valuation (value of stock, asset for mortgage etc.)</p><p><strong>f) Implementation</strong></p><p>How to implement your knowledge:</p><ul><li>Projects in courses of Coursera, Udemy etc</li><li>Hackathons of Analytics Vidhya, Kaggle etc to see where you stand among the crowds.</li><li>Live projects</li><li>In the Company and on the job</li></ul><p>Hope this helps!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/24/0*yV-w7O0gsMk-FMDt.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66e6c51ce066" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-science-for-beginners-66e6c51ce066">Data Science for Beginners</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microfinance Companies: The new channel for Consumer Durables]]></title>
            <link>https://medium.com/@shailajagupta/microfinance-companies-the-new-channel-for-consumer-durables-8c7e7953ff3a?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c7e7953ff3a</guid>
            <category><![CDATA[microfinance]]></category>
            <category><![CDATA[rural-consumers]]></category>
            <category><![CDATA[fmcd-in-rural-india]]></category>
            <category><![CDATA[fmcd]]></category>
            <category><![CDATA[rural-marketing]]></category>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Fri, 09 Jun 2017 13:17:09 GMT</pubDate>
            <atom:updated>2017-06-09T13:17:36.973Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*kTjhq5IClRAWcEh4.jpg" /></figure><p>As the urban market stands saturated, all the Companies are moving towards rural. Though FMCG had long back ventured into the rural arena by coming up with sachets and small packaging of everything, FMCD (Fast Moving Consumer Durables — Products with very long shelf life such as Fans, iron, Mixer etc.) took little longer. The reason being that FMCG products are priced less and are easily affordable. However FMCD goods are not so easily affordable. To tackle the same, FMCD companies came up with EMIs. However even with EMIs the problem hasn’t completely been solved. The reason is that it is cumbersome to collect EMIs from these borrowers. What if they default? Also the retailers cannot keep track of the EMIs all the time. Moreover, FMCG companies have wide distribution channels even in the rural areas, which is not that true for FMCD companies.</p><h3><strong>Reaching out to the poor</strong></h3><p>So how do we reach out to the poor? Welcome to the world of microfinance. The microfinance companies give loans to people mostly in the rural areas, and each branch covers 2500 borrowers on an average. Any good microfinance company will have close to 300–600 branches pan India. You can easily reach close to 5 million people via just one company. Right now, all microfinance companies do not offer products. I work for Spandana Sphoorthy, a leading microfinance company and we deal with some vendors in the FMCD Sector. The daily work of any microfinance company includes Loan collection and loan disbursement. During loan collection, the field agents collect the money from houses. Each of these house has on an average 10–20 borrowers who are sitting to repay their bi — weekly, weekly or monthly installment. During these house visits, the field agents show the catalogue of various FMCD products to the borrowers. He takes down the demand of the products and next time delivers the same to the recipients. That’s however just one way of reaching out to these borrowers.</p><p>During loan disbursement, the branch is filled with people. On an average there are 10–20 borrowers per day in a branch. At this time, products like Pressure cooker, fans, chairs etc. can be shown to them. Each day results in at least 3–4 sales per branch. Some days the sales can go upto 20 as well. The field staff earns very humble salaries. Hence every penny counts for them. If incentivized properly, they are more than willing to sell these products. These two touch points can easily mean increased sales for the companies.</p><p>FMCD is a good idea in this case because of their longer shelf life and their higher ticket price. Since the sell cycles in Microfinance typically take longer, the perishable FMCG products may not always work out. The higher price means that installments look attractive to the borrowers who may not always be able to afford these products on a down payment. During my stint with the microfinance industry, I found the huge demand for Mobile phones, Refrigerators, sewing machines, washing machines etc. among the borrowers. They repay the installments for these products along with the regular installments.</p><h3>Importance of service</h3><p>The only caveat is the servicing part. The companies need to ensure that there is proper servicing for these products in case of any damage etc. Otherwise the damage for the FMCD company can be huge, it may lose its future potential customers as well, because word of mouth works a lot in villages.</p><h3>Conclusion</h3><p>Overall it’s a very lucrative proposition for any FMCD company looking to venture into the Rural market. They can easily use it as a separate channel for their sales. They should approach good microfinance companies, and send the right kind of products according to the insights from the microfinance companies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c7e7953ff3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fortune lies at the bottom of the pyramid: Spandana, a company which is changing lives]]></title>
            <link>https://medium.com/@shailajagupta/fortune-lies-at-the-bottom-of-the-pyramid-spandana-a-company-which-is-changing-lives-7ac88ad3775f?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ac88ad3775f</guid>
            <category><![CDATA[bottom-of-the-pyramid]]></category>
            <category><![CDATA[spandana-sphoorthy]]></category>
            <category><![CDATA[microfinance]]></category>
            <category><![CDATA[shaping-india]]></category>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Tue, 09 May 2017 05:43:33 GMT</pubDate>
            <atom:updated>2017-05-09T05:43:33.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*d4t5S2IVW0Tu6yPUCfYvuA.jpeg" /></figure><p>As CK Prahlad mentions, “<strong>If we stop thinking of the poor as victims or as a burden and start recognising them as resilient and creative entrepreneurs and value conscious consumers, a whole new world of opportunity will open up.”</strong></p><p>This is the same principle which Spandana Sphoorthy aims to abide by. It is a company which is committed to make a change in the way the poor are perceived. They are not perceived as a burden, but rather as co-creators who if given a chance could change and reshape the face of the Indian society.</p><p>Razia, a lady from the Parda village, Madhya Pradesh, is an old borrower of Spandana Sphoorthy. She has been with us since the past 6 years. Her family is in the traditional business of TV and electronics repair etc. “Madam, the money that we earned was not enough to meet the daily expenses”, she speaks in Malwi, a language I am yet to catch up with properly. My collection officer translates what she says for me. “We wanted to find an alternate source of income. We thought of buying this atta chakki since the villagers had to go to a far off village to get masalas and atta grinded.”.</p><p>“How much did the chakki cost you?”</p><p>“It cost me Rs. 75000. I paid in installments for the same. Rs.40,000 is still remaining.”</p><p>“How much extra do you earn from this machine?”</p><p>“I earn around 5–10,000 per month extra. However since we already have an atta chakki at our home, no one else in village will buy it. People from the adjoining village also come here. There is no competition”</p><p>“You can actually grind atta from the wheat and sell packaged wheat to mandi”, I gave this option so that she can expand the horizon of her thinking.</p><p>“Madam who has the resources to do that?”</p><p>True, I thought. May be I should get her connected to an NGO which does the same. But are there any NGOs in Neemuch? My upbringing in Delhi had made me forget that rural places like Neemuch can ill afford the luxuries of NGOs. NGOs also usually crop up in cities where the rich people can afford not to earn much to help the poor. It is difficult to expect people to help others when they themselves are struggling to make ends meet.</p><p>I was happy as of now with the little help I could do to this lady by giving her the loan. When banks aboard are crying foul over bad debts and NPAs, and the need to check the credit worthiness of the borrowers, borrowers like these are being given loan purely because of the trust that the company bestows in them. We don’t ask for mortgage of any kind. These loans are purely unsecured. One thing I realised in this one month of my training with Spandana, the poor usually don’t default. They don’t run away like Vijay Malaya with Rs.9000 crores. They usually pay back. If we trust them, they usually return the favour by being grateful and by paying us back the money.</p><p>Razia will be able to break even in around 15 months, even if she earns only Rs.5000 extra per month. “I will be able to save something for my daughter madam.” The real India is poor, really poor. It is not found in the AC comforts of our office or in the luxuries of a metro. It is found in states like Madhya Pradesh, Orissa, Chhatisgarh to name a few. However these people don’t have a victim mentality. They never question, “Lord , why me?”. They are determined to take life head on and find a way. Somewhere in the expectation mapping, I was expected to make a change in their lives. However I think that it was not a one way change. They changed me for the good as well. Social businesses are the need of the hour. They can help companies make money and tap the fortune at the bottom of the pyramid. These people are not technologically obsolete. I was shocked to see the widespread usage of the Bhim app. Is this not an opportunity for all the companies to expand their footprint? I invite you all to tap the bottom of the pyramid. Believe me, you wouldn’t be disappointed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ac88ad3775f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spandana Sphoorthy — Empowering women]]></title>
            <link>https://medium.com/@shailajagupta/spandana-sphoorthy-empowering-women-2532954be97b?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2532954be97b</guid>
            <category><![CDATA[helping-poor]]></category>
            <category><![CDATA[women-empowerement]]></category>
            <category><![CDATA[helping-society]]></category>
            <category><![CDATA[microfinance]]></category>
            <category><![CDATA[spandana-sphoorthy]]></category>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Tue, 09 May 2017 04:24:42 GMT</pubDate>
            <atom:updated>2017-05-09T08:08:39.390Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDIUgBUoQI4UCgUP_LGwPg.jpeg" /><figcaption>Sabina with her sewing machine</figcaption></figure><p>I am born and brought up in Delhi and was blessed to have a comfortable upbringing. I came to terms with the realities of rural India in the true sense after my stint with Spandana Sphoorthy. As a part of my training, I was sent to Madhya Pradesh where I interacted with the poor on a daily basis. The daily travel provided me a sneak peek into the lives of scores of women who are currently being helped by the company. Though Delhi is also home to many poor, Delhi poor are much better off due to the plethora of opportunities available to them.</p><p>To put things into perspective, we are a micro finance company which provides unsecured loans to women. Wait, did I say only women? Yes, the reason we only give loans to women is that they are generally more sincere and have a sense of responsibility. “Didi, I have been able to earn 1,20,000 additional income with the sewing machine I purchased”, said Sabina, with a beaming smile. She is one of the hundreds of women whose lives have been touched by our company. “Should I show you the machine?”, she asked with a child like excitement. “Oh sure! I would love to see it”.</p><p>We went to her house in a narrow lane. She proudly showed me the machine. We are doing a good job, I thought in my head. “Have you been associated with us before Sabina?”, “Yes didi, this is my second loan. This time company gave me a loan of Rs.30,000”. “Awesome”, was my reply.</p><p>Seeing me in the house, several other ladies surrounded me and asked, “Didi, we also need a loan! Please help us!”. One of the ladies told me something I will never forget, “Didi I have two daughters. The elder one goes to teach in a school and earns Rs. 6000 per month by teaching the students of class 10th and 12th. She is B.Sc. and M.Sc. She wants to do tailoring in part time so that she can fund her marriage. Can I get a loan?” I didn’t know what to say. I was wondering about how low the salaries can be in rural schools. I had never seen this kind of India. I remembered the times I hadn’t thanked god for all my blessings. A post graduation from IIM Indore had made sure that I have friends who earned decently well. Moreover even in Delhi, I didn’t really have the chance to interact with people as educated as M.Sc. earning as low as Rs.6000 per month. She deserves better, I thought. I was determined to help her and told her, “Sure. Form a group of 8–10 ladies and we shall process your loan”.</p><p>If you are confused as to why I told those women to form a group is because of the way microfinance works. We give unsecured loans, but we rarely give individual loans. We give loans to a group of women. The people in the group are responsible for each other. If one person in a group is unable to pay, then others pay for him/her. If a borrower is delinquent, there are less chances that she will be able to join any group from next time. So people usually pay. The testimony to that is the fact that the repayment rate for our company is around 99.9%, which by any standards is exemplary.</p><p>The way Spandana has been helping women come out of poverty is amazing. I hope it keeps touching lives the way it has been doing. I feel blessed to be a part of it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2532954be97b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spandana Sphoorthy — Changing lives; One loan at a time]]></title>
            <link>https://medium.com/@shailajagupta/spandana-spoorthy-changing-lives-one-loan-at-a-time-80ed1f805e8f?source=rss-673d053b01dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/80ed1f805e8f</guid>
            <category><![CDATA[spandana-sphoorthy]]></category>
            <category><![CDATA[women-empowerment]]></category>
            <category><![CDATA[india]]></category>
            <category><![CDATA[helping-poor]]></category>
            <category><![CDATA[microfinance]]></category>
            <dc:creator><![CDATA[Shailaja Gupta]]></dc:creator>
            <pubDate>Mon, 08 May 2017 17:26:10 GMT</pubDate>
            <atom:updated>2017-05-09T02:33:42.814Z</atom:updated>
            <content:encoded><![CDATA[<p>Shabnam, a lady in Neemuch runs a boutique. She bought fabric and material from the initial loan of Rs.20,000 we had given to her. Using this money she expanded her business and was able to earn upto Rs.100,000 per annum.</p><p>“Madam, loan has helped me to be independent. Earlier I was dependent on my husband for everything. But now I am self — dependent and happier. I want the same for my children. I will make them independent. Your company has helped me a lot and I am very thankful to you”, she says in half Hindi and half Malwi.</p><p>Then she took me to show the fabric she had purchased from the money. “Madam, I stitch clothes from these fabrics and ladies come over to buy the final product. Do you want to buy some stuff?”, she asked with a smiling face. I couldn’t help her by buying anything, however I had solace in the fact that I was able to help her by giving her the loan. And not just her, but millions of such women with hopes in their eyes and dreams in their heads.</p><p>Before joining this company, I did not know much about the micro finance industry. However after my rural stint I came to know the amazing work that my company does. I am blessed to work with such a company which is creating such a deep impact in people’s lives.</p><p>It gives me mental satisfaction and peace of mind to work for such a company which helps shape futures. The India we dream of can only happen if the bottom of the pyramid grows. Giving unsecured loans to such people by believing in them is a very noble deed. I wish this pie that we wish to share with them only grows with time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80ed1f805e8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>