<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manu Sharma on Medium]]></title>
        <description><![CDATA[Stories by Manu Sharma on Medium]]></description>
        <link>https://medium.com/@manusharma_66718?source=rss-cc90427ba155------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Oe0RkFMGJYcavsJx.</url>
            <title>Stories by Manu Sharma on Medium</title>
            <link>https://medium.com/@manusharma_66718?source=rss-cc90427ba155------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:48:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manusharma_66718/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Transformation & Scaling of Numeric Features: Intuition]]></title>
            <link>https://medium.com/data-science/transformation-scaling-of-numeric-features-intuition-7f4436e8e074?source=rss-cc90427ba155------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f4436e8e074</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Manu Sharma]]></dc:creator>
            <pubDate>Sun, 22 Mar 2020 10:25:31 GMT</pubDate>
            <atom:updated>2022-11-14T03:35:57.366Z</atom:updated>
            <content:encoded><![CDATA[<h3>Explained: ML Transformation &amp; Scaling</h3><h4>When to use Transformation or Scaling? Which scaler? Which algorithms?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xvP6JTJiR1wcuKvC" /><figcaption>All Different Entities should be on the same scale before modelling; Photo by <a href="https://unsplash.com/@paucasals?utm_source=medium&amp;utm_medium=referral">Pau Casals</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Scaling, Standardizing and Transformation are important steps of numeric feature engineering and they are being used to treat skewed features and rescale them for modelling. Machine Learning &amp; Deep Learning algorithms are highly dependent on the input data quality. If Data quality is not good, even high-performance algorithms are of no use. It’s as simple as <em>Garbage In : Garbage Out.</em></p><p>Data Scientists have so many preprocessing steps to test &amp; validate before the modelling. And that further adds the confusion around the effect and use of transformation and scaling. Here we will try to answer some of the questions while using a public dataset</p><blockquote>When to use Transformation or Scaling? Which scaler to use? Which algorithms benefit?</blockquote><p>Data used for this exercise is the <em>main datafile </em>from <em>Home Credit Default Risk.</em> Data is collated around to predict clients’ repayment abilities. Data is for all the loan applications, one row represents one loan application in our data sample.</p><p>DataSet can be downloaded from — <a href="https://www.kaggle.com/c/home-credit-default-risk/data">Data</a>. And Jupyter Notebook can be downloaded from — <a href="https://github.com/manu007sharma/Medium_Notebooks/tree/master/Scale_Transform">Notebook</a>.</p><p>Dataset is published for explaining the capability of each applicant’s repaying a loan? Below is the distribution of the Target feature and some of the independent features. Target feature has imbalanced data problem, the positive class only has 8% ratio of the full data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XpcSpdbu6kafaaDbHduRaA.png" /><figcaption>Target Feature: Loan Default</figcaption></figure><p>Below are some of the important numeric independent features and their histogram. They are picked just for the exercise explanation. They all are looking in different ranges and scales e.g. <em>AMT_ANNIUITY</em> is in <em>mn</em> range but <em>OWN_CAR_AGE</em> can max go till 90.</p><p><em>AMT_ANNUITY; AMT_CREDIT; AMT_GOODS_PRICE; AMT_INCOME_TOTAL; DAYS_BIRTH; OWN_CAR_AGE;</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_xc9DEhMBieNA4yYx3sYAg.png" /><figcaption>Histogram of numeric features: Original Data</figcaption></figure><p>Below is the original data’s combined boxplot, it is looking highly skewed because of the different scale and skewed features in one space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLpFeMDYcJu4gl-wnNuZRg.png" /><figcaption>Combined Box Plot: Original Data</figcaption></figure><h3>Transformation</h3><p>Normally distributed features are an assumption in Statistical algorithms. Deep learning &amp; Regression-type algorithms also benefit from normally distributed data.</p><p>Transformation is required to treat the skewed features and make them normally distributed. Right skewed features can be transformed to normality with <em>Square Root/ Cube Root/ </em><strong><em>Logarithm transformation.</em></strong></p><p>As per the above histograms<em> AMT_ANNUITY, AMT_CREDIT, AMT_GOODS_PRICE, AMT_INCOME_TOTAL, &amp; OWN_CAR_AGE </em>are skewed numeric features and <em>DAYS_BIRTH</em> are normally distributed.</p><p><strong>Skewness can be because of one of the two reasons</strong></p><ul><li><em>Present of extreme abnormal outliers, which may not be important to us.</em></li><li><em>Or Feature’s natural distribution is skewed, and the tail is important to us. This is a case in most real-life cases</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*iBUxUz_nSRToPAeVRcFtcg.png" /></figure><p><strong>Introduction of log transformation:</strong> As the left graph exhibits, the output of the Log function for positive values increases very slowly. So higher values are marginalized more as compared to the lower observations.</p><p><strong>Effects of Transformation: </strong>Skewed Numeric feature may get normally distributed after log transformation. For example, in the below graph <em>AMT_CREDIT</em> is normally distributed after the log transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qhkEebKjui3J_BlQf3P45A.png" /><figcaption>Before and After Log Transformation: AMT_CREDIT</figcaption></figure><ul><li><strong>Effect of log transformation on skewed target feature(case of regression):</strong> <em>log transformation may treat the skewed feature to normality. And, if our target feature is normally distributed, the algorithm will give equal importance to all the samples. Its also called homoscedasticity. It’s equivalent to treating the imbalanced data problem in categorical target feature like we have in our given dataset. So it’s good to have a normally distributed target feature.</em></li><li><strong>Effect of log transformation on the skewed independent feature:</strong> <em>log transformation may bring the independent feature to normality like above where AMT_CREDIT is nearly normal distributed after log. But it may not improve the relationship between the target and the independent features. So, treating independent skewed features may or may not benefit modelling accuracy, it all depends on the original causal relationship between the two.</em></li></ul><h3>Scaling</h3><p>Scaling is required to rescale the data and it’s used when we want features to be compared on the same scale for our algorithm. And, when all features are in the same scale, it also helps algorithms to understand the relative relationship better.</p><blockquote>If dependent features are transformed to normality, Scaling should be applied after transformation.</blockquote><p><strong>Which Algorithms may benefit after Scaling</strong>? Scaling is helpful in Distance-based algorithms and also in faster convergence</p><p><em>Linear &amp; Logistic Regression, KMeans/ KNN, </em>Neural Networks, <em>PCA</em> will benefit from scaling</p><p><strong>Which Algorithms may not benefit after Scaling?</strong> Some algorithms are independent of Scaling. Entropy &amp; Information Gain based techniques are not sensitive to monotonic transformation.</p><p><em>Tree-Based Algorithms, Decision Tree, Random Forest, Boosted Trees(GBM, light GBM, xgboost)</em> may not benefit from scaling.</p><p>During Scaling/ Standardizing/ Normalizing, we will follow sklearn vocabulary, so it’s a good choice to use general word S<em>caling</em> instead of <em>Standardizing</em> or <em>Normalizing</em></p><blockquote>Scaler model fitted on the train data will be used to transform the test set. Never fit scaler again on the test data</blockquote><p>Sklearn has following four scalers primarily</p><p>1. Minmax scaler<br>2. Robust scaler<br>3. Standard Scaler<br>4. Normalizer.</p><p><strong>Minmax scaler</strong> should be the first choice for scaling. For each feature, each value is subtracted by the minimum value of the respective feature and then divide by the range of original maximum and minimum of the same feature. It has a default range between [0,1].</p><p>Below is the histogram of all 6 feature after the Minmax scaling. We haven’t log-transformed any of the features before scaling. MinMaxScaler hasn&#39;t changed the internal distribution of the feature and also brought everyone on the same scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDMWAKBUG9TXx3oMG44y1Q.png" /><figcaption>Histogram after MinMax Scaling</figcaption></figure><p>Below is the combined box plot of all 6 features after scaling. And, all are in the range of [0,1 ]. The internal space between each feature’s values has been maintained and their relative distribution is also looking better as compared to the original data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lf6FCiM9WCWo5N-2u90mdg.png" /><figcaption>Boxplot after minmax scaler</figcaption></figure><p><strong>RobustScaler </strong>can be used when data has high outliers and we want to subside their effects. But unimportant outliers should be removed in the first place. RobustScaler subtracts the column’s median and divides by the interquartile range.</p><p>Following graph is a histogram of features after the Robust Scaler. Though the histogram looks similar to the original data distribution, their respective internal distance space is not maintained like the original data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1E0ZPPo3CkrdzINqKTZTQg.png" /><figcaption>Histogram after Robust Scaler</figcaption></figure><p>Also, as seen in the below box plot, now the range is not in [0,1]. And also the relative spaces between each feature’s values are distorted and not same now. Using robustscaler, in this case, will pass wrong information to the modelling process about your underlying data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69rDwfcikoHvoLiHdYioGw.png" /><figcaption>Boxplot after RobustScaler</figcaption></figure><p><strong>StandardScaler </strong>rescales each column to have 0 mean and 1 Standard Deviation. It standardizes a feature by subtracting the mean and dividing by the standard deviation. If the original distribution is not normally distributed, it may distort the relative space among the features.</p><p>Below is the histogram of features after applying standard scaler, Distribution is looking similar to the original data distribution, but they aren’t same, their respective internal distance of observations are changed during standard scaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltIGnVrKaIWqyfbyTAU1Iw.png" /></figure><p>Following is a combined boxplot of features after the standard scaling, As expected, it distorts the relative distances between the feature values, where after min-max they were looking better</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ULM1CJeKoJpV_mQBYbtBg.png" /><figcaption>Box Plot After Standard Scaler</figcaption></figure><p><strong>Normalizer</strong> is applied on rows, not columns, so sklearn users shouldn’t get confused and must not use normalizer. Some of the used applications cases of Normalizer are comparing multiple entities during the same time-series, i.e. stock movements of multiple stocks in a given period</p><h3>Conclusion</h3><ul><li>Skewed target feature should be treated for normality before modelling, especially when the outliers are also important</li><li>Treating a skewed dependent feature and its effect should be understood during the analysis</li><li>MinMaxScaler should be the first choice for scaling</li><li>Experiments and observations can help us further to decide on the right approach</li></ul><p>Thanks for reading. If you have liked this article, you may also like <a href="https://towardsdatascience.com/pandas-index-explained-b131beaf6f7b"><em>Pandas Index -Invisible </em></a><em>Soul,</em> <a href="https://medium.com/analytics-vidhya/pandas-stack-unstack-pivot-table-crosstab-5cb9a16fe020"><em>Pandas Pivot &amp; Stacking</em></a></p><ul><li><em>For more, please follow me on </em><a href="https://medium.com/@manusharma_66718"><em>Medium</em></a></li><li><em>Let’s Connect on </em><a href="http://www.linkedin.com/in/data2datasciences/"><em>Linkedin</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f4436e8e074" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/transformation-scaling-of-numeric-features-intuition-7f4436e8e074">Transformation &amp; Scaling of Numeric Features: Intuition</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas: Stack/Unstack, Pivot_table & CrossTab]]></title>
            <link>https://medium.com/analytics-vidhya/pandas-stack-unstack-pivot-table-crosstab-5cb9a16fe020?source=rss-cc90427ba155------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cb9a16fe020</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Manu Sharma]]></dc:creator>
            <pubDate>Sat, 14 Sep 2019 11:59:30 GMT</pubDate>
            <atom:updated>2025-09-18T06:11:02.501Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pandas: Pivot &amp; Multiindex</h3><h4>Pivot Table, Stack/ Unstack are essential pandas methods to work with <em>MultiIndex</em> objects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4g0BlUo7Hd7Flru1" /><figcaption>Out of clutter, find simplicity</figcaption></figure><p>Creating a robust prediction engine starts with creating the right dataset. In most of real-life data science problems, we don’t get a clean &amp; well-prepared dataset for model building. During this whole process,</p><ul><li>we define the problem and scope,</li><li>define and create the response variable, derive new features,</li><li>do a lot of data processing &amp; visualization.</li></ul><p>And all the above steps need pandas again. In the last exercise, we discussed index P<a href="https://towardsdatascience.com/pandas-index-explained-b131beaf6f7b">andas Index Explained</a>, which is crucial to understand pandas better. The previous article explained index usage for selection and filtering. In this exercise, we see the usage of <em>Stack/Unstack, Pivot_table, CrossTab </em>for data processing &amp; visualization. We will be using the <em>UCI Machine Learning Adult Dataset,</em></p><blockquote><strong>Business Problem:</strong> Classification (a person earns more than 50k or less); <strong>Response feature: </strong>Label ; <strong>Predictors</strong>: country, age, education, occupation, marital status etc.</blockquote><p>The following are explaining snippets from the notebook, GitHub repo for this exercise can be cloned from the link at the end. Reading the data now😌</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uEVGdbJc1tQZpR7CwgaBmQ.png" /></figure><p>Some of the predictor features are age, <em>workclass</em>, <em>education</em>, <em>marital_status</em>, <em>race, sex, native_country, hours_per_week </em>and response feature is <em>label</em>. Also, we are replacing label values <em>&lt;=50K, &gt;50K </em>by <em>0</em> and <em>1</em> respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sks2YDEJEc4R-a7P3abYIw.png" /></figure><p>The <em>bi-variate analysis </em>explains and validates the relationship between features. And categorical features need <em>groupby</em> and apply functions to understand their relationship with numeric and other categorical features. In this context Pandas Pivot_table, Stack/ Unstack &amp; Crosstab methods are very powerful.</p><h3>Pivot_table</h3><p>It takes 3 arguments with the following names: index, columns, and values. Categorical Features from the dataset can be passed in index and columns. The cell values of the new table are taken from a column given as the values parameter.</p><p>Coming back to our dataset, usually, earnings are frequently associated with the occupation. So, let’s pivot the occupation and see which occupations have high-earning people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-lMX3ma3CQI9U049StWCsw.png" /><figcaption>Exec Managerial, Prof-speciality and sales have the highest number of male and female people who earn more than 50K, it may come out as one important feature.</figcaption></figure><p>Multiple columns can also be used during pivoting, we can use pivot_table like the following for multiple columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8q0jHjSh2YhCOoRQCWZAWQ.png" /><figcaption>multi-index columns</figcaption></figure><p>Also, check <strong>pt.index </strong>and <strong>pt.columns,</strong> it will print the index and columns of pivot_table. Multi- index columns can be reduced to a single level and can also be renamed like following:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--ioPqCGpveCIFHpXzbH5w.png" /><figcaption>Single level columns after combining the previous two levels</figcaption></figure><h3>CrossTab</h3><p><em>pd.crosstab </em>can be used just like <em>pivot_table</em>, but by default, it computes a simple frequency table of two (or more) <strong>categorical features.</strong> Because <em>values</em> are not mandatory for crosstab, but <em>values</em> and <em>aggregation function</em> can be passed, if required.<br>Let’s look at a simple example of <em>crosstab</em> and plot it. <em>Education</em> is also one important factor in estimating earnings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O1DjxikykP3G5GkiMrsjEw.png" /><figcaption>frequency table of education and sex</figcaption></figure><p>Crosstab can be used just like pivot_table by passing <em>values</em> and <em>aggregation</em> parameters. Following is a Bar plot of people earn more than 50K across education and sex(male/female) features with the help of crosstab</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ozV8sz563AoEM0L-vi8edg.png" /><figcaption>cross tab plot: index on x-axis and values on the y-axis</figcaption></figure><p>pd.cross tab also has a parameter <em>Normalized</em>. While we do bivariate analysis, frequency tables can also be normalized to show <em>percentages</em> rather than counts:</p><pre>pd.crosstab(adult.education,adult.label_50, normalize = True)</pre><h3>Stack/ Unstack</h3><p>Stacking a DataFrame means moving the <em>innermost column </em>index to become the <em>innermost row index</em>. The inverse operation is called unstacking.<br><strong>What is Stacking</strong>? -If we have multiple-index columns and we reduce dataset columns by moving the innermost column level to the innermost row level. So the stack is <em>cols to rows(innermost).<br></em><strong>What is Unstacking?-</strong> If we have multiple-index rows and we reduce dataset rows by moving the innermost row level to innermost column level. So <em>unstack</em> is <em>rows to cols(innermost).<br></em>In our downloaded data, we don’t have any multiple indexes, but our pivoted dataset has multiple indexes. Viewing the pivot_table dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-YQNezcvy84O1kBEMKOACw.png" /><figcaption>Initial Pivot table</figcaption></figure><p><em>stack</em> moves the inner level column to inner level rows and unstacking can again be brought it back to the original pivot shape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQ7eDYNExPh_NTLJ94NJ6Q.png" /><figcaption>Pivot table after stacking</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUn6Z4BlMwrUs6bX0j_LRA.png" /><figcaption>Stack and unstack brought original pivot table</figcaption></figure><p>Stacking/Unstacking is possible at different levels directly with the parameter <em>level.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0PbQE00I-2bLNthi825Kg.png" /></figure><p>These methods really help us bring those insights and model explanations to come out. Hierarchical / Multi-level indexing is very exciting as it opens the door to some quite sophisticated data analysis and manipulation, especially for working with higher dimensional data. Jupyter nb for this exercise can be downloaded from the <a href="https://github.com/manu007sharma/PandasAll">Github repo.</a></p><p>Thanks for reading. If you have liked this article, you may also like <a href="https://towardsdatascience.com/pandas-index-explained-b131beaf6f7b"><em>Pandas Index -Invisible </em></a><em>Soul, </em><a href="https://medium.com/data-science/transformation-scaling-of-numeric-features-intuition-7f4436e8e074"><em>Scaling &amp; Transformation When &amp; Where</em></a></p><ul><li><em>For more, please follow me on </em><a href="https://medium.com/@manusharma_66718"><em>Medium</em></a></li><li><em>Let’s Connect on </em><a href="http://www.linkedin.com/in/data2datasciences/"><em>Linkedin</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cb9a16fe020" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/pandas-stack-unstack-pivot-table-crosstab-5cb9a16fe020">Pandas: Stack/Unstack, Pivot_table &amp; CrossTab</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas Index Explained]]></title>
            <link>https://medium.com/data-science/pandas-index-explained-b131beaf6f7b?source=rss-cc90427ba155------2</link>
            <guid isPermaLink="false">https://medium.com/p/b131beaf6f7b</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Manu Sharma]]></dc:creator>
            <pubDate>Tue, 19 Feb 2019 13:42:10 GMT</pubDate>
            <atom:updated>2025-09-18T06:01:30.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pandas is a best friend to a Data Scientist, and index is the invisible soul behind pandas</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xSA3noKb1Q5V9iKgbO4hdQ.jpeg" /><figcaption>index is like an address</figcaption></figure><p>We spend a lot of time with methods like <strong>loc, iloc, filtering, stack/unstack, concat, merge, pivot </strong>and many more while processing and understanding our data, especially when we work on a new problem. And these methods use indexes, even most of the <strong>errors</strong> we face are indices error. Index become more important in <strong>time series </strong>data. <strong>Visualisations</strong> also need good control over pandas index.</p><p>Index is like an address, that’s how any data point across the dataframe or series can be accessed. Rows and columns both have indexes, rows indices are called as index and for columns its general column names.</p><blockquote><strong>Machine/ Deep learning</strong> algorithms has a performance limit at prediction accuracy just like their ancestors, statistical models, so how do you get a significant better accuracy? Feed a better data for learning. Data Processing &amp; Feature Enginnering is the key</blockquote><p>Pandas have three data structures <strong>dataframe</strong>, <strong>series</strong> &amp; <strong>panel</strong>. We mostly use dataframe and series and they both use indexes, which make them very convenient to analyse.</p><p>Time to take a step back and look at the pandas&#39; index. It empowers us to be a better data scientist.</p><p>We will be using the <strong>UCI Machine Learning Adult Dataset, </strong>the<strong> </strong>following<strong> </strong>notebook has the script to download the data.</p><blockquote><strong>Business</strong> <strong>Problem</strong>: Classification (a person earns more than 50k or less) <strong>Predictor Variable: </strong>Label ; <strong>Predictors</strong>: country, age, education, occupation, marital status etc.</blockquote><p>The following Notebook is very easy to follow and also has small tips and tricks to make daily work a little better.</p><pre>adult  = pd.read_csv(&quot;<a href="https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data">https://archive.ics.uci.edu/ml/machine-    learning-databases/adult/adult.data</a>&quot;, names = [&#39;age&#39;,&#39;workclass&#39;,&#39;fnlwgt&#39;, &#39;education&#39;,    &#39;education_num&#39;,&#39;marital_status&#39;,&#39;occupation&#39;,&#39;relationship&#39;,&#39;race&#39;,&#39;sex&#39;,&#39;capital_gain&#39;,&#39;capital_loss&#39;, &#39;hours_per_week&#39;, &#39;native_country&#39;,&#39;label&#39;], index_col = False)</pre><pre>print(&quot;Shape of data{}&quot;.format(adult.shape))</pre><pre>adult.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4PLEyeg64vp_y_ZrcFWEQ.png" /></figure><p>Dataset has 32561 rows and 15 features, the leftmost series 0,1 2,3 … is index. Let’s Look at some more information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qolWcWq1K23Tr4aJQrFvkA.png" /></figure><p>Adult has rangeindex 32561 entries, an integer series from 0 to 32560.</p><h4>Selection methods</h4><ol><li>df.loc are for labels/ names</li><li>df.iloc are for position numbers</li></ol><blockquote>e.g. Lets assume Ram, Sonu &amp; Tony are standing at positions 1, 2 &amp; 3 respectively. If you want to call Ram you have two options, either you call him by his name or his position number. So, if you call Ram by his name “Ram”, you will use df.loc and if we will call him by his position number “1” we will use df.iloc.</blockquote><p>Before we understand <strong>loc</strong> <strong>&amp;</strong> <strong>iloc</strong> more, let&#39;s take a sample from our data for further analysis. We are taking a sample of 10000 observations, using pandas df.sample method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wLcDZBqX0Zh0lZVmrRX4uA.png" /></figure><p>Nice, now we have a dataset name df, and the leftmost series are 3,4,6,8, 11…Strange!</p><blockquote>It is because the rows will carry their original( old address ) index or index names from the adult dataset</blockquote><p>Naming our index will help us a little initially, its the indices from adult dataset.<br>look at the rows and column indices</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eg3Wj_DRZkfEbvhZ9sx8-g.png" /></figure><blockquote>Both rows and columns have indexes, and the name of index is ‘<strong>index_adult</strong></blockquote><p>Let&#39;s discuss a couple of examples on <strong>loc &amp; iloc</strong> methods</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LK2viPjEWk67uMMgUxocRg.png" /><figcaption>error on df.loc method</figcaption></figure><p>In the <strong>first example</strong> of .loc, it gave us an error</p><blockquote>because we have used .loc method and df has no row who has a name ‘2’, row index looks like a number to us, but they are name/label to .loc method</blockquote><blockquote>try replacing 2 with 3 or 4, it will work, because there are names ‘3’,’4&#39; as <strong>position names</strong></blockquote><p>In the <strong>second example</strong>, we are trying the same with .iloc, its a <strong>position number-</strong>based method</p><blockquote>“age” is first column so we will use its position which is 0</blockquote><blockquote>there will be a position 0,1,2,3 till the last row of df, so 2 will be the third row</blockquote><p>For our further analysis, let&#39;s Keep a few interesting variables only</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0JIq6QRgADtYLt6BueY8DQ.png" /></figure><p>Some of the times, it’s difficult to work with random numbers in index, at that time, resetting this index will make it a column and recreate another default index</p><blockquote>`<strong>reset_index</strong>()`will recreate index column every time we run it on same data <br>`<strong>drop = True</strong>` paramater won’t create that as column in the dataframe, look at the difference between following two dataset</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4XTjr2hpv79KWcMGbEfitw.png" /></figure><p>`<strong>inplace = Tru</strong>e` save us from assigning it to data again<br>we are not using `<strong>drop = True</strong>`, now df should have its last index as a column in it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjJGuhJ9Hci0KZXhZbKYuQ.png" /></figure><blockquote>df has another column <strong>index_adult</strong>, because of reset</blockquote><h4>Filtering</h4><p>Filter on India</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pk-kdbyF5LJPz3myXHLNFg.png" /></figure><p>After resetting our index, and applying a filter for India, we can see index hold itself from df, just like sampling, now the row index(4, 312, 637 ,902..) are from df and index_adult is the indices of these rows in adult</p><p>Let&#39;s look at the observations with more than 50k income across the gender</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*po5UFAakcUX6eOfP0icJ2g.png" /></figure><blockquote>Even this dataframe has an index, hard to recognise by looking at the dataframe, and individual items can be accessed like</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*tj938lCJs0xpOLyu1wPdGg.png" /></figure><p>Filter ind dataset for people with income more than 50K</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DRp4feaHoCbv9XtQbFg29Q.png" /></figure><blockquote>indices are intact with their rows, just like an address</blockquote><p>Very Impressive, these people are earning really good, Let&#39;s try to know their work hours since we don&#39;t have ‘hours_per_week’ in this data, we will bring it from <em>adult</em> with the help of indices</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*H5nT0PwUmYOpz43zulS1Uw.png" /></figure><blockquote>Indices made it very easy to bring more information easily, the above formula can be understood as filtering adult[‘hours_per_week’] on the <strong>address</strong> index_adult of ind_50</blockquote><p>mean of work hours per week for people who earn more than 50k</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XVpbKqAqt7eIpB6pscenBA.png" /></figure><p>Just with the use of index_adult, we were able to bring another column information easily</p><p>Index make filtering very easy and also give you space to move forward and backwards in your data</p><h4>one last use of the index for this intro exercise</h4><p>Filtering a complementary set from the data, just like train and test from the total dataset</p><p>we are slicing that part of ind, which is not in ind_50, i.e. people who are earning less than 50k</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XhTu5IwEa7XL41BtAPiIuQ.png" /></figure><blockquote>Nice! It looks like people who earn 50k &amp; more, work more hours per week in this data sample</blockquote><p>Jupyter nb can be downloaded from this <a href="https://github.com/manu007sharma/PandasAll">Github-repo</a>.</p><p>The true capability of pandas index can be realised only when we drill down our data with multi-indexing &amp; visualisations.<strong> </strong>Visit my next <a href="https://medium.com/@manusharma_66718/pandas-stack-unstack-pivot-table-crosstab-5cb9a16fe020">exercise</a> on <strong>stack/unstack, pivot_table &amp; crosstab</strong></p><p>Thanks for reading. If you have liked this article, you may also like <a href="https://medium.com/analytics-vidhya/pandas-stack-unstack-pivot-table-crosstab-5cb9a16fe020"><em>Pandas Pivot &amp; Stacking,</em></a><em> </em><a href="https://medium.com/data-science/transformation-scaling-of-numeric-features-intuition-7f4436e8e074"><em>Explained : ML </em>Transformation &amp; Scaling</a></p><ul><li><em>For more, please follow me on </em><a href="https://medium.com/@manusharma_66718"><em>Medium</em></a></li><li><em>Let’s Connect on </em><a href="http://www.linkedin.com/in/data2datasciences/"><em>Linkedin</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b131beaf6f7b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pandas-index-explained-b131beaf6f7b">Pandas Index Explained</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>