<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shivam Samaiya on Medium]]></title>
        <description><![CDATA[Stories by Shivam Samaiya on Medium]]></description>
        <link>https://medium.com/@shivamsamaiya?source=rss-192d191cb051------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KYnpzEeTj-A33qHkd__BvQ.jpeg</url>
            <title>Stories by Shivam Samaiya on Medium</title>
            <link>https://medium.com/@shivamsamaiya?source=rss-192d191cb051------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:16:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shivamsamaiya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beginner Friendly Exploratory Data Analysis on Haberman Breast Cancer Survival Dataset]]></title>
            <link>https://medium.com/aiguys/beginner-friendly-exploratory-data-analysis-on-haberman-breast-cancer-survival-dataset-4da95f314ad?source=rss-192d191cb051------2</link>
            <guid isPermaLink="false">https://medium.com/p/4da95f314ad</guid>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[haberman]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Shivam Samaiya]]></dc:creator>
            <pubDate>Mon, 08 Nov 2021 07:45:03 GMT</pubDate>
            <atom:updated>2021-12-22T17:58:11.599Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I will go through the Haberman dataset and will perform Exploratory Data Analysis which will be a good head start for someone who has just started their journey towards learning data science or data analytics. At the time of writing, I am also in the very early stage of learning data science concepts and techniques. So, I am open to feedback and constructive criticism as they will help me understand better. Okay, enough talking let&#39;s dive right into the dataset!</p><p>The dataset can be downloaded kaggle from the following link <a href="https://www.kaggle.com/gilsousa/habermans-survival-data-set">https://www.kaggle.com/gilsousa/habermans-survival-data-set</a>. If you are new to Kaggle, it is a website that has tonnes of datasets to perform, learn machine learning, participate in competitions, and connect with the community. From the link, you can download the dataset after logging in to Kaggle.</p><p>After downloading the CSV file, you will see 4 columns filled with some numbers without a header (column name) when you open it. So for that, you can again go to the above link and go to <strong>details </strong>for the CSV file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jxq8pz4gWGWs3C5ws5d1w.png" /><figcaption>Figure 1.</figcaption></figure><p>In the details section, you will find all the important information about the dataset, like the source of the dataset, information about columns, and what they indicate. The Haberman dataset contains information from a case study to recognize any pattern to predict breast cancer amongst 306 patients. The dataset has the age of the patients (numerical value), the year of the patient’s operation (value is reduced to -&gt; year-1900 for convenience, numerical), Number of positive axillary nodes detected (numerical), and a class label which has 2 classes -&gt; 1 and 2, where 1 is present when the patient has survived more than 5 years after operation and 2 when patient died within 5 years of operation.</p><p>Now data analysts or data scientists need to have some domain knowledge regarding the problem that they are solving. In the given dataset, we (as in non-medical professionals) have no idea about the <strong>Number of Positive Axillary Nodes Detected. </strong>So let&#39;s find some basic knowledge about this term.</p><blockquote><a href="https://en.wikipedia.org/wiki/Positive_axillary_lymph_node#:~:text=A%20positive%20axillary%20lymph%20node,whether%20cancer%20cells%20are%20present.">According to Wikipedia, A <strong>positive axillary lymph node</strong> is a lymph node in the area of the armpit (axilla) to which cancer has spread. This spread is determined by surgically removing some of the lymph nodes and examining them under a microscope to see whether cancer cells are present.</a></blockquote><p>Now we can start with importing the dataset into the Jupyter notebook and proceed with the analysis.</p><p>First, we will import all the necessary libraries and then the dataset.</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><p>Since our dataset has no column names, we should import the dataset and assign the column names to each column. After that, we will see the top 5 rows just to check.</p><pre>haberman=pd.read_csv(r&#39;C:\Users\u363200\Downloads\haberman dataset\haberman_columns.csv&#39;)<br>haberman.columns=[&#39;Age&#39;,&#39;Operation_Year&#39;,&#39;Positive_axillary_nodes_detected&#39;,&#39;Survived_5years_or_not&#39;]</pre><pre>haberman.head()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*w5DNIjkKXWe_pQIEY_iwVQ.png" /><figcaption>Figure 2.</figcaption></figure><p>Okay, now remember<strong><em> our objective is to identify the variables/features which are relevant to find whether the patient survived 5 years or more or not. </em></strong>So, we can say that our inputs are age, operation year, positive axillary nodes detected and output is survived 5 years or not.</p><p>Let&#39;s check some high-level information and statistics about the dataset.</p><pre>print(haberman.shape)</pre><p>Output-</p><p>(306,4)</p><p>Here dataset.shape gives us the dimensions of the table i.e. number of rows and number of columns. We can say that the total data points which we have are 306 and the total columns or features (input+output) are 4.</p><p>Next, we would like to see some statistical information about numerical features present in the dataset.</p><pre>haberman.describe()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*GiScUD2kiKRJLVrxpO64Sg.png" /><figcaption>Figure 3</figcaption></figure><p>dataset.describe() gives us all the high-level statistical information about the numerical dataset. Here std stands for standard deviation, 25% is the value at 25th percentile (25 % of all values are below this point) and similarily with 50% and 75%.</p><p>Now we need to check the metadata i.e. data about the dataset (types of each column, missing values, etc). For that following code is used</p><pre>haberman.info()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*DFouBDDmBZRBayXRKH09bA.png" /><figcaption>figure 4</figcaption></figure><p>Here we can see that all of the data columns are of integer type and all values are non-null. Since this is a classification problem on identifying which features can help us get some pattern for successful survival of patients for 5 or more years and we are not going to apply any machine learning model, for now, we can change the data type of Survived_5years_or_not to string type with value “yes” when the patient survived 5 or more years(=1) and “no” for not survived 5 years.</p><p>To do that we can create a simple mapping function that will map all the 1’s with “yes” and 2’s with “no”.</p><pre>boolean_convertor={1:&#39;yes&#39;,2:&#39;no&#39;}<br>haberman[&#39;Survived_5years_or_not&#39;]=haberman[&#39;Survived_5years_or_not&#39;].map(boolean_convertor)<br>haberman.head()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*gluQ6FHv2ckg1QOWY0UmPQ.png" /></figure><p>First, I have created a dictionary named boolean_convertor which has keys as 1,2 (which are input ) and values as “yes”,” no” (which are output). Then I have used .map() to map this dictionary on the Survived_5years_or_not. Finally, I printed the first 5 rows to check if mapping worked. It works!</p><p>The number of data points for each category can be calculated using .value_counts(). This function will give us the number of data points (rows) available for each category.</p><pre>haberman[&#39;Survived_5years_or_not&#39;].value_counts()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*9Gsgm_xFmcRVuD5Pbm8TJg.png" /><figcaption>figure 6</figcaption></figure><p>So there are 225 data points out of 306 where the patient has survived for 5 or more years and 81 data points were unfortunate patients who could not survive. It is an imbalanced dataset (both category data points are different).</p><p>To analyze all columns at once with categories for “yes” and “no”, we can use pair plots. Pair plots are a great tool to visualize 2 features or columns at a time for multivariate analysis (more than 2 features, independent variables). It&#39;s actually a n*n matrix of graphs where n is the feature / independent variable. Let’s how we can plot pair plots</p><pre>sns.set_style(&quot;whitegrid&quot;);<br>sns.pairplot(haberman, hue=&quot;Survived_5years_or_not&quot;, height=4);<br>plt.show()</pre><p>Output-</p><figure><img alt="pair plot" src="https://cdn-images-1.medium.com/max/971/1*mMSQdMllm6MojjAwbjcB6Q.png" /><figcaption>Figure 7</figcaption></figure><p>Let’s understand the code first then I will try to get insights from the pair plot.</p><p>The sns.FacetGrid() creates a grid/matrix for your variables and takes input from the dataset, the category is taken in the form of color (hue) i.e. your categories will have different colors. The .set_style(“whitegrid”) sets your background/canvas as white with grid lines.</p><p>If you google ‘FacetGrid’ the <a href="https://www.tutorialspoint.com/seaborn/seaborn_facet_grid.htm">TutorialPoints </a>definition says</p><blockquote>“FacetGrid object <strong>takes a dataframe as input and the names of the variables</strong> that will form the row, column, or hue dimensions of the grid. The variables should be categorical and the data at each level of the variable will be used for a facet along that axis.FacetGrid object <strong>takes a dataframe as input and the names of the variables</strong> that will form the row, column, or hue dimensions of the grid. The variables should be categorical and the data at each level of the variable will be used for a facet along that axis.”</blockquote><p>It takes input as<em> .pairplot(dataset,hue=categorical_column,height=size or shape)</em> and in the last line .add_legend() will us the legend to understand which color is representing which category.</p><p>Here we can see those diagonal elements have different graphs than the rest of the others. The graphs apart from the diagonal ones have 2 different features which can be seen on the x and y-axis, so when we plot 2 numerical different features we get scatter plots (<a href="https://www.mathsisfun.com/data/scatter-xy-plots.html">https://www.mathsisfun.com/data/scatter-xy-plots.html</a> great article to learn about scatter plots).</p><p>Now the diagonal elements have both x and y-axis as same columns hence, instead of scattering plots density plots are present. Density plots are smoothed versions of the histogram in very vague language, but both give us the distribution of data points (frequency of values across a range).</p><p>Okay, let’s focus on the objective here (which is to find a graph that can help us easily differentiate orange color (no) from blue (yes)). Ideally, we want a graph where we can see clearly the orange part and blue part separated. In our case, there is no graph fulfilling our condition. All the graphs shown have well-mixed categories which are hard to separate. But, the probability density function or pdf plot for Positive_axillary_nodes_detected shows some difference between orange and blue. <strong><em>So, we can try performing univariate analysis on Positive_axillary_nodes_detected for distinguishing the Survived_5years_or_not column.</em></strong></p><p>Let us divide our dataset into 2 datasets (<strong><em>Survived_5years_or_not=”yes” and Survived_5years_or_not=” no”)</em> </strong>for making the univariate analysis easier on <strong><em>Positive_axillary_nodes_detected </em></strong>easier.</p><pre>haberman_yes=haberman[haberman[&#39;Survived_5years_or_not&#39;]==&quot;yes&quot;]<br>haberman_no=haberman[haberman[&#39;Survived_5years_or_not&#39;]==&quot;no&quot;]</pre><p>Now we will create Probability Density Functions (PDF) and Cumulative Density Functions (CDF) for both Positive_axillary_nodes_detected in both our datasets haberman_yes and haberman_no.</p><p>PDF in very simple language are smoothed versions of histograms, they give the probability of occurring of any particular small interval (because the probability of some event happening at an exact point is always 0, we work for very small intervals). Some of all the values of the Y-axis in pdf will lead to 1 and all the values are between 0 and 1. Remember PDF has a very technical definition that may not be absorbed by all that’s why I have put it in very simple language. (<a href="https://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm">https://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm</a> refer this for detailed definition)</p><p>Similarly, CDF (Cumulative Density Functions) are the integrals of PDF over an interval. At any point x=c on the CDF curve, it tells us the probability that event has occurred before CDF(x=c)=P(x≤c) (sum of all probabilities from x=0 to x=c). Integrating PDF will give us CDF and differentiating CDF will give us PDF over an interval. They both help us understand the data and its behavior in univariate analysis.</p><p>Lets plot cdf and pdf for haberman_yes[‘Positive_axillary_nodes_detected’]-</p><pre>counts, bin_edges = np.histogram(haberman_yes[&#39;Positive_axillary_nodes_detected&#39;], bins=np.arange(-2,22,2), density = True)<br>pdf = counts/(sum(counts))<br>cdf = np.cumsum(pdf)<br>plt.plot(bin_edges[1:],pdf)<br>plt.plot(bin_edges[1:], cdf)</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*AY-GKd2YQKFquPKQ8VGfag.png" /><figcaption>figure 8. CDF and PDF for haberman_yes[‘Positive_axillary_nodes_detected’]</figcaption></figure><p>The above code shown first creates count (frequency) and bin_edges which will be used to create a histogram for our desired column (‘positive nodes’) then pdf (here a list )can be created from a histogram by simply dividing count by sum of counts. And finally, NumPy has a simple function .cumsum() which creates CDF (again a list )by taking input of pdf. At last, I have plotted them with bin_edges (it has started from [1:] so that number of elements are matched)</p><p><strong>Insights-</strong></p><p>From figure 8 we can see that CDF at Positive_axillary_nodes_detected =5 is nearly 0.85 which means that<strong><em> 85 percent of all the patients who survived 5 or more years after the operation had positive axillary nodes detected less than 5.</em></strong></p><p>Similarly lets create cdf and pdf for haberman_no[‘Positive_axillary_nodes_detected’]-</p><pre>counts, bin_edges = np.histogram(haberman_no[&#39;Positive_axillary_nodes_detected&#39;], bins=np.arange(-2,22,2), density = True)<br>pdf = counts/(sum(counts))<br>cdf = np.cumsum(pdf)<br>plt.plot(bin_edges[1:],pdf)<br>plt.plot(bin_edges[1:], cdf)</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*0c9bOk1rmHKD43ihrJfYPQ.png" /><figcaption>fig 9. CDF and PDF for haberman_no[‘Positive_axillary_nodes_detected’]</figcaption></figure><p><strong>Insights-</strong></p><p>CDF(x=5) is nearly 0.6 and CDF(x=20) is almost 1. So, from here we can say that <strong><em>approximately 40 % of all the patients who could not survive 5 or more years after the operation had positive axillary nodes detected between 5 to 20.</em></strong></p><p>Now let us gather some more statistical information from the data like mean, median, std. For that, we can either use our created haberman_yes and haberman_no datasets but it&#39;s not feasible when you will have more than 4 or 5 categories. So instead we can use .groupby() which automatically aggregates information based on categories of the input column. For example, if we want the mean for category “yes” and “no” of survived 5 years or more for positive_axillary_nodes_detected we can do the following operation.</p><pre>haberman.groupby(‘Survived_5years_or_not’).Positive_axillary_nodes_detected.mean()</pre><p>Output-</p><pre>Survived_5years_or_not<br>no     7.456790<br>yes    2.791111<br>Name: Positive_axillary_nodes_detected, dtype: float64</pre><p>It says that on average the patients who survived more than 5 years after the operation had an average of 2.8 positive axillary nodes detected while those couldn’t have average of 7.46 positive axillary nodes detected.</p><p>But, there’s a catch since we know that means can be easily corrupted with outliers (i.e. more the outliers present, the more deviated mean ) and we can see from the pdf and CDF both “yes” and “no” categories have outliers.</p><p>In these situations, we can rely on medians to understand the central tendency of our data as they are more robust with the presence of outliers.</p><pre>haberman.groupby(‘Survived_5years_or_not’).Positive_axillary_nodes_detected.median()</pre><p>Output-</p><pre>Survived_5years_or_not<br>no     4<br>yes    0<br>Name: Positive_axillary_nodes_detected, dtype: float64</pre><p>This gives a fairly better picture of the central tendency of both categories.</p><p>Now we should see how the spread is the data for both categories. For that ideally, we use standard deviations (STD). STD is the square root of the average of the square of the distance from the mean. If the spread is low, it means they are densely grouped, or if the spread is high, it means they are widely present.</p><pre>haberman.groupby(‘Survived_5years_or_not’).Positive_axillary_nodes_detected.std()</pre><p>Output-</p><pre>Survived_5years_or_not<br>no     9.185654<br>yes    5.870318<br>Name: Positive_axillary_nodes_detected, dtype: float64</pre><p>The same problem is encountered as mean since the data has outliers and std has a square component which enhances the distance it is not a good practice to inference any conclusion from std.</p><p>Instead, we can use Median Absolute Deviation (MAD) which is the median of absolute difference of all values from the median. Here’s how we can import it-</p><pre>from statsmodels import robust<br>print(robust.mad(haberman_yes[&#39;Positive_axillary_nodes_detected&#39;]))<br>print(robust.mad(haberman_no[&#39;Positive_axillary_nodes_detected&#39;]))</pre><p>Output-</p><pre>0.0<br>5.930408874022408</pre><p>Insight-</p><p><strong><em>It shows that for the “yes” category spread is almost zero i.e. all the values are nearly situated around the central tendency while for the “no” category MAD is 5.93 which values are spread widely in that category.</em></strong></p><p>Let&#39;s check if see outliers from the box and whiskers plot. Box and Whiskers Plot is a great tool to identify the percentiles of the data for example what is the 90th percentile value? which means which value is above 90 percent of the data. Box plot divides the data into 4 groups-&gt; 0-25th percentile values(Q1), 25-50th percentile values(Q2), 50–75th percentile values(Q3) and 75–100(Q4) percentile values. Then it shows a box for 25th to 75th percentile values with a line for 50th percentile value (median). And all other values are plotted in whiskers (like cat’s whiskers). It also shows us outliers in the data if they are present. Any value which is either less than Q1–1.5(IQR) or Q3+1.5(IQR) where IQR=Q3-Q1 are qualified as outliers.</p><p>Lets plot box plots -</p><pre>sns.boxplot(x=’Survived_5years_or_not’,y=’Positive_axillary_nodes_detected’, data=haberman)<br>plt.show()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*dH6ToupMcBPaxRruzN_jQQ.png" /><figcaption>fig 10. Box plot</figcaption></figure><p>Insight-</p><p><strong>The plot shows that there are an ample amount of outliers present in the “yes” category with 75 percent of the values falling below 3. In the “no” category data is more spread compared to “yes” with IQR (75th percentile-25th percentile) equal to 10.</strong></p><p>We know that histogram and pdf give the distribution of the data and box plot gives us quartiles (or percentile values) of the data, what if we had some graph or method which combines both of them and brings out the best from them. For that, we have violin plots which not only tell us the quartiles and percentiles but also give us the distribution of the data.</p><pre>sns.violinplot(x=&quot;Survived_5years_or_not&quot;, y=&quot;Positive_axillary_nodes_detected&quot;, data=haberman, size=8)<br>plt.show()</pre><p>Output-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*YEXfzz8sX9NWVNMuFJunqQ.png" /><figcaption>fig 10. violin plot</figcaption></figure><p>Here we can see the “no” category with a central tendency around 5 is more dispersed than the “yes” category with a central tendency around 0.</p><p><strong>Final Conclusion</strong></p><ol><li>From the pair plot, we found that the most important feature amongst all given features for classification of survived or not above 5 years is Positive axillary nodes. Research also shows that it is one of the primary variables in identifying breast cancer among women.</li><li>From there we started univariate analysis on positive axillary nodes detected with our class labels “yes” and “no” for survival within 5 years of not.</li><li>PDF and CDF gave us important information that around 80%-85% of all the patients that have survived above 5 years had positive axillary nodes between 0 to 4 while around 40 % of patients who unfortunately could not survive above 5 years had positive axillary nodes between 5 to 20.</li></ol><p>My name is Shivam Samaiya and I recently started my journey of learning Data Science. I am currently working as a Trainee Analyst with WNS Global Services. If you think there are some improvements needed or some feedbacks are there please let me know or if you want to connect with me, ping me on <a href="https://www.linkedin.com/in/shivam-samaiya/">LinkedIn</a>. The link to the Jupyter Notebook for the same is <a href="https://drive.google.com/drive/folders/1Dj0TjJiXldspu6aHBaPl90ndQnDMPr67?usp=sharing">here</a>.</p><p>References</p><ol><li>Kaggle-<a href="https://www.kaggle.com/gilsousa/habermans-survival-data-set">https://www.kaggle.com/gilsousa/habermans-survival-data-set</a></li><li><a href="https://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm">https://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm</a></li><li><a href="https://towardsdatascience.com/will-habermans-survival-data-set-make-you-diagnose-cancer-8f40b3449673">https://towardsdatascience.com/will-habermans-survival-data-set-make-you-diagnose-cancer-8f40b3449673</a></li><li><a href="https://en.wikipedia.org/wiki/Positive_axillary_lymph_node#:~:text=A%20positive%20axillary%20lymph%20node,whether%20cancer%20cells%20are%20present">https://en.wikipedia.org/wiki/Positive_axillary_lymph_node#:~:text=A%20positive%20axillary%20lymph%20node,whether%20cancer%20cells%20are%20present</a>.</li><li><a href="https://www.mathsisfun.com/data/scatter-xy-plots.html">https://www.mathsisfun.com/data/scatter-xy-plots.html</a></li><li><a href="https://www.tutorialspoint.com/seaborn/seaborn_facet_grid.htm">https://www.tutorialspoint.com/seaborn/seaborn_facet_grid.htm</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4da95f314ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aiguys/beginner-friendly-exploratory-data-analysis-on-haberman-breast-cancer-survival-dataset-4da95f314ad">Beginner Friendly Exploratory Data Analysis on Haberman Breast Cancer Survival Dataset</a> was originally published in <a href="https://medium.com/aiguys">AIGuys</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>