<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zach Zazueta on Medium]]></title>
        <description><![CDATA[Stories by Zach Zazueta on Medium]]></description>
        <link>https://medium.com/@zachary-a-zazueta?source=rss-8b9b7ba80485------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O8D4IrYr7I5nHVGAHyvqbw.jpeg</url>
            <title>Stories by Zach Zazueta on Medium</title>
            <link>https://medium.com/@zachary-a-zazueta?source=rss-8b9b7ba80485------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:46:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zachary-a-zazueta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SHAP Library in Python]]></title>
            <link>https://zachary-a-zazueta.medium.com/shap-library-in-python-80ea1fb64e13?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/80ea1fb64e13</guid>
            <category><![CDATA[feature-importance]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[shap]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Sat, 31 Oct 2020 02:35:24 GMT</pubDate>
            <atom:updated>2020-10-31T02:35:24.829Z</atom:updated>
            <content:encoded><![CDATA[<p>Every profession has their unique toolbox, full of items that are essential to their work. Painters have their brushes and canvas. Bakers have mixers, pans, and ovens. Trades workers have actual toolboxes. And those in a more corporate environment will have a suite of hardware and software necessary to complete the task at hand. Data scientists tend to fall into this last bucket.</p><p>I’m of the opinion that one of the most exciting tools in a data scientist’s toolbox are the visualization libraries available for use. Data viz is a great way for technical operators to bridge a gap and bring insights to a non-technical audience.</p><p>A brilliant library that was released in late 2016 is the SHAP library. The SHAP (SHapley Additive exPlanations) library was developed to unpack some of the “black box” aspects of machine learning models, and to better explain the output of models. In the developers’ own words:</p><p><em>“SHAP connects game theory with local explanations, uniting several previous methods and representing the only possible consistent and locally accurate additive feature attribution method based on expectations.”</em></p><p>**Quick note** The remainder of this post will provide a quick introduction to SHAP and I’ll show how I’ve used it to better understand a few models I’ve built. I am still learning a lot about this library, so please do additional research as you utilize these methods yourself. Any feedback is also appreciated!</p><p><strong>Working with SHAP</strong></p><p>The SHAP visualizations use what are known as SHAP Values to plot information. SHAP Values are the determined impact each feature will have on an output, derived from breaking down the prediction. Samuele Mazzanti does a great job explaining how SHAP values are calculated <a href="https://towardsdatascience.com/shap-explained-the-way-i-wish-someone-explained-it-to-me-ab81cc69ef30">here</a>.</p><p>The library is easily downloaded with a pip install.</p><pre>!pip install shap</pre><p>SHAP is versatile and has “explainer” models for every type of ML model. A general explainer can be called with the shap.KernelExplainer, but it tends to perform more slowly than using the explainer that is specific to the type of model you have built. It is best to call the appropriate explainer when you are ready to begin plotting.</p><p>SHAP has explainers for tree models (e.g. XGBoost), a deep explainer (neural nets), and a linear explainer (regression).</p><p>After calling the explainer, calculate the shap values by calling the explainer.shap_values() method on the data.</p><pre>import shap</pre><pre>#Load JS visualization code to notebook<br>shap.initjs()<br><br>explainer = shap.TreeExplainer(xgbclassifier)<br>shap_values = explainer.shap_values(xgbX_train)</pre><p>Then we get to visualize it using one of the many plots available!</p><pre>shap.summary_plot(shap_values, xgbX_train)</pre><p>Here are two examples of personal projects where I’ve used SHAP to better understand the underlying model outputs.</p><p>In this example, I was working with data about the hospital readmittance of patients who had some length of inpatient treatment for diabetes. You can find the full project on my Github <a href="https://github.com/zachzazueta/Diabetes_Readmittance_Predictions">here</a>. In the summary plot, we are able to see a few things that will drive the model output up or down. A model output of 1 would signify that the patient would be readmitted for additional inpatient care while a 0 reflected no readmittance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*b0FfHTbdziLsJgcq47dwxw.png" /><figcaption>Determining patient readmittance</figcaption></figure><p>Each feature for each data object in the set is represented by a point on the graph. If feature values are high, they will show as red — if they are blue, they have lower values. The position of the point along the x-axis shows whether the feature value is pushing the predicted output towards 0 or 1. So we can see that if a patient had a high number of previous inpatient (number_inpatient) procedures or emergency room visits (number_emergency) preceding the current visit, they were more likely to be readmitted again. Lower age (age) drew the outcome closer to non-readmittance, as did having a lower number of medications (num_medications).</p><p>In this second example, we explore the features that are causing customers of a telecommunications company to fall into one of four clusters, the output of a K-prototype cluster analysis. You can find the full project <a href="https://github.com/zachzazueta/telecom_churn_project">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*oEspA-B4hvZkWyS_YhjqfA.png" /><figcaption>Inspecting customer clusters</figcaption></figure><p>I use a bar plot style for the summary plot here. The size of the bars shows the importance of the feature, so we can see that Total Charges and Internet Service had significant importance on the clustering. Streaming TV, streaming movies, and Churn have strong importance, and the 7 features following still have moderate explanatory importance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*M6X8cX3ac1ICqymMaHixDg.png" /><figcaption>Cluster prototypes</figcaption></figure><p>Inspecting the prototype modes of the clusters (where Class 0 aligns with Cluster 1, Class 1 -&gt; Cluster 2…), these make intuitive sense — Total Charges was very important for those in Cluster 4, and they tend to subscribe to all services. Internet Service was the most important feature for Cluster 3, and they had No Internet Service. Opting out of Streaming TV and Movies allowed Cluster 2 to stand apart from Clusters 1 &amp; 4.</p><p>Hopefully the above provides a quick insight as to how SHAP can be applied to data science work. I think it’s a great tool and I’m looking forward to working with it more in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80ea1fb64e13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K-Prototypes clustering — for when you’re clustering continuous and categorical data]]></title>
            <link>https://zachary-a-zazueta.medium.com/k-prototypes-clustering-for-when-youre-clustering-continuous-and-categorical-data-6ea42c2ab2b9?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ea42c2ab2b9</guid>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[k-modes]]></category>
            <category><![CDATA[cluster-analysis]]></category>
            <category><![CDATA[k-prototype]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Fri, 23 Oct 2020 20:58:40 GMT</pubDate>
            <atom:updated>2021-08-07T13:14:25.683Z</atom:updated>
            <content:encoded><![CDATA[<h3>K-Prototypes clustering — for when you’re clustering dynamic, real world data</h3><p>Clustering is one of the most popular types of unsupervised machine learning. Clustering techniques allow us to group data objects into similar classes in such a way that items within a group share similar characteristics, while items in different groups are not similar at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*eNWN14P6oQwySgxLuMqB9g.png" /><figcaption>cluster analysis 101</figcaption></figure><p>It could be said that K-means clustering is the most popular non-hierarchical clustering method available to data scientists today.</p><p>For K-means, for each of the predetermined number of K clusters (this is the part that makes it a non-hierarchical algorithm), a seed is selected and each data object (row) in the set is assigned to one of the clusters according to the seed it is closest to. Then for each cluster, a new mean is calculated based on the objects which have been assigned and all objects are once again distributed across all K clusters. The process repeats until there is no change to the computed cluster means, and therefore no objects shift their cluster assignment.</p><p>Now, K-means is great for data sets that are all continuous variables, but not so much for when you have data that are categorical. The reason for this is that K-means uses Euclidean distance to measure the similarity of objects… and there aren’t Euclidean distances between data segmented by categories. This can pose a problem, especially when you have a lot of customer data coming from sources outside of something like analytics about their interaction with the company webpage or just looking at past spending habits. Luckily, Zhexue Huang <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.15.4028&amp;rep=rep1&amp;type=pdf">delivered</a> two extensions to the K-means algorithm in 1998, which have allowed for the application of clustering on more real-world data sets that contain a mix of continuous and categorical data.</p><p>For the remainder of this post, let’s consider a <a href="https://www.kaggle.com/blastchar/telco-customer-churn/data">data set</a> from an old Kaggle competition. The data are attributes for customers of a telecommunication company, and the goal of the competition was to predict whether users would churn (leave the services of the company). I’m using the data here to better understand how the customers grouped, so the goal is less predictive and more understanding the client base.</p><p>Here is a partial sample of the data (this is the head of just a few rows):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*CXdto3FEngczS4FxFP_h8Q.png" /><figcaption>partial head of the data frame</figcaption></figure><p>The first addition Huang gave us is the K-modes algorithm, which is used to cluster categorical data. Instead of reducing the Euclidean distance between cluster objects and cluster means (centroids) to reduce a cost function, K-modes replaces cluster means with modes and uses a “matching dissimilarity” measure to update cluster modes. This frequency-based method reduces the clustering cost function.</p><p>In other words, instead of measuring Euclidean distance on vector data, with the K-modes algorithm the “distance” calculated is simply the number of disagreements between each data point and cluster mode centroid. Looking at the first 10 rows shown above and ignoring the continuous TotalCharges and Churn categories for now, let’s see how that plays out.</p><p>The value counts of those rows are as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*EJIMpDOu3io1NkDY__VFEw.png" /><figcaption>value counts</figcaption></figure><p>So the mode centroid for the categorical variables (ignoring Churn and Total Charges for now) would be the following:</p><ul><li>‘Male’ for Gender</li><li>‘No’ for Has phone service</li><li>‘DSL’ for Internet service</li><li>‘Month-to-month’ for Contract length</li><li>‘Mailed check’ for Payment method</li></ul><p>And then for each object, you could calculate the distance between that centroid and the object by summing the number of dissimilarities there are to the centroid mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*kt9dtsxTyL1k2JclQ83p9Q.png" /><figcaption>Green denotes a match to the ‘Mode’ row</figcaption></figure><p>Rows 5 and 8 are the least similar, where their distance of 4 comes from the fact that of the 5 mode characteristics of the group, they only match 1 (it happens to be Month-to-month billing for both). Rows 2 and 10 both have a distance of 0 because they match the group mode identically.</p><p>For the whole process, each object has a mode distance measure calculated for each of K centroid modes (we’ve only shown the first centroid above). If the distance between the object and it’s assigned centroid was greater than the distance to another centroid, the object would be reassigned to another centroid. Similar to K-means, this process is repeated until there are no more object re-assignments and group modes are stable. This is the gist of K-modes.</p><p>This brings us to K-prototypes, which is simply a joining of the K-means and the K-modes methods.</p><p>Finding the optimal K for number of clusters can be tricky, and there are a few methods to do so. One method is by using the Calinski-Harabasz score. The Calinski-Harabesz score is simply a calculation of the variance ratio, which is the ratio of the intra-cluster variance (dispersion between points in a cluster) to the inter-cluster variance (the dispersion between clusters). It is ideal to have a higher score, as that indicates a better performance. Use of CH will be featured in a future post.</p><p>Another way to find the optimal number for K is to construct an elbow plot, which will show at which number of K clusters the cost begins decreasing in a linear fashion. For the K-prototypes function, cost is defined as the sum distance of all points to their respective cluster centroids.</p><p>In addition to the number of clusters K, we have to determine how to select the starting points for the clusters. There are two methods to initialize the clusters with K-Prototypes, Huang and Cao. Selecting ‘Huang’ as the init, the model will select the first k distinct objects from the data set as initial k-modes and then assign the most frequent categories equally to the initial k-modes. The ‘Cao’ approach selects prototypes for each data object based on the density of the data point and the dissimilarity value.</p><p>In the below code, I construct an elbow plot to determine the optimal K by doing the following:</p><ul><li>Iterate through a range of numbers 2 through 10, specifying the number of clusters for the K-prototype function. As you can see I have initialized the function with the Huang approach.</li><li>To build the clusters, I fit.predict the data, specifying which of my columns are categorical.</li><li>I append the cost and number of clusters used to compute that cost to appropriate lists.</li><li>Finally, I can plot a simple scatterplot and am able to see where the cost starts to flatten off.</li></ul><p>Here is the code and the plot:</p><pre>costs = []<br>n_clusters = []</pre><pre>cat_cols = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14]</pre><pre>for i in tqdm(range(2, 10)):<br>    try:<br>        kproto = KPrototypes(n_clusters=i, init=&#39;Huang&#39;, verbose=2)<br>        clusters = kproto.fit_predict(data_corr, categorical=cat_cols)<br>        costs.append(kproto.cost_)<br>        n_clusters.append(i)<br>    except:<br>        print(f&quot;Can&#39;t cluster with {i} clusters&quot;)</pre><pre>fig = go.Figure(data=go.Scatter(x=n_clusters, y=costs))<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*dktsxu85Lhk4ug8P6GXXBQ.png" /><figcaption>Elbow plot</figcaption></figure><p>**Side note — you can see that I run my range in tqdm, which provides a cool progress bar while the code runs. Read more about it <a href="https://github.com/tqdm/tqdm">here</a>!**</p><p>Typically I would select 3 clusters, as that is where the cost flattens significantly, however when I ran that, all of the clusters had a mode of ‘No’ for Churn, which is a key feature we are interested in, so I ran it with 4 clusters to show these final clusters, which provide us with Group 1 where mode on Churn is ‘Yes’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*M6X8cX3ac1ICqymMaHixDg.png" /><figcaption>Clusters with K-Prototypes (categorical Modes and continuous Means)</figcaption></figure><p>Group count is always important in cluster analysis — a group with very few members could be significant and identify outliers, but might not provide information about the whole. The member counts were as follows:</p><ul><li>Group/Cluster 1 — 1,371</li><li>Group 2 — 2,314</li><li>Group 3 — 1,528</li><li>Group 4 — 1,819</li></ul><p>Note also that TotalCharges has been scaled. Because cluster analysis is distance based, it is important to scale all numerical data. Other continuous variables in this data set would also have been scaled had they not been removed due to collinearity. Read another of my posts that discusses more instances of scaling data <a href="https://zachary-a-zazueta.medium.com/data-transformation-in-ml-standardization-vs-normalization-70ba26de9060">here</a>.</p><p>As you can see, there are some interesting differences. While two groups are more likely to have Fiber Optic than DSL (Groups 1 &amp; 4 vs Group 3), there is a split between Groups 1 and 3 when we look at the services they elect. The customers in Group 1 are churning more frequently than Group 3, however they do not pay for extras like online security, backup, tech support, or device protection.</p><p>Senior Citizens do not seem to be the majority in any of the cluster groups, however many of the trends seen in Group 3 align with what we saw for Seniors during data exploration prior to modeling — no internet service, long contracts, no extra frill services, and lowest overall total charges.</p><p>There are certainly more insights to gain from this information. We could run predictive models on this data and determine whether this clustering could point to customers who were most likely to leave. We can explore the importance each variable has on determining which cluster the object fall into — I will likely go through those steps in a future post and use the SHAP library to do so.</p><p>If you would like to see the full code, including data sourcing, exploration, cleaning, and analysis, check out my Github <a href="https://github.com/zachzazueta/telecom_churn_project">here</a>. Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ea42c2ab2b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data exploration made easy — subplots in Matplotlib]]></title>
            <link>https://zachary-a-zazueta.medium.com/data-exploration-made-easy-subplots-in-matplotlib-df357d23bd72?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/df357d23bd72</guid>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[subplot]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Sat, 17 Oct 2020 20:58:05 GMT</pubDate>
            <atom:updated>2020-10-22T17:26:46.502Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data exploration made easy — subplots in Matplotlib</h3><p>Doing more work with less effort is the name of the game in coding. For data scientists, this is a huge advantage to using tools like Python to assist with extracting as much information from data as possible in an efficient way.</p><p>Data exploration is one of the early steps of the data science process. During this step, it is important to gain an understanding of the structure and content of the information you are working with. There are functions like .describe() and .info() which make it easy to get cursory summary statistics about numerical data, identify skew, see null values, and identify data types.</p><p>Beyond this, it is often helpful to begin to visualize the data. This will also help the analyst to spot outliers and see the shape of the data. A popular Python library for visualization is Matplotlib and pyplot, a collection of functions that make Matplotlib function like MATLAB. Using Matplotlib.pyplot, you can create a number of plots, from box plots to histograms, to begin to paint a picture with the data.</p><p>Setting up a basic plot is easy:</p><pre>Import matplotlib.pyplot as plt</pre><pre>%matplotlib inline</pre><p>You write the matplotlib inline to display plots directly in your Jupyter Notebook. For this exercise, we are using data from student test performance at a few high schools.</p><p>Here is an example of how you would plot a bar plot to compare the number of students who were at a passing level vs those who were not.</p><pre>subjects = list(set(scores[&#39;Subject&#39;]))<br>passing = []<br>notpassing = []<br>y = {}<br><strong>for</strong> i <strong>in</strong> subjects:<br>    x = scores[scores[&#39;Subject&#39;] == i]<br>    passing.append(len(x[x[&#39;Passing&#39;] == &#39;Passing&#39;]))<br>    notpassing.append(len(x[x[&#39;Passing&#39;] == &#39;Did not pass&#39;]))<br><br>y[&#39;subjects&#39;] = subjects<br>y[&#39;passing&#39;] = passing<br>y[&#39;notpassing&#39;] = notpassing<br><br>df = pd.DataFrame.from_dict(y)<br>df = df.set_index(&#39;subjects&#39;)<br><br>df.plot(kind=&#39;bar&#39;,stacked=<strong>True</strong>)<br>plt.title(&#39;Count of students passing&#39;)<br>plt.show();</pre><p>And the output would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*b_fawVeDW5GV75QTk2BlVw.png" /><figcaption>Single bar chart</figcaption></figure><p>Now, you could write out a cell of code to make a plot for each of the various comparisons you would like to make. However, that goes against our ideal of doing more for less. This is where the beauty of subplots comes in. With subplots, you are able to generate an array of plots, iterating through data and creating multiple plots at once. This provides insights faster, allowing for deeper insights and more targeted analysis.</p><p>To establish subplots, begin by generating the plot figure and axes. You can also specify the number of rows and columns in the array.</p><pre>Fig, axs = plt.subplots(n_rows, n_columns, figsize=(14,24))</pre><p>You can also specify the figure size when you initialize the plot.</p><p>Following the initialization, specify the space you need between plots using the plt.subplots_adjust() function. Wspace and hspace will determine the space between each of the plots on the sides and header.</p><pre>plt.subplots_adjust(wspace=0.6, hspace=0.6)</pre><p>Next, you would set up your data into manageable lists. I’ve also structured the lists into a dictionary to assist with plot labels.</p><pre>schools = list(set(scores[&#39;High School&#39;]))<br>frpl = list(set(scores[&#39;FRPL&#39;]))<br>gender = list(set(scores[&#39;Gender&#39;]))<br>ethnicity = list(set(scores[&#39;Ethnicity&#39;]))<br><br>metrics = {&#39;High School&#39;: schools, &#39;FRPL&#39;: frpl, &#39;Gender&#39;: gender, <br>           &#39;Ethnicity&#39;: ethnicity}</pre><p>The way I’ve structured my data in this example, I iterate through my dictionary keys and use them to filter down the primary data set, adding the counts of passing and not passing scores to lists. The lists are then fed into a dictionary, which I then use to construct a dataframe. This is where I begin to construct my subplots.</p><p>We can use variables to select the rows and columns of the axis of the subplot. Because I am constructing stacked bar plots, I call two barplots to the same axis position. It is easy enough to then call in the dictionary keys as labels for the plots.</p><p>To generate the tick marks, I have generated a set of the values in each of the variables I am using to split the data. I then specify the number of tick marks and the labels by passing the set into the plt.set_xticks and plt.set_xticklabels calls. You can also adjust the font size and rotation for the tick marks.</p><p>I have also chosen to plot the percentage of students passing the tests in each of the demographic groups.</p><pre>a = 0<br><br><strong>for</strong> i, j <strong>in</strong> enumerate(metrics.keys()):<br>    passing = []<br>    notpassing = []<br>    y = {}<br>    z = metrics[j]<br>    <strong>for</strong> k <strong>in</strong> z:<br>        x = scores[scores[j] == k]<br>        passing.append(len(x[x[&#39;Passing&#39;] == &#39;Passing&#39;]))<br>        notpassing.append(len(x[x[&#39;Passing&#39;] == &#39;Did not pass&#39;]))<br>        <br>    y[j] = z<br>    y[&#39;passing&#39;] = passing<br>    y[&#39;notpassing&#39;] = notpassing<br>    <br>    N = len(passing)<br>    ind = np.arange(N)<br>    ticks = list(range(N))<br><br>    df = pd.DataFrame.from_dict(y)<br>    df = df.set_index(j)<br><br>    axs[a,0].bar(ind, df[&#39;passing&#39;], width=0.6)<br>    axs[a,0].bar(ind, df[&#39;notpassing&#39;], bottom=df.passing,width=0.6)<br>    axs[a,0].set_title(&#39;Count of test takers&#39;)<br>    axs[a,0].set_xlabel(j)<br>    axs[a,0].set_xticks(ticks)<br>    axs[a,0].set_xticklabels(z, fontdict = {&#39;rotation&#39;: 45})<br>    <br>    df[&#39;passrate&#39;] = (df[&#39;passing&#39;] / (df[&#39;passing&#39;] + df[&#39;notpassing&#39;]))*100<br>    <br>    axs[a,1].bar(ind, df[&#39;passrate&#39;], width=0.6)<br>    axs[a,1].set_title(&#39;Percentage of students passing&#39;)<br>    axs[a,1].set_xlabel(j)<br>    axs[a,1].set_xticks(ticks)<br>    axs[a,1].set_xticklabels(z, fontdict = {&#39;rotation&#39;: 45})<br>    a += 1</pre><p>Here is the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*oS_yWNMcqRCV55FcoNICag.png" /></figure><p>So with one block of code, I’m able to produce 8 graphs; 4 pairs of informative graphs that tell me quite a bit about the information I’m working with!</p><p>Hopefully this provides you with a good introduction to subplots in matplotlib.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df357d23bd72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian vs Frequentist approach to finding probability]]></title>
            <link>https://zachary-a-zazueta.medium.com/bayesian-vs-frequentist-approach-to-finding-probability-a7cd04657800?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7cd04657800</guid>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Fri, 09 Oct 2020 23:20:26 GMT</pubDate>
            <atom:updated>2020-10-09T23:20:26.280Z</atom:updated>
            <content:encoded><![CDATA[<p>The comparison between the way things are and the way things ought to be is one that is made frequently. Good ice cream should be inexpensive, if not a free, public good, but oftentimes it is quite expensive. Exercise should be something we all strive for — it makes us feel good and there are health benefits; however, it is often very difficult to find the motivation to make it to the gym. The conflict between an ideal and an observed reality is present in everyday life, and statistics is no exception. Such conflict exists in the interpretation of probability, in the comparison between the Bayesian approach and the Frequentist approach.</p><p>These two approaches or philosophies are the two arms of inferential statistics, the branch of statistics that allows generalizations to be made about entire populations of data based on observations of some amount of sample data. This breaks from descriptive statistics, which are information points about some tangible data set. Inferential methods allow for assumptions to be made about much larger sets of information without having to observe them all, but rather by taking information from a number of samples and <em>inferring </em>an outcome. An example would be estimating the average height of all adult women in South America by taking representative samples from a number of regions in the continent without having to measure all females over the age of 18. As you can imagine, this makes life considerably easier.</p><p><strong>Frequentist vs. Bayesian</strong></p><p>Frequentists rely on what is observed — “what is” — when determining the probability of an event. To a frequentist, probability is related to what is observed in the frequency of repeated random events. While rooting in long-term frequencies of information presented in repeated samples, frequentists produce p-values that tell us the probability of a false positive result based on data from an experiment.</p><p>For Bayesians, probability is a more general concept, which allows non-repeatable events to have probabilities assigned to them. Probability is more related to the certainty or uncertainty of events; Bayesians root in the degrees of belief and logical support for the probability of a given outcome — the “what should be”. Bayesians would start with an estimated and prior probability — information gained from past observations — and then apply Bayes theorem to narrow the probability distribution around the parameter. Use of prior probabilities is the main objection of frequentists towards Bayesians. For example, a bad prior probability can have significant negative impacts on the model</p><p>As Jake VanderPlas stated in his 2014 <a href="https://www.youtube.com/watch?v=KhAUfqhLakw">talk</a> at the SciPy conference — frequentists consider models and the parameters for those models as fixed, while data varies around the models. Bayesians see data as being fixed while the models and their parameters shift according to the data. The dichotomy between the methodologies was further explained when he presented these two views of how uncertainties in findings are explained:</p><blockquote>“If this experiment is repeated many times, in 95% of these cases the computed confidence interval will contain the true <strong><em>θ</em></strong>” — Frequentist</blockquote><blockquote>“Given our observed data, there is a 95% probability that the value of <strong><em>θ</em></strong> lies within the credible region” — Bayesians</blockquote><p>What is present here is the difference in what varies vs what remains fixed. We can see that the frequentist allows the interval to vary while for Bayesians, it remains constant (the credible region). Additionally, the computed value of <strong><em>θ </em></strong>varies for Bayesians while holding fixed for the frequentist.</p><p>While the debate between the two philosophies rages on, it is important to separate the method of thought from the actual method of practice. WIth big data, we can begin to understand what prior statistics might be for a data set where we can then estimate prior quantities frequentistically to carry out Bayes calculations on the data.</p><p>Hopefully the above distinctions can help you understand that the following is true. The Bayesian will compute a fixed region that is considered to be credible. They will then determine the probability that a parameter will fit inside that interval. A frequentist will focus on a model parameter and then determine how well the test will generate an interval in which the parameter fits.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7cd04657800" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data transformation in ML — Standardization vs Normalization]]></title>
            <link>https://zachary-a-zazueta.medium.com/data-transformation-in-ml-standardization-vs-normalization-70ba26de9060?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/70ba26de9060</guid>
            <category><![CDATA[feature-scaling]]></category>
            <category><![CDATA[standardization]]></category>
            <category><![CDATA[normalization]]></category>
            <category><![CDATA[data-scaling]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Thu, 01 Oct 2020 13:23:53 GMT</pubDate>
            <atom:updated>2020-10-01T13:23:53.953Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data transformation in ML — Standardization vs Normalization</h3><p>Many things in life come in a variety of shapes, sizes, flavors, etc. It is this variety that is said to be “the spice of life”. Unfortunately, data scientists often have to save the variety for after hours and get the data they are working with to become rather similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*IJZ9GyucdKwVtrXomPyKXQ.png" /><figcaption>credit to Analytics Vidhya</figcaption></figure><p>When working with real world data sets, it is common to find yourself working with a mix of continuous variables which span a wide range of information. If you are looking at a data set for student’s academic scores, the range of points for each assignment, quiz, or test might vary — a quiz might be scored out of 20 points while a test might be scored out of 100 points. If you are comparing automobiles, you might be comparing MPGs (on a range of 15–45 mpgs) to top speeds (80–130 mphs) to driving range on a full tank of gas (280–410 miles).</p><p>There might also be a difference in the scale of some of the data / data with heavy skew… For example, exploring sale prices of homes in given city might show that 80% of the homes are valued under $1 million with the remaining 20% ranging between $1–10 million. And those prices might be next to a feature for lot size, where the data describe sizes between 0.25 acres — 10 acres.</p><p>Having data that are on such widely different scales or with such variance that are representing such different things can impact predictive machine learning models. Algorithms see the numbers and would assume that greater values insinuate greater influence on the outcome. Knowing how to manipulate these data in such a way that comparing them and placing them into models in ways that the models gain the most predictive power is called ‘feature scaling’ and is a crucial step — some would argue the most crucial step — in the data pre-processing step of the data science cycle.</p><p><strong>Methods</strong></p><p>Two approaches to feature scaling are <strong>standardization </strong>and <strong>normalization</strong>. Standardization is the practice of making features look more or less normally distributed. It shifts values to where they are centered around the mean with the mean set to 0 and where the distribution of the rescaled data have a unit standard deviation. Normalization is the process to shift and rescale data so the data range between [0,1]. By scaling data before employing a distance based algorithm, all features contribute equally to the result. When using gradient descent algorithms, features on a similar scale allow for a faster convergence towards the minima.</p><p><strong>Scaling Methods in Python</strong></p><p>Scikit-Learn’s preprocessing library contains a number of transformer methods:</p><p><strong>StandardScaler</strong></p><ul><li>Removes mean and scale data to unit variance</li><li>Cannot guarantee balanced feature scales in the presence of outliers</li></ul><p><strong>MinMaxScaler</strong></p><ul><li>Rescales data so all values are in a 0–1 range</li><li>Also sensitive to outliers, as inliers are often squeezed into a small range</li></ul><p><strong>RobustScaler</strong></p><ul><li>Centering and scaling are based on percentiles — median is removed and data are scaled according to interquantile range</li><li>Median and IQR are robust to outliers, as opposed to other measures like min, max, mean, standard deviation</li></ul><p><strong>PowerTransformer</strong></p><ul><li>Implements the Yeo-Johnson and Box-Cox transforms to make the data more Gaussian-like by finding optimal scaling factors to stabilize variance and minimize skewness through MLE.</li></ul><p><strong>Normalizer</strong></p><ul><li>Rescales the variable so every sample has unit norm independent of the distribution of samples.</li></ul><p>When you apply these methods to training and testing sets, you would first call a Transformer<strong>.fit</strong>(data) method to compute the pertinent information (min, max, mean, std) from the data to be used for later scaling. Then you would use the Transformer<strong>.transform</strong>(data) to scale the training and testing data sets at the appropriate times.</p><p><strong>Where to scale in ML</strong></p><p>On a high level, here are some tips on when to apply scaling. For machine learning methods which use distance measurements in their algorithms scaling is key. <strong>K-Means</strong> uses Euclidean distance measure — <strong>K-Prototypes</strong> would then also require scaling because it combines K-Means and K-Modes. <strong>K-Nearest Neighbors</strong> and <strong>SVMs</strong> also use distance between data points to determine similarity, so they should be scaled. <strong>Principal Component Analysis (PCA) </strong>works to select features with maximum variance, so it is important to standardize feature variance. Algorithms like neural networks, and linear and logistic regression, <strong>which use gradient descent as an optimization technique,</strong> require data to be scaled because the feature value X is present in the gradient descent formula.</p><p><strong>Tree-based models</strong> and <strong>Naive Bayes</strong> are two examples where the model is fairly insensitive to feature scaling because they are not distance based. Tree based models split nodes based on single features and the split on a feature is not influenced by other features. In contrast to the rule based tree algorithms, Naive Bayes is designed to handle differences in data scale and variance and assigns weights to features accordingly.</p><p><strong>Selecting a transformer</strong></p><p>When data does not follow a Gaussian distribution, <strong>normalization</strong> is typically the route to take. If the data are Gaussian, <strong>standardization </strong>is likely the path. If outliers are present in the data, robust transformers or scalers (like the RobustScaler) are most appropriate.</p><p>Hopefully this serves as a good starting point for knowing why we sometimes need to scale data, when we should apply these methods, and what some of the methods for these transformations are and how they operate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70ba26de9060" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting to SQL Server in Python]]></title>
            <link>https://zachary-a-zazueta.medium.com/connecting-to-sql-server-in-python-bc3c6b7aafb7?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc3c6b7aafb7</guid>
            <category><![CDATA[pyodbc]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[sqlite3]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Sat, 26 Sep 2020 15:43:21 GMT</pubDate>
            <atom:updated>2020-09-26T15:47:50.817Z</atom:updated>
            <content:encoded><![CDATA[<p>“SQL is an important tool for any data scientist” — is my entry for the understatement of the year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_05Cono50j85GkbCk4OzQ.jpeg" /></figure><p>As information jockeys, data scientists use SQL to query the data they will need for analysis from established databases. Depending on the demands of your position as a data scientist, you might even be asked to help construct a database from your employer’s data sources, which SQL is also helpful for. Being able to select the information that is pertinent to the current project at the start of the project is a simple efficiency to learn and will save time on the project.</p><p>Early on in my journey to learn data science, I wanted to be able to leverage Python with data pulled from a Microsoft SQL Server Database and after learning about the use of sqlite3 in Python, I figured out how to connect directly to streamline my work, pulling data directly into a Jupyter notebook. Continue reading if you would like to learn how to do this!</p><p>If you’re not familiar with the sqlite3 Python module, it is a module which provides a SQL interface to SQLite databases. There is plenty of literature out on how to use SQLite in Python, but I found this blog post to be most <a href="https://towardsdatascience.com/python-sqlite-tutorial-the-ultimate-guide-fdcb8d7a4f30">helpful</a>.</p><p>If you aren’t building your own database with SQLite and you want to connect to a database that is already established, you can use the pyodbc module. Pyodbc allows a user to connect with a DataBase Management System (DBMS) database using the ODBC driver. A quick pip install should get you up and running, but you can find full installation docs <a href="https://github.com/mkleehammer/pyodbc/wiki/Install">here</a>. Once you have the pyodbc installed (full documentation for pyodbc can be found <a href="https://github.com/mkleehammer/pyodbc/wiki">here</a>) you will need to install the ODBC driver for the DBMS you want to connect to — a quick Google search should direct you to the appropriate download.</p><p>Now I’ll walk through an example of writing a connection to a Microsoft SQL Server DB and extracting a query.</p><p>Similar to sqlite3 module, to connect to the database you will need to establish a connection object to represent the database. Pyodbc passes an ODBC connection string to the local driver manager which in turn calls the relevant database driver which in turn calls the database to request the connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*0h3DxN9Q-a-lil-kedAj1Q.png" /><figcaption>Generate a connection object</figcaption></figure><p>For Microsoft SQL Server 2017, there are a number of connection strings to select from depending on whether you are connecting to a db with standard security, connecting to a trusted connection, or if you want to enable Multiple Active Result Sets (MARS). I used the following to connect to a trusted connection:</p><blockquote>‘Driver={ODBC Driver 13 for SQL Server}; Server=MyServerAddress; Database= myDataBase; Trusted_Connection=yes;’</blockquote><p>Find other connection strings configured for numerous DBMSs <a href="https://www.connectionstrings.com/">here</a>.</p><p>Once you have your connection established, there are two methods you can use to extract a query. The first way is to establish a cursor object. A cursor object enables the traversal over the rows of a result set pulled from a database. A cursor will allow you to process individual rows in the set by managing the context of fetch operations. If the .cursor() method is called without specified class and index arguments, it creates a DB-API style cursor which can use numerous operations, like find(), execute(), fetchone(), and fetchmany().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*8B_sbjMuewvY8eX0PMraDw.png" /><figcaption>Fetch all rows to dataframe</figcaption></figure><p>As you can see in the above code, you can then select a subset of the query to pull — fetchone() would pull just one row, fetchall() selects all rows, and fetchmany(x) pulls the next x remaining rows and returns them as a list of tuples.</p><p>The second method is to use pandas.read_sql function, which is a wrapper function for the read_sql_table and read_sql_query calls shown above. This function allows the user to input a written query or database table name and the connection string and feed the result directly into a dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*pFqT8quzR-cuOqTdfKeXtw.png" /><figcaption>Read a query straight into a dataframe</figcaption></figure><p>And just like that you have your data ready to use in a dataframe!</p><p>Some helpful tips to keep in mind:</p><ul><li>Queries can be written in python exactly as they would be in a DBMS.</li><li>Convert data types directly in the query to reduce conversion operations.</li><li>When writing long queries, wrap the query string in triple quote marks — this allows strings to span multiple lines, so you can indent for readability.</li></ul><p>Hopefully this helps you streamline your data sourcing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc3c6b7aafb7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chi-square Test for Feature Reduction in Python]]></title>
            <link>https://zachary-a-zazueta.medium.com/chi-square-test-for-feature-reduction-in-python-32ac7fb7078f?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/32ac7fb7078f</guid>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[chi-square-test]]></category>
            <category><![CDATA[scipy]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 00:01:56 GMT</pubDate>
            <atom:updated>2020-09-16T00:01:56.275Z</atom:updated>
            <content:encoded><![CDATA[<p>“There are two kinds of people in this world…”</p><p>Are words we’ve all heard before, typically from an elder relative trying to make sense of some bigger picture for you. As cliche as the statement is, demographic information — identifiers which tell us something about the members of a population — are often split into just a few distinct buckets, allowing populations to be split. Gender, car ownership, job industry, being a parent or not, current employment status — these are just a few examples of the myriad data points that can be used to segment groups of people. Choices people make over the course of their lives will deposit them into a number of buckets defined by the data scientist interested in understanding their patterns and habits in the hopes of retaining them as a customer, delivering them relevant product ads, or targeting them for the most beneficial information. As people grow and change, the buckets they fall into and how they are categorized will change.</p><p>Data scientists are often curious about how the various categorizations — the independent demographic features of their observed sample — impact the dependent target variable, thereby triggering an outcome. Do people with siblings tend to be dog owners more than they are cat owners? Do people with different types of Master’s degrees have a different preference for vacation destination or are the observed number of MFAs vs MBAs flocking to warmer climates in the winter similar? Questions like these can be answered using the Pearson Chi-squared test.</p><p>The Pearson Chi-squared test (often just “Chi-squared”) is a statistical hypothesis test used to determine if there is a significant difference between the observed and expected distributions in one or more categories of a contingency table.</p><p><strong>Contingency Tables</strong></p><p>Contingency tables are a type of frequency distribution tables which are used to summarize a relationship between two or more variables. A contingency table is defined such that there would be a different number of observations across each row or population but a similar proportion across each column or group. A key assumption of the Chi-squared test is that each observation (row) is only contributing data to one cell in the contingency table. For example, if we had 50 participants in a study where they each tried to solve 2 crossword puzzles, it would not be appropriate to conduct a Chi-square test on frequency data that looked like the below because each participant is contributing to two cells:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*NRL4dpB_-LpeujRBY6vcAw.png" /><figcaption>Contingency Table</figcaption></figure><p>If we were studying 100 participants who were randomly assigned puzzle 1 or 2, we could then use Chi-squared to determine whether there was independence between puzzle assignment and solving rate.</p><p>Other rules for the Chi-square test are that the total number of observations in the contingency table should be greater than 20. Also, if the count of a cell in the contingency table is fewer than 5, the <a href="https://www.statisticshowto.com/what-is-the-yates-correction/">Yates correction</a> for continuity should be used. The chi-squared test is used for categorical data but can be applied to a continuous variable after the data is binned.</p><p><strong>Pearson’s Chi Squared Test</strong></p><p>The Pearson Chi-squared test calculates a test statistic and p-value that determine whether there is evidence to reject the null hypothesis that there is no difference in the observed and expected frequencies of two or more variables. If evidence is found to reject the null — the observed and expected frequencies of the variables are not similar — then there is evidence that the variables are dependent on one another.</p><p>To run the chi-squared test in Python, find the chi2_contingency method in the scipy.stats library. The method takes in a contingency table, a parameter for the Yates correction, and a specification on which statistic the test should calculate. It outputs the test statistic, the p-value, the degrees of freedom, and a table of the expected values of the distribution.</p><p>Interpreting the output is relatively straightforward — if the test statistic is greater than or equal to the critical value then we have evidence to reject the null hypothesis. For user knowledge, the critical value for a chi-squared distribution can be calculated in Python using the scipy.stats.chi2.ppf (percentage point function) method by inputting the ppf = 0.95 (to cover 95% of the observations from the selected sample) and degrees of freedom. DF can be calculated using the formula:</p><p><strong>DF = (n_rows - 1) * (n_columns - 1)</strong></p><p>Referring to the number of rows and columns of the contingency table.</p><p>The test results can also be interpreted using the p-value — if the calculated p-value is less than or equal to the selected level of significance (alpha), there is evidence to reject the null hypothesis. If p is greater than alpha, the null cannot be rejected with the selected samples.</p><p><strong>Example: ML classification for Customer segmentation</strong></p><p>I’ll now provide a brief example of using Chi-squared to reduce features in a data set prior to determining feature importance and fitting a model. Customer demographics can be observed in this data set from a <a href="https://www.kaggle.com/blastchar/telco-customer-churn">Kaggle competition</a> containing information about customers of a telecommunications company. Many of the identifying customer data were categorical — things like whether or not the customer had multiple phone lines or whether they had dependents. The goal of the project was to classify customers who churned — whether or not they terminated their service in the last month.</p><p>After removing a few features (due to multicollinearity) during the data cleaning phase, I still had 15 features, which was pushing towards the higher side for the K-Prototypes cluster analysis I wanted to conduct. While I plan to use PCA to reduce features, using Chi-squared in advance could potentially whittle down the list first, with the aim of my feature selection being to select features that the response was highly dependent on.</p><p>After conducting quick visual analysis of the count distribution of the variables, I was ready to use the Chi-square method to dig deeper. Two examples are as follows:</p><p>I first compared frequency distribution for the gender category against churn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*M2JlCPKZRD-BazVJDauS5g.png" /><figcaption>Generated with seaborn countplot</figcaption></figure><p>It appears that the number of males and females who had a Churn value of 0 (did not churn) and 1 (did churn) were approximately equal. It is not likely that we will find evidence to reject a Chi-squared test, but we explore it anyway. The &lt;crosstab&gt; method in pandas allows a user to generate a contingency table from two features of a data frame. I after generating the table, I can use the chi2_contingency test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*2RLdt7tPMCvgZO4-8Uu3RA.png" /></figure><p>Again, outputs are as follows:</p><ul><li>Test statistic — 0.475</li><li>P-value — 0.4904</li><li>Degrees of Freedom — 1</li><li>Expected values — [[2,557.27, 2,605.73],[ 925.73, 943.27]]</li></ul><p>EVs for the expected values table are calculated using the formula:</p><p><strong>E = n * p</strong></p><p>Where n is the sample size and p is the P(AnB) = P(A) * P(B).</p><p>For Females who did not churn(0), we have</p><p>E = 7,032 * ((3,483/7,032) * (5,163/7,032)) = 7,032 * 0.3637 = <strong>2,557.27</strong></p><p>Inputting ppf = 0.95 and DF = 1 into the chi2.ppf(0.95, 1), we get a critical value of 3.84. The test statistic and p-value are lower than the critical value and alpha respectively, meaning we do not have evidence to reject the null and the features are independent of one another.</p><p>A second feature I considered was the customers’ Payment Method. Customers could pay by one of four methods — electronic check, mailed check, bank transfer, or credit card. The countplot against churn was:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*v1G1DYi2zDral2PGVyh3zg.png" /></figure><p>At first glance, it appears that customers who did not churn had similar frequency for all four methods while the customers who left were using electronic checks at about 3–4 times the rate as other payment methods. This warrants additional investigation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*9-54iMTqUuhPUKhzSwqoHQ.png" /></figure><p>The outputs:</p><ul><li>Test statistic — 645.43</li><li>P-value — 1.43 * 10^-139</li><li>DF — 3</li><li>Expected values — [[1,132.16, 1,116.74, 1,736.42, 1,177.68], [ 409.84, 404.26, 628.58, 426.32]]</li></ul><p>As we can see, there are significant differences between the values in the generated crosstab and the expected values table. The test statistic and the p-value are significant, providing evidence to reject the null hypothesis and say there is a difference between the observed and expected frequencies of payment method used by customers.</p><p>The takeaway is that I will not include gender in the data I consider at later stages in my modeling process, but will be sure to include payment method because we learn something from the dependence between payment method and churn.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32ac7fb7078f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parametric vs non-parametric statistical tests in Python]]></title>
            <link>https://zachary-a-zazueta.medium.com/parametric-vs-non-parametric-statistical-tests-in-python-9c7ab48e954a?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c7ab48e954a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[parametric-tests]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[statistical-test]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Sun, 06 Sep 2020 22:57:54 GMT</pubDate>
            <atom:updated>2020-09-07T16:09:37.346Z</atom:updated>
            <content:encoded><![CDATA[<p>Once one has a good understanding of <a href="https://medium.com/@zachary.a.zazueta/levels-of-data-measurement-2ba7d858904d">the data they have to work</a> with, they next need to decide what they aim to answer with this information. Understanding the problem at hand is part of the Business Understanding step in the Data Science Process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYbymHifAk7aQ1pHm_IdMQ.png" /><figcaption>The Data Science Process</figcaption></figure><p>A business question with a data solution can often be posed as a hypothesis. For example “Is there a difference in the customer conversion rate between our old website design and a proposed new layout?” Having a hypothesis to test is a must-have before statistical testing can occur.</p><p>Two types of hypotheses are exploratory and confirmatory; as the names might suggest, exploratory analysis seeks to uncover the “why” and dig into the data while confirmatory hypotheses are more applicable when you have a pretty good idea of what is going on with the data and need evidence to support thinking. It is important to decide <em>a priori </em>which of your hypotheses belong to these categories. It has been <a href="https://blog.efpsa.org/2019/11/20/exploratory-and-confirmatory-hypothesis-testing/">argued</a> that limiting exploratory hypothesis testing can help to increase certainty in results.</p><p>Once the hypothesis has been determined, the next question to answer is “am I comparing the mean or the median of two groups?”. Parametric tests will compare group means, while non-parametric tests compare group medians. A common misconception is that the decision rests solely on whether the data is normally distributed or not, especially when there is a smaller sample size and distribution of the data can matter significantly. Other factors should also be considered.</p><p>Parametric tests are widely regarded as handling data that is normally distributed — data with a Gaussian distribution — well. However, parametric tests also:</p><ul><li>Work well with skewed and non-normal distributions.</li><li>Perform well when the spread of each group is different or the groups have different amounts of variability.</li><li>Typically have more statistical power than non-parametric tests.</li></ul><p>If sample size is sufficiently large and <em>group mean</em> is the preferred measure of central tendency, parametric tests are the way to go.</p><p>If <em>group median</em> is the preferred measure of central tendency for the data, go with non-parametric tests regardless of sample size. Non-parametric tests are great for comparing data that is prone to outliers, like salary. They are also useful for data with small sample size and/or non-normal, and are especially useful for working with ordinal or ranked data. You should also stick with non-parametric tests for ordinal and ranked data.</p><p>Some of the most commonly used statistical parametric tests and their non-parametric counterparts are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*dT939VBGfjuwUppsbkmSrw.png" /><figcaption>Where n = sample size</figcaption></figure><p>There are also tests which compare correlation — looking for associations between variables e.g. Pearson, Spearman, Chi-Squared — and regression tests — seeing if a change in one or more independent variables will predict the change in a dependent variable e.g. simple &amp; multiple regression.</p><p><strong>A quick overview of when you might use each of the above tests:</strong></p><p>The <strong>Paired t test</strong> is used when you are looking at one population sample with a before and after score or result. This could be comparing a classroom of students beginning of year proficiency on reading to their end of year proficiency to determine if there was growth or decrease in understanding. The non-parametric counterpart is the <strong>Wilcoxon Signed Rank test</strong>, which can be used to determine whether two dependent samples were selected from populations having the same distribution and takes into account the magnitude and direction of the difference.</p><p>The <strong>Unpaired t test</strong>, also widely known as the 2-sample or independent t test, is used to compare two samples from different, unrelated groups to determine if there is a difference in the group means. The <strong>Mann-Whitney U test</strong>, also known as the Wilcoxon rank-sum test, is similar to the Wilcoxon Signed Rank test but measures the magnitude and direction of the difference between independent samples.</p><p>Finally, the <strong>One-way ANalysis Of VAriance (ANOVA)</strong> is used to determine difference in group means for two or more groups where there is one independent variable with at least two distinct levels. An example of this would be predicting the weight of a dog based on breed given a set of dogs of different breeds. The <strong>Kruskal Wallis test</strong>, an extension of the Mann-Whitney U test for comparing two groups, can be used to compare medians of multiple groups where the distribution of residuals is assumed to not be normal.</p><p>There are certain assumptions that are made for data that is to be analyzed using parametric tests. The four assumptions are that 1) the data is normally distributed (or that difference between the samples is normally distributed for paired test), 2) there is similarity in variance in the data, 3) sample values are numeric and continuous, and 4) that sample observations are independent of each other. The below functions from the statsmodels.api module allow us to explore these assumptions during data exploration.</p><pre>statsmodels.api.graphics.plot_regress_exog()</pre><pre>statsmodels.api.graphics.qqplot()</pre><p>Let’s examine how to call up these tests in Python 3. First, the parametric data:</p><p>The <strong>stats</strong> module is a great resource for statistical tests.</p><p>Paired t test is</p><pre>scipy.stats.ttest_rel</pre><p>Unpaired t test is</p><pre>scipy.stats.ttest_ind</pre><ul><li>For ttest_rel and ttest_ind, the P-value in the output measures an alternative hypothesis that 𝜇0 != 𝜇1; for one-sided hypothesis, e.g. 𝜇0 &gt; 𝜇1, divide p by 2 and if p/2 &lt; alpha (usually 0.05).</li></ul><p>One-way ANOVA is</p><pre>scipy.stats.f_oneway</pre><ul><li>A significant P-value signals that there is a difference between some of the groups, but additional testing is needed to determine where the difference lies.</li></ul><p><strong>For the non-parametric data:</strong></p><p>Wilcoxon Signed Rank is</p><pre>scipy.stats.wilcoxon</pre><p>Wilcoxon Rank-Sum is</p><pre>scipy.stats.ranksums</pre><ul><li>Signed rank and rank-sum tests should be used for continuous distributions.</li></ul><p>Kruskal Wallis is:</p><pre>scipy.stats.kruskal(group1, group2, group3)</pre><ul><li>Similar to ANOVA, rejection of the null hypothesis does not tell us which of the groups is different, so additional post hoc group comparison is necessary.</li></ul><p>In terms of takeaways, it is never good practice to conclude on the results of one test, but significant findings should lead to additional investigation. Bonferroni corrections, a topic for another time, can be used to reduce spurious positives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c7ab48e954a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Levels of Data Measurement]]></title>
            <link>https://zachary-a-zazueta.medium.com/levels-of-data-measurement-2ba7d858904d?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ba7d858904d</guid>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Sun, 30 Aug 2020 21:22:00 GMT</pubDate>
            <atom:updated>2020-08-30T21:22:00.224Z</atom:updated>
            <content:encoded><![CDATA[<p>It is widely reported that over 80% of a data scientist’s time is spent cleaning and engineering data. Great effort is put into preparing the information that will feed a data scientist’s models. Including irrelevant information or messy data in the modeling cycle can lead to models that are inaccurate or show false insights.</p><p>As such, one of the first steps of data cleaning is understanding what features or attributes will be available for the work and what type of attribute they will be. Data can be measured on four main scales: Interval, Ratio, Nominal, and Ordinal. Knowing the difference in these scales allows the data scientist to correctly structure the available data and informs them what statistical methods can be applied to said data.</p><p>Data that can be classified on the Interval and Ratio scales will be numeric. Interval and Ratio data can be both continuous and discrete. With these variables, defined values separate units of measure encountered in the data set. The key difference between these data measure types is that Interval data have no “true zero”.</p><p>Interval data are ordered data. There is also a known distance between data points that has credible meaning. An example of this is time — we know that 30 minutes separate 1:00 and 1:30; similarly, 30 minutes also separates 1:30 and 2:00, meeting the consistent values separating units of measure. It is also widely agreed that 1:00 comes before 1:30, satisfying order. However, there is no absolute zero to a variable like time. The lack of a “true zero” limits the arithmetic operations which can be applied to interval data to addition and subtraction — we can measure the number of days since a customer last made a purchase, or the number of days we’ve been dealing with the coronavirus pandemic (at least an eternity, right?), but we would not divide November by March.</p><p>Another example of an interval variable is a student’s SAT scores; numbers between 0 to 200 are not used when the College Board scales raw scores. Now, interval data can contain a zero — it is just that “0” does not carry as much meaning as “true zero”. Temperature is a great example of this. Considering the interval variable of temperature in Fahrenheit — we can have 0o F, but that does not mean there is no temperature, it just means you better be wearing a winter jacket.</p><p>When you have ordered numeric data with a clear definition of zero, you are dealing with Ratio data. Classic examples of ratio data are height/weight and price. If a person has zero height or weight, they don’t exist; a customer can’t pay less than $0.00 for an item at a store.</p><p>Measures of central tendency — mean, median, mode, and standard deviation — can be computed for these numerical data measures, but it is only with the presence of the true zero in ratio measures that a wide range of descriptive and inferential statistics can be applied. Interval and Ratio levels of measurement are considered to be much more exact than their qualitative counterparts.</p><p>Qualitative data are either nominal or ordinal. Qualitative data that are ordered are considered Ordinal level and those that are unordered are considered Nominal level.</p><p>Ordinal data are usually numbered to represent rank or order in a list, but the numbers typically reflect opinion or observation and are not mathematical measures. Ordinal data are often seen as ranked responses, like from a survey. Unlike interval data, the difference between the unit responses is not necessarily uniform as gaps between non-numeric concepts can vary — e.g. the cognitive difference between neutral and agree is potentially wider than the distance between agree and strongly agree.</p><p>Ranked placement in a race is another strong example of how ordinal data lack standard unit variance. The difference between 1st and 2nd place might be a matter of 1/10th of a second while the difference between 2nd and 3rd place could be 4/10ths of a second — the inconsistency between placing makes the division between placements less meaningful.</p><p>Because there is this variance in measures, there is only so much information that can be extracted from ordinal data. Central tendency can be captured through mode and median, but a purist will insist that a mean cannot be defined from an ordinal set. There are some measures that can be taken during data collection to maximize information gain for ordinal data — e.g. designing a customer satisfaction survey that aligns to the Likert scale — but as the data shift away from continuous numerical data, its usefulness can be debated.</p><p>Nominal measures are often considered to be the lowest level of data classification as they provide the least information. Nominal data is often seen in binaries (sale was made online or not), categories (a customer’s identified race), and sets of things (e.g. types of tomatoes — cherry, Roma, San Marzano…). Nominal data can have numbers assigned to them — e.g. 1 for Female, 2 for Male, 3 for Other, 4 for Unspecified — but unlike with ordinal data, the numbers do not reflect that one category is better than another. Mode is the measure of central tendency for nominal data; other quantitative descriptions are not appropriate for measuring these unordered categorical features.</p><p>Nominal data is interesting in that it has a few subcategories. One subcategory is “nominal with order” — this describes data that has some order but is not ranked — hot/warm/cold. This is difficult to separate from ordinal. Then there is “nominal without order” — this would be something like eye color or race where there is grouping into unique categories. Finally, there are “dichotomous” nominal data. These data are further broken down into either binary data or discrete or continuous dichotomous variables. Binary variables are variables assigned either a 0 or 1, e.g. Heads(0) or Tails(1) for a coin flip. Discrete dichotomous variables are where there is no possible outcome other than one of two options, e.g. whether <a href="https://en.wikipedia.org/wiki/The_dress">the dress</a> was blue and black or white and gold. Continuous dichotomous variables represent outcomes where there are possibilities in between, e.g. if a house is within 5 miles of the town center it is considered to be in a metropolitan area, but if it is more than 5 miles from town center, it is listed as a rural property.</p><p>While numeric data can have arithmetic measures applied, measurements for nominal and ordinal data types are largely reduced to such measures as proportionality and frequency of distribution of the attributes they inform us on. Even with this reduced capacity, there are tests that can be applied if the data scientist has a sound understanding of the data they are working with. I will aim to dive into that in my next post!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ba7d858904d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hypertune your process before you hypertune your parameters:]]></title>
            <link>https://zachary-a-zazueta.medium.com/hypertune-your-process-before-you-hypertune-your-parameters-8169d504dcc8?source=rss-8b9b7ba80485------2</link>
            <guid isPermaLink="false">https://medium.com/p/8169d504dcc8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[pycharm]]></category>
            <category><![CDATA[dash-app]]></category>
            <dc:creator><![CDATA[Zach Zazueta]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 00:52:34 GMT</pubDate>
            <atom:updated>2020-06-05T00:52:34.206Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why it is important to know what your end goal is…</strong></p><p>When I set out on a two month journey to complete my capstone project for Flatiron School, effectively ending the first chapter of my story to become a data scientist, I knew that I wanted to have something more tangible than a business presentation to show for the work. It is my belief that the best data scientists are those who can put the information they are able to wrangle into the hands of a client or concerned party and allow them to make their own discoveries. Data scientists are gatekeepers of a hidden world of the information that surrounds us, and it is our job to guide people toward key insights.</p><p>With that goal in mind, I knew I wanted to build a Dash app. It sounded like an exciting way to put ML into production and would be my first attempt at bringing ML to life outside of a Jupyter Notebook. I also felt it would work well with the data I was planning to work with — bike rides taken by users of the Citibike bike-share program. My aim was to develop a model that could accurately predict whether there would be bikes available at a given bike station at a given time in the future, so with a little work, I knew I could make a great interactive dashboard in Dash that would allow a user to select a station of their choice and a future date and get a prediction of whether there would be a High, Moderate, or Low availability of bikes for them to use.</p><p>Not having used Dash before, I wanted to collect and clean my data and build out my classification models before really wrapping my head around what this new process would be. My sole focus during the outset was making progress in the data. This ended up being somewhat costly — by not having a clear understanding of what the final product would be, I did not know what my feature selection needed to be. Not knowing what features I needed to use meant that my first round of model tuning was somewhat useless, as I had to re-tune and re-tune the models as my inputs changed.</p><p>The first step to Dash is getting familiar with PyCharm. I had only been coding in Jupyter Notebooks, but PyCharm was widely lauded as being a favorite IDE for Dash. For those who are unfamiliar, PyCharm is an IDE, or Integrated Development Environment, essentially another coding platform which allows you to write and run code. From my basic use of PyCharm, I can say it offers some great features, like high-quality completion and offers support of code written in other languages, specifically SQL and other database languages. That said, it is a hefty program that is memory intensive and can take some time to initially set up.</p><p>Once you have the program installed, you might want to consider setting up a virtual environment. If you already have an environment set up for your Jupyter use, you can clone that in a virtual environment. I plan to write another post on that process soon, as it can be rather confusing. This is an important step because the environment will be what the Project Interpreter references — this is where your installed libraries will be pulled from. You can adjust which virtual environment you reference by navigating to File → Settings → Project Interpreter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUFtJsYZEqLHJ28N-HqFmQ.png" /><figcaption>PyCharm Project Interpreter</figcaption></figure><p>If you don’t have a library you want to work with, it is easy enough to pip install the package or you can find it in a list of available items to bring in directly through PyCharm. Pretty cool stuff!</p><p>Here is where I ran into some trouble. Because I already had my dataset completed and models that had been trained, tuned, and tested, once I was up and running with Dash, I began to conceptualize what an application might look like with the model inputs, NOT what would be realistic for an end user to be able to provide to an application in the interface. Once I “put pen to paper” to map out what information a user would be able to provide, I realized there were a few things I was planning to give my model that would not be known by the user. For example, I had utilized historic ride information to generate a field of the number of bicycles in a station at a given day and time. I was feeding in what the exact bike count at the station was for each event, when really I would need to have a generalized or average count of bikes at a station for each unique station/day of the week/hour of the day which could fill into the model without user input. I also was using weather temperature as an input, which was not something that a user would want to have to provide themselves. Instead, I should have set up some integration to a weather API that would predict hourly weather for the next 7 days. Some of these problems I was able to fix, others I am still working through.</p><p>So here are my takeaways from the project:</p><p>By under thinking at the outset, I ended up building the plane while I was flying it. And the real cost of that was time — time spent writing code to structure the initial round of data that was eventually rewritten; time spent while models iterated through the process of grid search cross validation for the nth-round after I’d made changes; and time spent being more than a little exasperated when I found myself looking at the first build of the application, realizing it didn’t make much sense.</p><p>I hope the below questions will help guide any readers looking to embark on a similar project where a comfortable, consumable product is the aim — I know they are questions I’ll start with next time!</p><p>1. What is the consumer supposed to be able to take away from their interaction with your product?</p><p>a. For me, this was something I knew early on — I wanted to know what the likelihood of a bike being available was at a given date and time.</p><p>2. How much information will the consumer be able to bring to the table their interaction with your application? How much will you as the data scientist need to provide for them?</p><p>a. This is where I jumped the gun on assuming the count of bikes in a station at a given time would be common knowledge, an approach I later changed.</p><p>3. What is the structure of the interface and how will choices made by the user build into an input for your ML model to model.predict(input)?</p><p>a. Working through this problem was probably some of the most rewarding work on the project. It required learning dynamic dropdown input lists in Dash and was a great practice exercise structuring dictionaries.</p><p>You can find a gif of the app in the Readme of the project <a href="https://github.com/zazuetaz/Capstone_project">here </a>(my next step is figuring out how to deploy a Dash app in Heroku!).</p><p>Best of luck in your Dash learnings!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8169d504dcc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>