<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ceyda Akbulut on Medium]]></title>
        <description><![CDATA[Stories by Ceyda Akbulut on Medium]]></description>
        <link>https://medium.com/@cey-akbulut?source=rss-86b005addb18------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cHLA78J5VzhR2q2HMm8gdQ.jpeg</url>
            <title>Stories by Ceyda Akbulut on Medium</title>
            <link>https://medium.com/@cey-akbulut?source=rss-86b005addb18------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:44:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cey-akbulut/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cluster Analysis on Adult Dataset]]></title>
            <link>https://medium.com/geekculture/cluster-analysis-on-adult-dataset-9ac3e8263aff?source=rss-86b005addb18------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ac3e8263aff</guid>
            <category><![CDATA[dbscan]]></category>
            <category><![CDATA[machinelerning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <dc:creator><![CDATA[Ceyda Akbulut]]></dc:creator>
            <pubDate>Sun, 23 Jan 2022 19:23:59 GMT</pubDate>
            <atom:updated>2022-01-24T07:20:47.747Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks to <strong>Hyosang Kim(FU Berlin), Raniyaharini Rajendran(FU Berlin)</strong> for this teamwork!</p><p><strong>Code:</strong> <a href="https://colab.research.google.com/drive/1EdAXpKzQ8GzWKgPQ1EHay80P0AHTms-t#scrollTo=rujaEpoG0T2h"><em>https://colab.research.google.com/drive/1EdAXpKzQ8GzWKgPQ1EHay80P0AHTms-t#scrollTo=rujaEpoG0T2h</em></a></p><p>We will perform cluster analysis on the dataset used for the previous article on Medium. Let`s remember the general info about this set.</p><p>Dataset characteristics: Multivariate</p><p>Attribute characteristics: Categorical, Integer</p><p>Number of Instances(Total): 48842</p><p>Number of Instances(Train):32561</p><p>Number of Instances(Test):16281</p><p>Number of Attributes:14</p><p><strong>Data Preprocessing and Transformation</strong></p><p>We applied the same steps as we did in the last article for data integration, data cleaning, identify or remove outliers…</p><p><strong>Feature Selection</strong></p><p>First, we analyse the type of the attributes. Age field has been considered as a numerical value. Education and education number fields have the same meaning but in different forms.We accept these two fields as ordinal-categorical value. For the remaining columns, see the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*Z7R9hEeBBO0H8Yyg" /></figure><p>Since our dataset consists of features of different types and our method is unsupervised learning, we have to design the way that we will follow. In general, for feature selection there are different methods. If the dataset includes only numerical values PCA can be useful or if the dataset has just categorical values, CA or MCA may be other ways to analyse correspondence between attributes. In our situation, f<a href="https://en.wikipedia.org/wiki/Factor_analysis_of_mixed_data">actor analysis of mixed data </a>(FAMD) could be good. There is very limited information about the FAMD method(prince library). Therefore, we used PCA and MCA methods separately after splitting our dataset to numerical and categorical columns. After PCA, MCA techniques, we had different columns (for PCA we selected 3 components, for MCA 5 components). As we know, after feature selection methods, the dataset should include similar features. So, we have decided to change our methods to select effective features.</p><p>Main idea is detecting the correlations between attributes.Our new methods are correlation matrix for numerical columns and chi-square test for categorical columns.</p><p>The <a href="https://www.statisticssolutions.com/free-resources/directory-of-statistical-analyses/chi-square-2/">Chi-Square test of independence</a> is used to determine if there is a significant relationship between two nominal (categorical) variables. Therefore, we will only consider nominal and binary values for this method. Education and education-num have the same meaning with different forms. Therefore, we will consider only one of them in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/0*IIZ0xborW5LRDi_Y" /></figure><p><em>categorical_main_data=main_data[[‘workclass’,’education’,’marital-status’,’occupation’,’relationship’,’race’,’sex’,’native-country’]]</em></p><p><em>numerical_main_data=main_data[[‘fnlwgt’,’capital-gain’,’capital loss’,’hours-per-week’]]</em></p><p><em>numerical_main_data[‘age’]=pd.to_numeric(main_data[‘age’])</em></p><p><strong>Correlation for numerical values</strong></p><p>Correlation varies should be between -1 and +1. The meaning of this range is:</p><ul><li>-1: perfect negative linear correlation</li><li>+1:perfect positive linear correlation</li><li>No correlation</li></ul><p>Correlation can be derived using following formula:</p><p><em>Correlation = Covariance(X,Y) / SQRT( Var(X)* Var(Y))</em></p><p>Features with high correlation are more linearly dependent and hence have almost the same effect on the dependent variable. So, when two features have high correlation, we can drop one of the two features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*__joScm1EsZEFvfa" /><figcaption><strong>Figure</strong> Correlation matrix with colors</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*tQa_YIOAMnh_blJT" /><figcaption><strong>Figure</strong> Correlation matrix with colors and numerical values</figcaption></figure><p>The correlation coefficients along the diagonal of the table are all equal to 1 because each variable is perfectly correlated with itself and the correlation matrix is perfectly symmetrical. In general,correlation coefficient values below 0.3 are considered to be weak; 0.3–0.7 are moderate; &gt;0.7 are strong. <strong>As we can see, the max value(without diagonal) is 0.119828 and the min value is -0.004473. These correlations are weak. Therefore, we cannot say that there are correlated values.</strong></p><p><strong>Chi-square test for categorical values</strong></p><p>The chi-square independence test is a procedure for testing if two categorical variables are related in some population</p><p>The chi-square test statistic is calculated as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/0*AZIho6lvuySvx9M7" /></figure><p><strong>Null hypothesis:</strong> Selected attributes are independent.</p><p><strong>Alternative hypothesis:</strong> Selected attributes are dependent.</p><p><strong>P &gt; 0.05</strong> is the probability that the null hypothesis is true.</p><p><strong>workclass-marital-status</strong></p><p>We start to analyze with two different categorical columns, workclass and marital-status.</p><p><strong>Chi2:</strong></p><p>1765.0402635587825</p><p><strong>The p-value of the test:</strong></p><p>0.0</p><p><strong>Result: 0 &lt;0.05, alternative hypothesis is correct.</strong></p><p>Null hypothesis: Selected attributes are independent.</p><p>Alternative hypothesis: Selected attributes are dependent.</p><p>P &gt; 0.05 is the probability that the null hypothesis is true.</p><p><strong>workclass-occupation</strong></p><p><strong>Chi2:</strong></p><p>12315.810397342453</p><p><strong>The p-value of the test:</strong></p><p>0.0</p><p><strong>Result: 0 &lt;0.05, alternative hypothesis is correct.</strong></p><p>Null hypothesis: Selected attributes are independent.</p><p>Alternative hypothesis: Selected attributes are dependent.</p><p>P &gt; 0.05 is the probability that the null hypothesis is true.</p><p>We collect all chi-square test results(p-values) in a list as below. p&lt;0.05 situations highlighted by red color. We see that workclass-marital status, workclass-occupation,workclass-occupation,workclass-relationship, marital status-occupation,maritalstatus-relationship,occupation-native-country,occupation-sex,relationship-sex are dependent.In this situation, we have to remove one of the pairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*whmA6ELL2aBLKFn6" /></figure><p>After checking p values, we decided to remove occupation and relationship from our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*JbwffnJA1ohAaZOD" /></figure><p>For the ordinal data, we will continue with the education column. Last version of our dataset:</p><p><strong>age:</strong> continuous.</p><p><strong>workclass:</strong> Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, never-worked.</p><p><strong>fnlwgt:</strong> continuous.</p><p><strong>education:</strong> Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.</p><p><strong>marital-status:</strong> Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.</p><p><strong>race:</strong> White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.</p><p><strong>sex:</strong> Female, Male.</p><p><strong>capital-gain:</strong> continuous.</p><p><strong>capital-loss:</strong> continuous.</p><p><strong>hours-per-week:</strong> continuous.</p><p><strong>native-country:</strong> United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinidad Tobago, Peru, Hong, Holland-Netherlands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/0*3BBqabflvTXjd_vW" /></figure><p><strong>Dimension Reduction</strong></p><p><strong>FAMD</strong></p><p>Before performing clustering, we need to process the dataset into an executable format. Because we have the dataset consisting of both numerical and categorical features, factor analysis with mixed data type (FAMD) would be useful to integrate the types of features so that we can use it for clustering. This is because unlike OHE, it is possible to give the<strong> same weights </strong>to calculate values of projected inertia. As we already have done the feature selection above, the point of this step is processing the dataset.</p><p><strong>Parameter Tuning of FAMD</strong></p><p>n_components: the number of components</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*nsOKM2lEcUDp8cV_" /></figure><p>To calculate the number of principal components used for FAMD, we fit the model with different parameters and observe the explained inertia by the number of components on the graph. As you can see on the above graph, it should be<strong> very high value</strong> for 90% of variance to be explained.</p><p><strong>n_iter:</strong> the number of iterations used for computing SVD</p><p>We did not change the default n_iter since we already have high n_components and increasing the number of iterations would worsen the time complexity.</p><p><strong>Clustering</strong></p><p><strong><em>Choice of algorithms</em></strong></p><p><strong>K-means Clustering :</strong></p><p>The k-means clustering method is an unsupervised ML technique used to identify clusters of data objects for a dataset.</p><p><strong>Parameter Tuning</strong></p><p>The hyper-parameters for k-means clustering as given by Sklearn are: n_clusters, init,n_init, max_iter, tol, verbose, random_state, copy_x, algorithm.</p><ul><li>n_clusters — 3 The optimal number of clusters to be formed</li><li>init — k-means++ selects initial cluster centres for kmeans clustering to speed up convergence.The final results will be the best output of n_init consecutive runs in terms of inertia.</li><li>max_iter — The number of times the algorithm will run for different centroid seeds(default=10)</li><li>tol — Maximum iterations in a single run(default = 300)</li><li>verbose — default=0</li><li>random_state — Centroid initialization for random numbers (Here it is decalred as none)</li><li>copy_x — Typically used inorder to centre the data (Here it is true). If copy_x is True, then the original data is not modified. If False, the original data is modified, and put back before the function returns, but small numerical differences may be introduced by subtracting and then adding the data mean. Note that if the original data is not C-contiguous, a copy will be made even if copy_x is False. If the original data is sparse, but not in CSR format, a copy will be made even if copy_x is False.</li><li>algorithm — Default=auto (for backward compatability). The classical EM-style algorithm generally uses “full”. The “elkan” variation is more efficient on data with well-defined clusters, by using the triangle inequality. However it’s more memory intensive due to the allocation of an extra array of shape (n_samples, n_clusters).</li></ul><p>For now “auto” (kept for backward compatibility) chooses “elkan” but it might change in the future for a better heuristic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fF741pp8s3cY_Y9n" /></figure><p><strong>Evaluation measures</strong></p><p>Here, we have used two measure to evaluate the model performance:</p><ul><li><strong>Intrinsic measures</strong> — Silhouette Score, Calinski Harabasz Score</li></ul><p>To measure a cluster’s fitness within a clustering, we can compute the average silhouette coefficient value of all objects in the cluster. To measure the quality of a clustering, we can use the average silhouette coefficient value of all objects in the data set.</p><p>The score is defined as ratio between the within-cluster dispersion and the between-cluster dispersion.</p><ul><li><strong>Extrinsic measures </strong>— Homogeneity Score, V measure Score</li></ul><p>A clustering result satisfies homogeneity if all of its clusters contain only data points which are members of a single class.It’s bounded between <em>0</em> and <em>1</em>, with low values indicating a low homogeneity.</p><p>V measure score can be used to evaluate the agreement of two independent assignments on the same dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*8FuP8QsJlX9DaNOL" /></figure><p><strong>DBSCAN :</strong></p><p>Why does it suit our dataset?</p><ol><li>(Partitioning Clustering) K-means, K-medoids: As K-means are used above, it would be effective to use the one with different types of clustering.</li><li>(Hierarchical Clustering) Agglomerative, DIANA: Inefficient for large datasets.</li><li><strong>(Density Clustering) DBSCAN:</strong></li></ol><ul><li>Pros: resistant to noise and handle clusters of different shapes and sizes.</li><li>Cons: However, it fails to identify clusters of varying densities. Also, it has the problem in high-dimensional data due to the curse of dimensionality. But our dataset does not have high-dimensionality since the number of observations is bigger than the number of features.</li></ul><p>The DBSCAN is density based clustering, which is resistant to noise so that it can handle clusters of different shapes and sizes. It is not suitable for high-dimensional data due to the curse of dimensionality. However, our dataset does not have high-dimensionality since the number of observations is bigger than the number of features.</p><p><strong>Parameter Tuning of DBSCAN</strong></p><p>min_samples: minimum number of instances</p><p>This is the minimum number of instances to form a cluster. Following the common method for tuning this parameter, it should be equal or greater than the number of dimensionality. First of all, we gave min_samples = 2 * n_components.</p><p>eps: the fewest number of points required to form a cluster</p><p>This is determined by running the K-nearest neighbours algorithm. First of all calculate the average distances between each point and its K-nearest neighbours, where K will be the min_samples that we defined above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FhiEFE-7f7dA3aj4" /></figure><p>The graph shows the points sorted by distance to the 260th number of neighbours. Until the epsilon reaches to the value around 8, the increase of the sorted points seems to be acceptable. Finally, we set the epsilon value to 8.</p><p>However, eps = 8, min_samples = 260 setting makes only one cluster with outliers. Therefore, we tried to adjust the parameters from setting new min_samples, performing K-NN algorithm, then picking the optimal epsilon value that can classify at least 3 clusters with 2 for main clusters and one for outliers.</p><p><strong>Final parameters set for DBSCAN</strong></p><p><strong>eps:</strong> 6.8</p><p><strong>min_samples:</strong> 130</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*knOLJK8Avmo3NhHl" /></figure><p><strong>Evaluation measures</strong></p><p>DBSCAN has their own way of measuring the performance of clustering, which is called DBCV. It is a better evaluation measure with the ability to handle the noise and non-globular shaped clusters.</p><p>However, DBCV can only be accessed by code implementation from github and running it on our code did not end.</p><p>Therefore we used the same evaluation measures with the one for K-means clustering. However, it is likely to be biased to have higher values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*mnZm21T55ENQz-Y0" /></figure><p><strong>Bias and Fairness in Clustering</strong></p><p>Bias of the training model in machine learning could have originated from data itself which reflects the existing prejudices. This historical unfairness is the one that we are trying to measure from our training models. A model is fair when the errors are distributed similarly across protected groups. In our task, the protected attribute is ‘sex’, which refers to the different physiological characteristics of males and females.</p><p>Fairness in clustering is different from the one for classification as it focuses on the distribution of instances in a cluster which is supposed to be balanced even if they are divided by the protected attribute. It is easier to understand if referring to this picture from (<a href="http://aitime-lundao.oss-cn-beijing.aliyuncs.com/AitimeReport/20210826/1629963170842">http://aitime-lundao.oss-cn-beijing.aliyuncs.com/AitimeReport/20210826/1629963170842</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0nX_QNxOrtfC4jbs" /></figure><p>Therefore, we are going to see the distribution of instances in one cluster divided by protected attribute. The instances with the same cluster label are supposed to be fairly numbered when counting with and without protected attribute.</p><p><strong>Dataset</strong> = fair_data</p><p><strong>Protected attribute </strong>= ‘sex’ {‘ Female’, ‘ Male’}</p><p><strong>Cluster labels</strong> = y_hat</p><p>Here, y_hat changes depending on the cluster model and the parameter settings.</p><p><strong>Measures of Fairness — Statistical Parity Difference (SPD)</strong></p><p>This evaluates that clustering model is unbiased if the prediction cluster label is independent of the protected attribute S so that</p><p><strong>P(y_hat| S ) == P(y_hat)</strong></p><p>This can be measured by checking the difference between the probability of prediction of y conditioned by whether s = ‘ Male’ or s = ‘ Female’. And the equation for measurement is,</p><p><strong>SPD = P( y_hat = 1 | S = ‘ Male’ ) — P( y_hat = 1 | S = ‘ Female’ )</strong></p><p>If subjects in both protected and unprotected groups should have equal probability of being assigned to the positive class.</p><p><strong>Fair range</strong></p><p>To evaluate the fairness from each clustering model, we refer to the following paper for the fair range (https://arxiv.org/pdf/2110.13029.pdf).</p><p>If the measures fall into the range between -0.1 to 0.1, we can say that the model does not contain a bias towards the protected attribute S in our dataset. In other words, if the measures are below -0.1 or above 0.1, we can say that the model is biased regarding the attribute S.</p><p><strong>SPD from K-Means</strong></p><p><strong>Dataset</strong>: fair_data</p><p><strong>Protected attribute</strong>: ‘sex’ column from dataset</p><p><strong>Y_hat</strong>: ‘kmeans_y_0’, ‘kmeans_y_1’ columns from dataset</p><p><strong>Result tables</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*SWYMlMP0xxkzxqjO" /><figcaption>n_cluster = 3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*_j6W1c_ddmlIlhpF" /><figcaption>n_cluster = 5</figcaption></figure><p><strong>SPD from DBSCAN</strong></p><p><strong>Dataset</strong>: fair_data</p><p><strong>Protected attribute</strong>: ‘sex’ column from dataset</p><p><strong>Y_hat</strong>: ‘dbscan_y_0’, ‘dbscan_y_1’ columns from dataset</p><p><strong>Result tables</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*axPE3GAmqQAl_1MZ" /><figcaption>eps = 6.8, min_samples = 130</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*vrGTzXbIALQZ1Thz" /><figcaption>eps = 6.5, min_samples = 140</figcaption></figure><p>The results from the models show that when coordinating parameters, the number of clusters gets bigger and the overall SPD becomes smaller. However this fact does not prove that the model with more clusters would result in less bias. As the instances are more sparse in the model with greater number of clusters, the distributional difference between two clusters would be also small.</p><p>Also, comparing the SPD tables between two different clustering methods, there was no meaningful difference.</p><p>4 tables show that our dataset models cannot guarantee the perfect fair clustering as at most 2 labels in one SPD model shows the value out of the fair range (-0.1, 0.1).</p><p><strong>Reference</strong></p><p><em>K-means clustering</em></p><p><a href="https://realpython.com/k-means-clustering-python/#how-to-perform-k-means-clustering-in-python">https://realpython.com/k-means-clustering-python/#how-to-perform-k-means-clustering-in-python</a></p><p><a href="https://towardsdatascience.com/clustering-with-k-means-1e07a8bfb7ca">https://towardsdatascience.com/clustering-with-k-means-1e07a8bfb7ca</a></p><p><em>Understanding FAMD</em></p><p><a href="https://www.datacamp.com/community/tutorials/introduction-factor-analysis">https://www.datacamp.com/community/tutorials/introduction-factor-analysis</a></p><p><a href="https://www.wikiwand.com/en/Factor_analysis">https://www.wikiwand.com/en/Factor_analysis</a></p><p><a href="http://data-mining-tutorials.blogspot.com/2013/03/factor-analysis-for-mixed-data.html">http://data-mining-tutorials.blogspot.com/2013/03/factor-analysis-for-mixed-data.html</a></p><p><em>Parameter tuning</em></p><p><a href="https://medium.com/@tarammullin/dbscan-parameter-estimation-ff8330e3a3bd">https://medium.com/@tarammullin/dbscan-parameter-estimation-ff8330e3a3bd</a></p><p><em>Fair clustering</em></p><p><a href="https://arxiv.org/pdf/2110.13029.pdf">https://arxiv.org/pdf/2110.13029.pdf</a></p><p><a href="http://aitime-lundao.oss-cn-beijing.aliyuncs.com/AitimeReport/20210826/1629963170842">http://aitime-lundao.oss-cn-beijing.aliyuncs.com/AitimeReport/20210826/1629963170842</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ac3e8263aff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/cluster-analysis-on-adult-dataset-9ac3e8263aff">Cluster Analysis on Adult Dataset</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Full Data Analysis For a Classification Task (Supervised Learning)]]></title>
            <link>https://medium.com/geekculture/full-data-analysis-for-a-classification-task-supervised-learning-e6cc4a85f7ee?source=rss-86b005addb18------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6cc4a85f7ee</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification-algorithms]]></category>
            <dc:creator><![CDATA[Ceyda Akbulut]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 15:58:50 GMT</pubDate>
            <atom:updated>2021-12-22T08:02:28.307Z</atom:updated>
            <content:encoded><![CDATA[<p>Thanks to <strong>Hyosang Kim(FU Berlin), Raniyaharini Rajendran(FU Berlin)</strong> for this teamwork!</p><p><strong>Supervised learning (SL)</strong> is the <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a> task of learning a function that <a href="https://en.wikipedia.org/wiki/Map_(mathematics)">maps</a> an input to an output based on example input-output pairs.[1] The majority of practical machine learning uses supervised learning. Supervised learning problems can be further grouped into regression and classification problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WNnfb0MTq6cQk2BI.png" /></figure><p>Today, we will run a full data analysis. For the analysis, we will use the Adult dataset(<a href="https://archive.ics.uci.edu/ml/datasets/adult">https://archive.ics.uci.edu/ml/datasets/adult</a>), which contains census data from the US. The goal is to predict whether a person’s income exceeds 50K dollars per year or not. We consider the positive class, an income of over 50K per year.</p><p><strong>Understanding Dataset</strong></p><p>To see what we have, let’s read the files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*scCQgEct_jjK5BdlTx3SNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*V7XgyRtwR1dAT2QOfH2dcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*laLXeZfEQ0ucbBenuyWM5A.png" /></figure><p>As we see, we have to add column names manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*nofvGTS59nAKs49CDI4bgQ.png" /></figure><p>Labels: &gt;50K,&lt;=50K</p><p>Dataset characteristics: Multivariate</p><p>Attribute characteristics: Categorical, Integer</p><p>Number of Instances(Total): 48842</p><p>Number of Instances(Train):32561</p><p>Number of Instances(Test):16281</p><p>Number of Attributes:14</p><p><strong>age:</strong> continuous.</p><p><strong>workclass:</strong> Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, never-worked.</p><p><strong>fnlwgt:</strong> continuous.</p><p><strong>education:</strong> Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.</p><p><strong>education-num:</strong> continuous.</p><p><strong>marital-status:</strong> Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.</p><p><strong>occupation:</strong> Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspect, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.</p><p><strong>relationship:</strong> Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.</p><p><strong>race:</strong> White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.</p><p><strong>sex:</strong> Female, Male.</p><p><strong>capital-gain:</strong> continuous.</p><p><strong>capital-loss:</strong> continuous.</p><p><strong>hours-per-week:</strong> continuous.</p><p><strong>native-country:</strong> United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinidad Tobago, Peru, Hong, Holland-Netherlands.</p><p>Under the related folder, there are two different .names files which include test and train data separately. We need a combined dataset to execute data preprocessing steps. So, we will read two files and merge them with the help of the “indicator” column to divide them again for building model part</p><p>We have one merged dataset with an indicator column. It is ready to go!</p><p>After reading files and adding column names, the time is combining the train and test dataset with the help of “indicator” column to understand which row is for train or test in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*q_oev0dvXZXYPIa3T79KxA.png" /></figure><p>Now, we will transform the raw data into an understandable format.Typical tasks in data preprocessing,</p><p><strong>Data integration</strong></p><p><em>Integration of multiple databases, data warehouses, or files(entity identification,value resolution)</em></p><p><strong>Data cleaning</strong></p><p><em>Fill in missing values</em></p><p>Missing data can occur when no information is provided for one or more items or for a whole unit. Missing data is a very big problem in real-life scenarios. For our dataset, first we used <em>isna()</em> function to see if we have missing values or not. The result was null because of the representation of missing values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/0*ToLCcs1iyswajE_E" /></figure><p>In the adult dataset, missing values had an alphanumeric value “?”. Therefore, changing the function to detect these values is our second way. With the str.contains function <em>(str.contains(‘\?’))</em> , the missing values are visible directly.The column list has ‘?’ as a missing value. We can also recognise them with the frequency table. This table is useful for finding the mode value for the related column. For example,</p><p><em>workclass</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*gir6UJqngdRNRcrz" /></figure><p><em>occupation</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/0*41yD-8FRRzhJY2oQ" /></figure><p>There are four main steps below to handle these values.</p><ul><li>Ignore observation.</li><li><strong>Replace by most frequent value (Mode).</strong></li><li>Develop a model to predict missing values.</li><li>Treat missing data as just another category</li></ul><p>We would like to continue with replacing by most frequent value method (mode). New frequency tables of workclass, occupation and native-country columns are below.</p><p><em>workclass</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/0*cbR6C82zf6lKnDLc" /></figure><p><em>occupation</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/0*ROCot1u5noD9_fq9" /></figure><p><em>native-country (top 19)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/0*6oaF6USUg786cR8n" /></figure><p><em>Smooth noisy data</em></p><p><em>Identify or remove outliers</em></p><p>Outlier detection is used for anomaly detection, where one is interested in detecting abnormal or unusual observations. There are four main approaches to detect unusual values.</p><p><strong>Numeric Outlier</strong><br>This is the simplest, nonparametric outlier detection method in a one dimensional feature space. Here outliers are calculated by means of the IQR (InterQuartile Range).The first and the third quartile (Q1, Q3) are calculated. An outlier is then a data point x_i that lies outside the interquartile range. Using the interquartile multiplier value k=1, the range limits are the typical upper and lower whiskers of a box plot.</p><p>Z-Score</p><p>DBSCAN</p><p>Isolation Forest</p><p>For our dataset, we will interpret boxplots &amp; histograms and apply the first one, <strong>Numeric Outlier</strong>.We also interpret boxplots and histograms of attributes.</p><p>Univariate analysis is perhaps the simplest form of statistical analysis. Like other forms of statistics, it can be inferential or descriptive. The key fact is that only one variable is involved. For continuous variables</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/0*zaZh2jboBmPjA1Z0" /></figure><p>Our histogram graphs for numerical values,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/0*ytah8D9vKfMc0A2d" /><figcaption>Histograms of each numerical columns before removing outliers from dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*1261zIMfoXYPy_N9DcDG3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*oc8gPw6LSl8ZjOb2" /></figure><p>Boxplot of ‘fnlwgt’ column(one of them) before the outlier removal process is as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/0*BoyRCwSSuaSkJqoZ" /><figcaption>Boxplot for ‘fnlwgt’</figcaption></figure><p>Our dataset has categorical and numerical values. For categorical columns, frequency tables are useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/198/0*1CjheVEPinajaPKc" /><figcaption>Frequency table for ‘occupation’</figcaption></figure><p>As we can see easily from boxplots and histograms, the dataset has a lot of outliers and the distribution of numerical columns are not uniformly distributed. For the <strong>fnlwgt </strong>column, the skewed distribution is asymmetrical because a natural limit prevents outcomes on one side and it has a right-skewed distribution.</p><p>We coded a function to calculate IOR for each numerical attribute.We find the outliers with the help of IOR method(numeric values,k=1)changed the values of outliers by mean for each numerical column.</p><p><strong>How to handle Outliers?</strong></p><ol><li>Set up a filter in your testing tool.</li><li>Remove or change outliers during post-test analysis.</li><li><strong>Change the value of outliers.</strong></li><li>Consider the underlying distribution</li><li>Consider the value of mild outliers.</li></ol><p>To remove outliers, we will change the value of outliers. Deleting observations is not logical because of amount of the outliers. We also don’t know if these are natural outliers or artificials.</p><p><strong>Note1:</strong> If the skewness is between -0.5 and 0.5, the data are fairly symmetrical. If the skewness is between -1 and — 0.5 or between 0.5 and 1, the data are moderately skewed. If the skewness is less than -1 or greater than 1, the data are highly skewed.<br><strong>Sub-result1:</strong> Replacing the values of outliers with median was not effective(Tried).<br><strong>Sub-result2:</strong> As a second method, we tried to remove outliers from our dataframe by IQR method. It was not efficient.<br><strong>Sub-result3:</strong> Log transformation was also not effective for outliers.<br><strong>Sub-result4:</strong> Z-score was also not effective for outliers.<br><strong>Result:</strong> We found the outliers with the help of IOR method(numeric values) and changed the values of outliers by mean for each colum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*DtW-Yf7mT83HJCKi4Mg4rw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/0*13PwoEnXFDhUBM3U" /><figcaption>Boxplot for ‘<strong>fnlwgt</strong>’ after the related process</figcaption></figure><p>After the data preprocessing part, we checked the boxplots and histogram graphs again. We see that the amount of outliers are definitely less than before. Distribution of numerical columns is still not uniformly distributed.</p><p>To check the class imbalance, the visualization of count of the different labels was enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/0*6Nd4NMQlPR8ai4Nx" /><figcaption>Visualization of the number of different labels</figcaption></figure><p>Class imbalance is easily identified by looking at the distribution of the target class within the dataset. As we can see easily, <strong>our dataset suffers the class imbalance problem</strong>.</p><p>When a class imbalance exists within the training data, machine learning models will typically over-classify the larger class(es) due to their increased prior probability. As a result, the instances belonging to the smaller class(es) are typically misclassified more often than those belonging to the larger class(es).</p><p>Other disadvantages:</p><ul><li>Unequal Cost of Misclassification Errors</li><li>Skewed Class Distribution</li><li>Challenge for predictive modeling</li><li>Over-classify the majority group due to its increased prior probability</li></ul><p><strong>To Combat Imbalanced Training Data</strong></p><p>In general, main options are as below:</p><ul><li>Collect More Data</li><li>Change Performance Metric</li><li><strong>Random Undersampling and Oversampling</strong></li></ul><p>Random oversampling involves randomly selecting examples from the minority class, with replacement, and adding them to the training dataset. Random undersampling involves randomly selecting examples from the majority class and deleting them from the training dataset.</p><p>In general <strong>oversampling is better</strong>, because you keep all the information in the training dataset.</p><ul><li>Try Different Algorithms</li><li>Try Penalized Models</li></ul><p>We will continue with <strong>Random Oversampling</strong>.</p><p>After coding and executing this part as you can see, the new distribution graph of the <strong>‘results’ </strong>column is as below and it seems better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/0*-TsrBrEfzAHgeXn_" /></figure><p>After an oversampling method, the new distribution graph is as above. The updated number of instances in the train dataset is <strong>49438.</strong></p><p>A correlation matrix is simply a table which displays the correlation. The measure is best used in variables that demonstrate a linear relationship between each other.</p><p>Correlation varies between -1 and +1.</p><p>-1: perfect negative linear correlation</p><p>+1:perfect positive linear correlation</p><p>0 :No correlation</p><p>Correlation can be derived using following formula:</p><p>Correlation = Covariance(X,Y) / SQRT( Var(X)* Var(Y))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L6Ad1_ID6w4j-2ud" /><figcaption>Correlation matrix with colors</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/0*AKEjQQpUIT-2tRFD" /><figcaption>Correlation matrix with colors and numerical values</figcaption></figure><p>The correlation coefficients along the diagonal of the table are all equal to 1 because each variable is perfectly correlated with itself and the correlation matrix is perfectly symmetrical. In general,correlation coefficient values below 0.3 are considered to be weak; 0.3–0.7 are moderate; &gt;0.7 are strong. <strong>As we can see, the max value(without diagonal) is 0.137879 and the min value is -0.092355. These correlations are weak. Therefore, we cannot say that there are correlated values.</strong></p><p><strong>Feature selection</strong></p><p>After completing the data preprocessing steps and checking the correlation matrix, we would like to continue with the same columns plus indicator column. This <strong>index</strong>,<strong>ind (test,train) </strong>column will be helpful when we want to divide a dataset to test and train parts. Since we applied the oversampling method, the new dimension of our dataset is (65719,16) in total (49438: train, 16281: test).</p><p>Note: Feature selection part is the other topic of next text. We will see more details. For now, keep going!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/0*Il0SmxwdUYzT6uhs" /></figure><p><strong>Classification Models</strong></p><p>Here we are going to experiment with five supervised models namely- KNN, Decision Tree, Naive Bayes, SVM and Perceptron.</p><p><strong>K-nearest neighbors algorithm (k-NN)</strong></p><p>k-nearest neighbors algorithm (k-NN) is a non-parametric classification method. In k-NN classification, the output is a class membership. An object is classified by a plurality vote of its neighbors, with the object being assigned to the class most common among its knearest neighbors (k is a positive integer, typically small).</p><p><strong>Decision tree:</strong></p><p>Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.</p><p><strong>Naive Bayes:</strong></p><p>Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features given the value of the class variable.</p><p><strong>Support vector machines:</strong></p><p>Support vector machines (SVMs) are a set of supervised learning methods Perceptron. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier.</p><p><strong>Perceptron:</strong></p><p>A perceptron is a single-layer neural network. They consist of four main parts including input values, weights and bias, net sum, and an activation function.</p><p><strong>Evaluation setup</strong></p><p>Cross-validation is a resampling procedure used for the evaluation of machine learning models for the given dataset. 10-fold stratified cross validation has a single parameter called k that refers to the number of groups that a given data sample can be split into.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*EjAVORLIcJ_5mG0w" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/0*c2HY_DvM-GusvbxA" /></figure><p><strong>Evaluation Measures</strong></p><p><strong>Confusion matrix:</strong> The confusion matrix is a way of tabulating the number of misclassifications</p><p><strong>Accuracy:</strong> Classification Accuracy is what we usually mean, when we use the term accuracy. It is the ratio of the number of correct predictions to the total number of input samples.</p><p><em>Accuracy = TP+TN/TP+FP+FN+TN</em></p><p><strong>Precision:</strong> Precision is the ratio of correctly predicted positive observations to the total predicted positive observations.</p><p><em>Precision = TP/TP+FP</em></p><p><strong>Recall (Sensitivity):</strong> Recall is the ratio of correctly predicted positive observations to the all observations in actual class</p><p><em>Recall = TP/TP+FN</em></p><p><strong>F1 score:</strong> F1 Score is the weighted average of Precision and Recall.</p><p>F1 Score = 2*(Recall * Precision) / (Recall + Precision)</p><p><strong>AUC plot and ROC curve: </strong>ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:</p><ul><li>True Positive Rate(TPR)</li><li>False Positive Rate(FPR)</li></ul><p><strong>TPR= TP/TP+FN</strong></p><p><strong>FPR = FP/FP+TN</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*d-zgBiQ7FytQurV7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*ig6C8f9NZaCd72--" /><figcaption><strong>AUC Plot and ROC Curve for KNN</strong></figcaption></figure><p><strong>Model Parameter Tuning</strong></p><p>Hyperparameter tuning is the process of choosing a set of optimal <a href="https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)">hyperparameters</a> for a learning algorithm.</p><p>Steps involved in hyper parameter tuning:</p><ul><li>Choose the appropriate algorithm for the model</li><li>Decide the parameter space (Parameters vary for different classifiers)</li><li>Decide the method for searching parameter space</li><li>Decide the cross-validation method (We have used 10-fold stratified cross validation)</li><li>Decide the score metrics to evaluate your model</li></ul><p>There are initially two searches for hyperparameter tuning. They are:</p><ul><li>GridSearchCV</li><li>RandomizedSearchCV</li></ul><p>Here, we have used GridSearchCv since it considers all possible combinations of hyper parameters. RandomizedsearchCV on the other hand uses few samples which are randomly selected. Therefore GridSearchCV is much more efficient than RandomizedSearchCV.</p><p><strong>KNN: </strong>Best Parameters: {‘metric’: ‘manhattan’, ‘n_neighbors’: 17, ‘p’: 1, ‘weights’: ‘uniform’}</p><p><strong>Decision tree: </strong>Best Parameters: {‘criterion’: ‘gini’, ‘max_depth’: 7, ‘min_samples_leaf’: 3, ‘min_samples_split’: 2}</p><p><strong>Naive Bayes: </strong>Best Parameters: {‘var_smoothing’: 1.2328467394420658e-05}</p><p><strong>SVM: </strong>Best Parameters: {C=0.1, gamma=1, kernel=rbf; total time= 1.1min}</p><p><strong>Decision Boundary</strong></p><p>A decision boundary is a line (in the case of two features), where all (or most) samples of one class are on one side of that line, and all samples of the other class are on the opposite side of the line. The line <em>separates</em> one class from the other. If you have more than two features, the decision boundary is not a line, but a hyper-plane in the dimension of your feature space.</p><p>In our dataset the class where the decision boundary was plotted included more than two features. Majority of the classes included five features. Decision boundary support is provided to classes with two features. Therefore in our problem we included filler feature values which hold the other features that are ignored. For example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/0*DU6tAyPdDpqg0l20" /><figcaption><strong>Decision Boundary for KNN</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/0*wKO21F4tsVCgC9_Z" /><figcaption><strong>Decision Boundary for Decision Tree</strong></figcaption></figure><p><strong>Effect of Model Parameters</strong></p><p>Every classifier has its own set of parameters out of which many play a significant role in increasing the performance of the model. Here, let’s check the parameters affecting performance for the different classifiers.</p><p><strong>Parameters for KNN</strong></p><p>‘n_neighbors’: [3,5,7,9,11,13,15,17,19,21,23],</p><p>‘p’: (1,2),</p><p>‘weights’:[‘uniform’, ‘distance’],</p><p>‘metric’:[‘euclidean’,’manhattan’, ‘minkowski’]</p><p>Best Parameters: {‘metric’: ‘manhattan’, ‘n_neighbors’: 17, ‘p’: 1, ‘weights’: ‘uniform’}</p><p>After hyperparameter tuning, the accuracy of the model has increased from 0.72 to 0.80.</p><p><strong>Parameters for Decision Tree</strong></p><p>“criterion”:[“gini”,”entropy”],</p><p>“max_depth”:range(1,15),</p><p>“min_samples_split”:range(1,10),</p><p>“min_samples_leaf”:range(1,5)</p><p>Best Parameters: {‘criterion’: ‘gini’, ‘max_depth’: 7, ‘min_samples_leaf’: 3, ‘min_samples_split’: 2}</p><p>After hyperparameter tuning, the accuracy of the model has increased from 0.77 to 0.83.</p><p><strong>Parameters for Naive Bayes</strong></p><p>‘var_smoothing’: np.logspace(0,-9, num=100)</p><p>Naive Bayes usually does not have any parameters. The only parameter used is var_smoothing. However this does not have any influence on the performance of the classifier.</p><p>Best Parameters: {‘var_smoothing’: 1.2328467394420658e-05}</p><p>After hyperparameter tuning, the accuracy of the model doesn’t change it remains constant at 0.79</p><p><strong>Parameters for SVM</strong></p><p>‘C’: [0.1,1, 10, 100],</p><p>‘gamma’: [1,0.1,0.01,0.001],</p><p>‘kernel’: [‘rbf’, ‘poly’, ‘sigmoid’]</p><p>Best Parameters: {C=0.1, gamma=1, kernel=rbf; total time= 1.1min}</p><p>Even after hyperparameter tuning, we can observe that the accuracy of the model remains constant.</p><p><strong>Fairness-aware Evaluation</strong></p><p>Bias of the training model in machine learning could have originated from data itself which reflects the existing prejudices. This historical unfairness is the one that we are trying to measure from our training models. A model is fair when the errors are distributed similarly across protected groups. In our task, the protected attribute is ‘sex’, which refers to the different physiological characteristics of males and females.</p><p>We can consider the data as x and using the machine learning model to compute a score f(x), which will be used for prediction, we can produce a binary outcome y_hat = {0, 1}. In our dataset, y_hat value 0 means the income is over 50K, 1 means the income is under 50K. Also, the protected attribute can be represented as s = {0, 1}. In our dataset, the binary value of s is shown as {‘ Female’, ‘ Male’}. ‘ Female’ is the protected group and ‘ Male’ is the non-protected group. Also, in our training model, we know the real class value y = {0, 1}. Data x will be associated with a protected attribute s, which will be verified with the fairness-aware measurements that will be followed.</p><p>Dataset: X_test_numerical, X_test[‘sex’], y_test, y_pred</p><p>Data = x { X_test_numerical, X_test[‘sex] }</p><p>Protected attribute = s { ‘ Female’, ‘ Male’ }</p><p>Predicted outcome = y_hat {0, 1}</p><p>Real class value = y {0, 1}</p><p><strong>Measures of Fairness</strong></p><p><em>Statistical Parity Difference (SPD)</em></p><p><em>Equal Opportunity (EOD)</em></p><p><em>Disparate Mistreatment</em></p><p><strong>Dataset</strong></p><p>Make x dataset merging X_test_numerical with the column X_test[‘sex’].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CrpbzL1-7jtPDeVJ" /><figcaption><strong>Table for each classifier combined with its prediction and real value y</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0dDRBP48-bwrswfX" /><figcaption>KNN</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MgieSocRD9ioeRsX" /><figcaption>Perceptron</figcaption></figure><p>We will apply functions for 3 measures to each classifier and the result table.</p><p>def statParityDif</p><p>def EqualOppDif</p><p>def DisparateMistreat</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*9Eqk3BU4ZWqxj_1c" /></figure><p><strong>Fair range and Histogram</strong></p><p>To evaluate the fairness from each classifier, we refer to the following paper for the fair range (<a href="https://arxiv.org/pdf/2110.13029.pdf">https://arxiv.org/pdf/2110.13029.pdf</a>).</p><p>If the measures fall into the range between -0.1 to 0.1, we can say that the classifier does not contain a bias towards the protected attribute S in our dataset. In other words, if the measures are below -0.1 or above 0.1, we can say that the classifier is biased regarding the attribute S.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8G8Awk5RyV6y32AL" /></figure><p>Except for the statistical parity difference of classifier Decision Tree, the measure values fall into the fair range [-0.1, 0.1]. The classifier Decision Tree shows slightly extreme values of measures compared to the other classifier. In conclusion there is no obvious discriminatory behavior from our training models.</p><p><strong>Mitigation of classifier discrimination</strong></p><p>Once the classifier has already been trained, it is hard to remove bias even for simple cases. However, there are some approaches to mitigate it, which is referred to from the page <a href="https://www.borealisai.com/en/blog/tutorial1-bias-and-fairness-ai">https://www.borealisai.com/en/blog/tutorial1-bias-and-fairness-ai</a>.</p><p>First of all, we can approach it in the pre-processing stage, which is about removing protected attributes by calculating the latent prejudice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/0*K7ayOgsMDjmZFieu" /></figure><p>This measurement shows the level of dependencies between the data x and the protected attribute p, which was s in the description above. If this measure increases, the protected attribute becomes more predictable from the data, which puts mathematical bounds on the potential discrimination of a classifier.</p><p>However, it has a disadvantage that even if you eliminate one protected attribute, the other features left could be strongly related to the protected attribute and could be used for discrimination. For example, race is highly associated with the address of the person, therefore removing race features only could not remove the bias from the classifier.</p><p>Next, there is an approach for manipulating the dataset. It is about reweighting the data pairs {x, y} where x is the dataset and y is the class label. This makes the disadvantaged group by the protected attribute p get a positive outcome more likely with the weights. They then train a classifier that makes use of these weights in its cost function. Alternatively, they propose re-sampling the training data according to these weights and using a standard classifier.</p><p>Also, in-processing algorithms may reduce the bias that classifier contains by measuring indirect-prejudice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*cZMDgXEJxdbC2Y3Q" /></figure><p>This measures the dependence between the labels y and the protected attribute p. If the measure is negligible, it means it is hard to predict the labels from the protected attribute, and there is no scope for bias.</p><p><strong>KNN Implementation</strong></p><p>When implementing K-nearest neighbours algorithms, we referred to this page. <a href="https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/">https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/</a></p><p><strong>Function Definition</strong></p><p><strong>Euclidean distance between two rows</strong></p><p>As the KNN is based on the standard that how close two instances are, the function to get the distance between them is needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IBmy2nk22APf6dyU" /></figure><p><strong>K-nearest neighbors based on the distances</strong></p><p>Based on the calculated distance between instances, we’re going to get the k smallest distance between the historical data and new instance, this is the function to get the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KgNRRAVTD3WsJB6S" /></figure><p><strong>KNN implementation result</strong></p><p>The implemented KNN is not possible to handle all the instances in the test set because of too big running time. However, when we tried with the smaller test set with instance 20, we observed the prediction accuracy is 0.75. Therefore, it is not meaningful to compare the implemented KNN with the library, it is still inspiring which methods or steps are used inside of the handy library and then understand the algorithm better.</p><p><strong>The slow running time of KNN</strong></p><p>It is slow compared to the other training model. This is because the model could not generalize in advance rather it has to look up all the historical data and if the size of the data is big, it slows down the computing process.</p><p><strong>The improvement</strong></p><p>If using cosine similarity than euclidean distance, the time complexity can be improved. Because euclidean includes more operations than cosine similarity.</p><p>Cosine simliarity: the cosine of the angle between two n-dimensional vectors in an n-dimensional space. It is the dot product of the two vectors divided by the product of the two vectors’ lengths (or magnitudes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rY3Wf4jOpppA_wu1" /></figure><p><strong>Full code: </strong><a href="https://colab.research.google.com/drive/132A4iWdRVVE0LsOdCt_n0UzS_QHDnvKR?usp=sharing"><strong>https://colab.research.google.com/drive/132A4iWdRVVE0LsOdCt_n0UzS_QHDnvKR?usp=sharing</strong></a></p><p><strong>Reference</strong></p><p><a href="https://medium.com/analytics-vidhya/undersampling-and-oversampling-an-old-and-a-new-approach-4f984a0e8392">https://medium.com/analytics-vidhya/undersampling-and-oversampling-an-old-and-a-new-approach-4f984a0e8392</a></p><p><a href="https://towardsdatascience.com/training-models-on-imbalanced-data-561fa3f842b5">https://towardsdatascience.com/training-models-on-imbalanced-data-561fa3f842b5</a>)</p><p><a href="https://corporatefinanceinstitute.com/resources/excel/study/correlation-matrix/#:~:text=A%20correlation%20matrix%20is%20simply,visually%20represented%20in%20a%20scatterplot.&amp;text=A%20correlation%20matrix%20consists%20of%20rows%20and%20columns%20that%20show%20the%20variables">https://corporatefinanceinstitute.com/resources/excel/study/correlation-matrix/#:~:text=A%20correlation%20matrix%20is%20simply,visually%20represented%20in%20a%20scatterplot.&amp;text=A%20correlation%20matrix%20consists%20of%20rows%20and%20columns%20that%20show%20the%20variables</a>.</p><p><a href="https://www.borealisai.com/en/blog/tutorial1-bias-and-fairness-ai">https://www.borealisai.com/en/blog/tutorial1-bias-and-fairness-ai</a>.</p><p><a href="https://arxiv.org/pdf/2110.13029.pdf">https://arxiv.org/pdf/2110.13029.pdf</a></p><p><a href="https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/">https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/</a></p><p><a href="https://en.wikipedia.org/wiki/Supervised_learning">https://en.wikipedia.org/wiki/Supervised_learning</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6cc4a85f7ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/full-data-analysis-for-a-classification-task-supervised-learning-e6cc4a85f7ee">Full Data Analysis For a Classification Task (Supervised Learning)</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Hierarchical Time Series]]></title>
            <link>https://medium.com/geekculture/introduction-to-hierarchical-time-series-c8fe5da149f6?source=rss-86b005addb18------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8fe5da149f6</guid>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[hierarchy]]></category>
            <dc:creator><![CDATA[Ceyda Akbulut]]></dc:creator>
            <pubDate>Sat, 08 May 2021 21:42:30 GMT</pubDate>
            <atom:updated>2021-05-09T09:38:01.046Z</atom:updated>
            <content:encoded><![CDATA[<p>A time series is a dataset that tracks a sample over time and most of the articles about this series work on a specific/particular level of aggregation. But in the real world and many applications, there are multiple time series that are hierarchically organized. The related datasets have several levels is based on geography, branches, or other features and challenges start when we try to observe the information while focusing on the more granular level. A hierarchical time series is a hero for this situation.</p><p>But what is the hierarchical time series?</p><p>HTS is a collection of time series that follows a hierarchical aggregation structure. Let’s think about the number of Corona cases and see the basic data about that from the worlddometer website. Breakdowns can be seen easily. For example, regions and countries. It can be extended by cities or states.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*Uh4Oy-ydMA_GO881tO330w.png" /><figcaption>Figure 1 The number of Corona cases</figcaption></figure><p>The tourism demand by region and purpose is also another example of hierarchical time series. The easiest example would be geographical splitting. What is the logic behind the hierarchical time series? The top-level is the most aggregate level of the data. As we can see, the top level is divided into two series at a level which is disaggregated into two and two series at the second level of the hierarchy (Fig. 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*hZv0-fJpeVKGlgKJe4kxjg.png" /><figcaption>Figure 2 Example for two-level hierarchical tree diagram</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*CDJTBX9oEFolwPizb103Sg.png" /><figcaption><strong>Figure 3</strong> A two-level hierarchical tree diagram.</figcaption></figure><p>y_t represents the <em>t</em>th level of the observation of the total series t=1,….., T.</p><p>y_j,t denotes the <em>t</em>th prediction of the series corresponding to node j.</p><p>For example, y_B,t denotes the tth observation of the series corresponding to node B at level 1.</p><p>The total number of our series is <em>1+2+2+2=7</em>, while the number of series at the bottom-level is <em>2+2=4</em>. The total number of series has to bigger than the number of series at the bottom level.</p><p>Let’s focus on the bottom level and see the equations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*OVK-yHQuBDoMlGZ2gXFQ_A.png" /></figure><p>We have also</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*nOGJacf7-Ib2trk8j5KeYw.png" /></figure><p>from above equations and these equations can be more efficiently represented using matrix notation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*kqZzfq8EBQRKRGD4r0oX1Q.png" /></figure><p>There are some main approaches to hierarchical time series forecasting like bottom-up, top-down, middle out. These methods first develop base forecasts by separately predicting each time series and then reconcile those base forecasts based on their inherent hierarchical structure.</p><p><strong>The bottom-up approach</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*1flYTDA-TehWdC9QGJpl9A.png" /><figcaption>Figure 4 Example for the bottom-up approach</figcaption></figure><p>The bottom-up method calculates base forecasts for bottom-level time series and then aggregates them for upper-level time series. The bottom-up method calculates base forecasts for bottom-level time series and then aggregates them for upper-level time series.</p><p>We first generate h-step-ahead forecasts for each of the bottom-level series:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*EYnl7aNN69sLD5UINd6kdw.png" /></figure><p>Summing these, we get h-step-ahead coherent forecasts for the rest of the series:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*dKtMVLfW471KhbJq5rc_BQ.png" /></figure><p>With this approach, we don’t lose information due to aggregations but it performs poorly on highly aggregated data. The other disadvantage is missing the relationship between the series, for example, between different countries.</p><p><strong>The top-down approach</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*acn-4OenCU4fqA391SDu7w.png" /><figcaption>Figure 5 Example for a top-down approach</figcaption></figure><p>The top-down method calculates base forecasts only for a root time series and then disaggregates them according to historical proportions of lower-level time series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/67/1*Td7vAcT1mpBVj4Lfevg48A.png" /></figure><p>be a set of disaggregation proportions that dictate how the forecasts of the total series are to be distributed to obtain forecasts for each series at the bottom level of the structure.</p><p>In our case, using proportions we get,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/85/1*KdqJyqBY0UkW6wq7kH84Xg.png" /></figure><p>Each proportion p_j reflects the average of the historical proportions of the bottom-level series y_j,t over the period t=1,…, T relative to the total aggregate y_t</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*VMSpS2ptLwfTbVFNB8HOGw.png" /></figure><p>This approach is the simplest and reliable forecast for higher level(s). But we have lower accuracy at the lower levels.</p><p><strong>The middle-out approach</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*7U_7yK9QGg9KYFsk7oRoJw.png" /><figcaption>Figure 6 Example for middle-out approach</figcaption></figure><p>The middle-out approach combines two methods, bottom-up and top-down approaches, and can be used on hierarchies with at least three levels. In this approach, we select the middle level and forecast it directly Other forecasts are generated for all the series at this level. For the series above the middle level, coherent forecasts are generated using the bottom-up approach by aggregating the “middle-level” forecasts upwards.</p><p>References</p><p><a href="https://towardsdatascience.com/introduction-to-hierarchical-time-series-forecasting-part-i-88a116f2e2"><em>https://towardsdatascience.com/introduction-to-hierarchical-time-series-forecasting-part-i-88a116f2e2</em></a></p><p><a href="https://robjhyndman.com/publications/hierarchical/"><em>https://robjhyndman.com/publications/hierarchical/</em></a></p><p><a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0242099"><em>https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0242099</em></a></p><p><a href="https://otexts.com/fpp2/reconciliation.html"><em>https://otexts.com/fpp2/reconciliation.html</em></a></p><p><a href="https://forecasters.org/wp-content/uploads/gravity_forms/7-2a51b93047891f1ec3608bdbd77ca58d/2014/07/Athanasopoulos_George_ISF2014.pdf"><em>https://forecasters.org/wp-content/uploads/gravity_forms/7-2a51b93047891f1ec3608bdbd77ca58d/2014/07/Athanasopoulos_George_ISF2014.pdf</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8fe5da149f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/introduction-to-hierarchical-time-series-c8fe5da149f6">Introduction to Hierarchical Time Series</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Based Forecasting: Non-Seasonal ARIMA Models]]></title>
            <link>https://medium.com/geekculture/time-series-based-forecasting-non-seasonal-arima-models-600064475924?source=rss-86b005addb18------2</link>
            <guid isPermaLink="false">https://medium.com/p/600064475924</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[arima]]></category>
            <dc:creator><![CDATA[Ceyda Akbulut]]></dc:creator>
            <pubDate>Sat, 17 Apr 2021 22:16:21 GMT</pubDate>
            <atom:updated>2021-04-18T20:02:44.952Z</atom:updated>
            <content:encoded><![CDATA[<p>ARIMA(p,d,q) models provide a different approach to time series forecasting, and it is a very popular statistical method form of Box-Jenkins model. Autoregressive integrated moving average can be useful in different fields like statistic to measure events that happen over a period and is also useful to predict future values in a series. What are the components of this model?</p><p>Let’s see the definition of AR, I, and MA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*NkNdUJz-adoQ7X3-5fQWqA.png" /><figcaption><strong>Figure 1</strong> Time series visualization</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*xOvnsQrS8XyfjcHmcjh4xA.png" /></figure><p><strong>Autoregressive Models</strong></p><p>In a multiple regression model, we forecast the variable of interest using a linear combination of predictors. In an autoregression model, we predict the variable by using a linear combination of past. An autoregressive model of order p can be written as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*StDXShdQyNz3kdomTp50qA.png" /></figure><p><strong>Moving Average Models</strong></p><p>A moving average model uses past forecast errors in a regression-like model. Please see the formula as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*3aQt4Rkfe7wDl5GGuq2dmA.png" /></figure><p><strong>Backshift</strong></p><p>The backward shift operator B is a useful when working with time series lags:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/68/1*3XyBV_xQVHsee6WD2QO5GA.png" /></figure><p>B has the effect of differencing the data back one period. A first difference can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*oWJSDA2-0c6HrsXqj4Orhw.png" /></figure><p>Note that a first difference is represented by (1−B). Similarly, if second-order differences must be computed, then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*gqZHfC4k4xlUup6-tW5pkQ.png" /></figure><p>In general, we can see that a dth order difference can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/69/1*wb6Ora_RAHkyFmBRz04_UQ.png" /></figure><p><strong>Non-seasonal ARIMA Models</strong></p><p>When we combine differencing with autoregression and a moving average model, we will find a non-seasonal ARIMA (Autoregressive integrated moving average) model. The model can be written as (differenced series)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*aW76WsToclajvGe_eSK5Qw.png" /></figure><p>Constant c has a significant influence on the long-term forecasts derived from these models. The value of d also affects the forecast ranges. The value of p is important if the data show cycles.</p><p>Find some special situations below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*6Cs6BIWq0DpgJ_65GTQqrg.png" /></figure><p>See some special cases of ARIMA models below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*fb_kb8axmIjmZGNn_-zmBw.png" /></figure><p>ARIMA function is useful to build the model by using Python or R, but there is a reality that anything automated can be a little bad. Therefore, understanding something of the behavior of the models is really important. For this reason, we require understanding how we find the parameters p,d,q manually. Let’s check some plots.</p><p><strong>ACF &amp; PACF Plots</strong></p><p>It is usually not possible to tell what values of p and q are correct for the dataset. It is sometimes possible to use the ACF plot, and the closely related PACF plot, to determine appropriate values. What are the ACF and PACF plots?</p><p>The ACF is a way to measure the linear relationship between an observation at time t and the observations at previous times.</p><p>PACF is the partial autocorrelation function that explains the partial correlation between the series and lags of itself. It finds correlation of the residuals with the next lag value.</p><p>Some examples to understand how we can choose the p and q values by using ACF and PACF plots.</p><p>The PACF shows an outlier at lag1 and the ACF shows a sharp pattern. It is an AR(1) model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*XI4ggPQfJePeAV-_BZydfQ.png" /><figcaption><strong>Figure 2 </strong>Example for ACF&amp;PACF graphs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*pY96MZsr-5JEIZ7G85o5Mw.png" /><figcaption><strong>Figure 3</strong> Example for ACF&amp;PACF graphs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*Zv6D8_Y-74lm84AhA75l8Q.png" /><figcaption><strong>Figure 4 </strong>Example for ACF&amp;PACF graphs</figcaption></figure><p><strong>References</strong></p><p><a href="https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/">https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/</a></p><p><a href="https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/#:~:text=ARIMA%2C%20short%20for%20&#39;Auto%20Regressive,used%20to%20forecast%20future%20values.">https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/#:~:text=ARIMA%2C%20short%20for%20&#39;Auto%20Regressive,used%20to%20forecast%20future%20values.</a></p><p><a href="https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/">https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/</a></p><p><a href="https://towardsdatascience.com/identifying-ar-and-ma-terms-using-acf-and-pacf-plots-in-time-series-forecasting-ccb9fd073db8">https://towardsdatascience.com/identifying-ar-and-ma-terms-using-acf-and-pacf-plots-in-time-series-forecasting-ccb9fd073db8</a></p><p><a href="https://otexts.com/fpp2/arima.html">https://otexts.com/fpp2/arima.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=600064475924" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/time-series-based-forecasting-non-seasonal-arima-models-600064475924">Time Series Based Forecasting: Non-Seasonal ARIMA Models</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Auto Regression for Multivariate Time Series Forecasting]]></title>
            <link>https://medium.com/geekculture/vector-auto-regression-for-multivariate-time-series-forecasting-9334d29591f3?source=rss-86b005addb18------2</link>
            <guid isPermaLink="false">https://medium.com/p/9334d29591f3</guid>
            <category><![CDATA[predictive-modeling]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[vector-auto-regression]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ceyda Akbulut]]></dc:creator>
            <pubDate>Sat, 10 Apr 2021 21:23:25 GMT</pubDate>
            <atom:updated>2021-04-12T23:24:55.780Z</atom:updated>
            <content:encoded><![CDATA[<p>Time is one of the most critical factor that decides whether a company will rise or not. That is why we see sales in stores or e-commerce platforms aligning with festivals. How can we analyze this kind of dataset?</p><p>Time series data is a set of collection of amounts that put together at equal time slots and ordered. Therefore, it is a sequence of discrete-time data. For example, a monthly number of sold fruits…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*_0uTjNQAtWudA5WTnI5lfw.png" /><figcaption><strong>Figure </strong>Example for time series</figcaption></figure><p>A multivariate time series has more than one time dependent variable(time series). Each variable depends not only on its past values but also has some relation to other attributes. This dependency is useful for predicting the future values. Which model can be used for this aim?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*7vvdQ1BznK8Oq2dGdKtQfA.png" /><figcaption><strong>Figure</strong> Univariate and multivariate graphs for time series</figcaption></figure><p>Vector Autoregression is one of the multivariate forecasting algorithm. It uses <strong>two or more</strong> time series <strong>influence each other</strong>. In this model, each attribute is a linear combination of the past values of itself and the past values of all the other variables. These past values of the series are used to forecast the future.</p><p>Let’s see the structure of the VAR model.</p><p>The typical AR(p) model is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*Qq9_K2lF9eGPO8pHiPK_SQ.png" /></figure><p>And a simple VAR(p) model can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*hioLKtoaFH0gLucocnTBlg.png" /></figure><p>And VAR(2) process of the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*y_i0qCTTejbXFjyo8wOSnQ.png" /></figure><p>For this model, we have multiple time series that influence each other, it is modeled as a system of equations with one equation per time series. If we increase the number of time series in the model, the system of equations will be larger automatically. For example, VAR(3) will be as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*U0qumPHPyvWSEW7G4kWtTg.png" /></figure><p>To build a model, we have to check a few properties of our dataset. For determining whether one time series can be used for predicting another, <strong>Granger’s Causality Test</strong> is a good choice. It’s a relationship shifted in time. It provides information <strong>about how the two series move together.</strong> If the probability value is less than the significance level (of 0.05), then the hypothesis would be rejected at that level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*DdPVbeeWeXCAfGLaIVNM_w.png" /><figcaption><strong>Figure</strong> Examples show how the two series move together</figcaption></figure><p>To show the existence of a statistically significant connection between two or more time series <strong>Cointegration test</strong> is helpful. It is a statistical property of time series variables. All the series we have must be integrated of order d (number of differencing required to make a non-stationary time series stationary). If a linear combination of this collection is less than d, then the collection is said to be co-integrated and it means they have a statistically significant relationship.</p><p>After checking these features, we split the dataset into training and test data. Stationary is another important issue to build the model. How can we determine if our dataset is stationary or non- stationary?</p><p>A time series is said to be stationary if its mean, variance remain constant over time, not a function. Most of the time series models work on the assumption that the time series is stationary. There are three important check points:</p><p>1. Constant mean</p><p>2. Constant variance</p><p>3. An autovariance that does not depend on time</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*op3vRlxDmo8bJ0iCHz87FA.png" /><figcaption><strong>Figure </strong>Different graphs shows different cases about stationary</figcaption></figure><p>It is almost impossible to make a series %100 stationary, but we have to try to take it as close as possible. There are a few reasons behind non-stationarity of a time series. Two common reasons:</p><p>· Trend</p><p>· Seasonality</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*tLjtNRyFnphTErItN-HowQ.png" /></figure><p><strong>Figure </strong>Different graphs which show the different non-stationary situations</p><p>We can check stationary using the following unit root tests:</p><p>· <a href="https://www.machinelearningplus.com/time-series/augmented-dickey-fuller-test/">Augmented Dickey-Fuller Test (ADF Test)</a></p><p>· <a href="https://www.machinelearningplus.com/time-series/kpss-test-for-stationarity/">KPSS test</a></p><p>· Plotting Rolling Statistics</p><p>If a series is found to be non-stationary, you make it stationary by differencing the series once and repeat the test again until it becomes stationary. Differencing handles both trend and seasonality.</p><p><strong>VAR Order Selection</strong></p><p>Before we can estimate a multivariate VAR model we need to specify the order p. The most common approach for model order selection involves choosing a model order that minimizes one or more information criteria (AIC, BIC, HQC) evaluated over a range of model orders. These criteria are:</p><p>· Akaike Information Criterion (AIC)</p><p>· Bayesian Information Criterion (BIC)</p><p>· Hannan-Quinn Criterion (HQC)</p><p>Usually, the AIC is the best criteria due to its favorable small sample forecasting features. The BIC and HQ, however, work well in larger datasets and have the advantage of being a consistent estimator.</p><p>Let’s see some examples with tables include criteria with lag information.</p><p>We choose p=1, because the lowest values seem here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*LiutHcTr-8Xd7xWrxSmbRA.png" /></figure><p>We choose p=3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*LE7YMxfGAu5WFsg91oK-Pw.png" /></figure><p>To choose the right order of the VAR(p), we fit increasing orders of VAR model iteratively and select the order that gives a model with the least AIC. We can also check other best fit comparison estimates of BIC and HQIC.</p><p>The other testing issue is serial correlation of residuals. It is used to check if there is any leftover pattern in the errors. If there is any correlation left in the residuals, then, there is some pattern in the time series that is still left to be explained by the model. In that case, there are a few classical actions.</p><p>· Increase the order of the model — p</p><p>· Persuade more predictors into the system</p><p>· Search a different algorithm for modeling the problem</p><p>Durbin Watson’s Statistic is a typical approach of testing for serial correlation of residuals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*9OFB_n5jw59q8_sH0QzT1Q.png" /></figure><p>The value of DW can change between 0 and 4.</p><p>· Positive serial correlation if it is closer to 0</p><p>· No significant serial correlation if it is closer to 2</p><p>· Negative serial correlation if it is closer to 4</p><p>After doing these operations and determining the parameters like p, d, etc., the model will be ready to apply. Then forecasts will be generated, but it is on the scale of the training data will be used by the model. Hence, to bring it back up to its original scale and de-difference it as many times we had differenced the original input data.</p><p>References</p><p><a href="https://onlinelibrary.wiley.com/doi/10.1002/9781118625590.ch7">https://onlinelibrary.wiley.com/doi/10.1002/9781118625590.ch7</a></p><p><a href="https://en.wikipedia.org/wiki/Cointegration">https://en.wikipedia.org/wiki/Cointegration</a></p><p><a href="https://www.machinelearningplus.com/time-series/vector-autoregression-examples-python/">https://www.machinelearningplus.com/time-series/vector-autoregression-examples-python/</a></p><p><a href="https://www.r-econometrics.com/timeseries/varintro/">https://www.r-econometrics.com/timeseries/varintro/</a></p><p><a href="https://en.wikipedia.org/wiki/Vector_autoregression#:~:text=Vector%20autoregression%20(VAR)%20is%20a,allowing%20for%20multivariate%20time%20series">https://en.wikipedia.org/wiki/Vector_autoregression#:~:text=Vector%20autoregression%20(VAR)%20is%20a,allowing%20for%20multivariate%20time%20series</a>.</p><p>Lecture notes of Umberto Triacca — Lesson 18: Building a Vector Autoregressive Model</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9334d29591f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/vector-auto-regression-for-multivariate-time-series-forecasting-9334d29591f3">Vector Auto Regression for Multivariate Time Series Forecasting</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimization Algorithm for Deep Neural Networks: Conjugate Gradient Method]]></title>
            <link>https://medium.com/geekculture/optimization-algorithm-for-deep-neural-networks-conjugate-gradient-method-d17e25068775?source=rss-86b005addb18------2</link>
            <guid isPermaLink="false">https://medium.com/p/d17e25068775</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[conjugate-variables]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[deep-neural-networks]]></category>
            <category><![CDATA[numerical-analysis]]></category>
            <dc:creator><![CDATA[Ceyda Akbulut]]></dc:creator>
            <pubDate>Tue, 30 Mar 2021 22:17:03 GMT</pubDate>
            <atom:updated>2021-04-13T09:42:16.103Z</atom:updated>
            <content:encoded><![CDATA[<p>Deep neural networks became one of the standard way to solve different problems like image classification, text classification, natural language processing, computer vision…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*emeIyvLUCm1rv-ewPU7fRA.gif" /><figcaption><strong>Figure 1</strong> <em>Image classification example 1</em></figcaption></figure><p>A deep learning problem needs some features to be solved like loss function or optimizers. To solve, we have to define a loss function first as usual. After this operation, the optimization algorithm is ready to minimize the loss function. Let’s continue with the optimization part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJu3jZwHxsCbc3RTfTBPkQ.gif" /><figcaption><strong>Figure 2 </strong><em>Image classification example 2</em></figcaption></figure><p>Optimizers are sometimes algorithms or methods. They are really efficient ways to change some attributes like weights, learning rates in deep neural network training phase. Choosing correct/the best optimization technique is important to reduce the losses. The main idea of that is providing the most accurate results with the least losses. But, is it easy?</p><p>On these days, one of the big problems is dealing with large and sparse data matrices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*5ookQNjZ0FhUHOYc9wQJMw.png" /><figcaption><strong>Figure 3 </strong><em>Sparse matrix</em></figcaption></figure><p>Finding inverses of these matrices is also another difficult problem. The classical iterative methods cannot be a solution for these matrices. Thankfully, Krylov subspace methods are the useful methods in order to handle these types of problems. The fundamental of Krylov subspaces methods is one of the <strong>TOP 10</strong> algorithm ideas of the 20th century. They are used in algorithms to compute an approximate solution by transformations. These algorithms find an approximation of x by using dimension reduction (from n-dimensional vector space into a lower m dimensional vector space, m is smaller than n or equals to n). With these algorithms, there is no requirement to estimate the inverse of matrix A explicitly.</p><p>Arnoldi, Lanczos, Conjugate Gradient, Generalized Minimum Residual methods are popular Krylov subspace methods. We will focus on Conjugate Gradient Method.</p><p>The conjugate gradient method is a mathematical way for the optimization of both linear and non-linear systems. Hestenes and Stiefel introduced this method to us for minimizing convex quadratic functions,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*P5CRhFVW0h2StS9xfGtpDQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*U7g2FQfmj6dkpJixtMLUkw.png" /><figcaption><strong>Figure 4 </strong><em>Conjugate Gradient Algorithm Visualization</em></figcaption></figure><p>It is an efficient way to deal with a symmetric and sparse linear system, and it is easy to use. Conjugate gradient is in general the best performing iterative scheme without a multigrid component for solving linear system of equations with an s.p.d matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*yUjFrRd1cEaEfr9opJZf7g.png" /><figcaption>Ax=b</figcaption></figure><p>First, the algorithm chooses an initial position and walk along a direction for each step. The CG algorithm determines the next direction vector as a conjugate version of before. The related directions are determined sequentially at each step. At step i, after finding the current negative gradient vector, the algorithm adds to it a linear combination of the previous direction vectors.</p><p>There are various strategies how they choose the next search directions and step size belong to different methods. But, we know two important advantages of CG algorithm especially. One of the is simple formula to find the new direction vector. The other one is uniform progress.</p><p><strong>Algorithm: Conjugate Gradient Method</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*Xr8Uy3EzsTaW5oesScCasg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*5q-T12yNd4f2z5RYAOI1OQ.png" /></figure><p><strong>Cost of Conjugate Gradient</strong></p><p>o One CG iteration</p><p><em>· One matrix-vector multiplication (line 6)</em></p><p><em>· Three additions of vectors in (lines 8, 9, 11)</em></p><p><em>· Three multiplications of vectors with a scalar (lines 8, 9, 11)</em></p><p><em>· Two inner product of vectors (lines 7, 10)</em></p><p>o Store four vectors</p><p><strong>Code of Conjugate Gradient Method in MATLAB</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*C1rOOqH5dtStFhSyXm66Xg.png" /></figure><p>The disadvantage of the Conjugate Gradient Method is that it is useful for only symmetric positive definite matrices. For non-symmetric matrices, there are other methods/ways which I will mention later. Nevertheless, this algorithm is one of the best performing Krylov subspace methods.</p><p><strong>References</strong></p><p><a href="https://d2l.ai/chapter_optimization/optimization-intro.html">https://d2l.ai/chapter_optimization/optimization-intro.html</a></p><p><a href="https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6">https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6</a></p><p><a href="http://www.maths.lth.se/na/courses/NUM115/NUM115-05/krylov.pdf">http://www.maths.lth.se/na/courses/NUM115/NUM115-05/krylov.pdf</a></p><p><a href="https://arxiv.org/pdf/1811.09025.pdf">https://arxiv.org/pdf/1811.09025.pdf</a></p><p><a href="https://en.wikipedia.org/wiki/Conjugate_gradient_method">https://en.wikipedia.org/wiki/Conjugate_gradient_method</a></p><p><a href="https://web.cs.iastate.edu/~cs577/handouts/conjugate-gradient.pdf">https://web.cs.iastate.edu/~cs577/handouts/conjugate-gradient.pdf</a></p><p><a href="https://www.wias-berlin.de/people/john/LEHRE/lehre.html">https://www.wias-berlin.de/people/john/LEHRE/lehre.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d17e25068775" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/optimization-algorithm-for-deep-neural-networks-conjugate-gradient-method-d17e25068775">Optimization Algorithm for Deep Neural Networks: Conjugate Gradient Method</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>