<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Peter Oliver Caya on Medium]]></title>
        <description><![CDATA[Stories by Peter Oliver Caya on Medium]]></description>
        <link>https://medium.com/@peterolivercaya?source=rss-3953329470ae------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*J1H8de_UWyriB1iOeW-wqg.jpeg</url>
            <title>Stories by Peter Oliver Caya on Medium</title>
            <link>https://medium.com/@peterolivercaya?source=rss-3953329470ae------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:32:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@peterolivercaya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data transformations for modeling loan default]]></title>
            <link>https://medium.com/risk-and-statistical-learning/data-transformations-for-modeling-loan-default-1fe1dbab2be9?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fe1dbab2be9</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[economics]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Tue, 27 Sep 2022 09:51:50 GMT</pubDate>
            <atom:updated>2022-09-27T09:51:50.167Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eh9rmemrP1UPTnMmmbxxhA.png" /></figure><p>I’ve been doing some work in my free time to create models which forecast loan performance for banks using macro-economic variables. Outside of obtaining bank level data on loan performance, the most crucial part of this has been choosing what economic data to include as features — more specifically, what steps need to be taken to ensure the stability and out-of-sample performance of the model.</p><p>Macro-economic data is free, widely available, comes from reputable sources, and is clean but the trends and distribution of the data means each series needs to be evaluated in context to the modeling method to decide if transformations are necessary. Ignoring the underlying theory of the model method can lead to a false sense of security, degradation of the model overtime, or countintuitive model theory.</p><p>This post will highlight the effects that can come from using untransformed data. Some common transformations will also be discussed. Finally, the method of fractional differencing will be discussed as well as its motivation and potential advantages over more common transformations.</p><h4>Motivation for transforming data series</h4><p>To start, we can consider the home price index (HPI) and the unemployment rate as key features to performance of residential real estate loans. These features are both easily accessed and intuitively connected to events like loan delinquency. I’ve downloaded the economic data series from FRED using the fredr library. I downloaded the residential loans that are 90+ DPD from the <a href="https://www.fhfa.gov/DataTools/Downloads/Pages/National-Mortgage-Database-Aggregate-Data.aspx">FHFA’s National Mortgage Database page</a>. I’ve plotted the results below for comparison. Because these three series have very different scales, I’ve normalized the data so that it fits on the same scale for the purpose of comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhcjffkfyvA1FY5vBv_Vew.png" /><figcaption>Comparison of 90+ DPD with relevant economic data</figcaption></figure><p>We can see from the chart that there is a clear relationship: As unemployment climbs and HPI falls, the rate of 90+ DPD increases. This matches our theoretical expectations. The clear relationship makes fitting a linear regression model a tempting solution but this approach will also lead to the structure of the data to influence the regression coefficients for our model.</p><p>Consider HPI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29RLn3QXaVoUzncOu-cMjA.png" /><figcaption>Plot of historical HPI</figcaption></figure><p>We can see that the HPI values vary but there is almost always an upwards trend. Fitting a model on this data will like cause it to be overfit on the time-period and the quality of the results is prone to degrading outside the training range. This is especially likely if there is a decline in HPI which is problematic for risk models because the model accuracy would degrade exactly where it would be needed most!</p><p>Linear regression will only assess the average impact of the HPI on the rate pf 90+ DPD and will lack sensitivity to rises or drops. This is also a problem for theory since in the past the trigger for high 90+ DPD loans has been a sharp drop in HPI, not the level of the variable itself.</p><h4>Theoretical issues for macro-economic data</h4><p>The untransformed data may also be problematic for reasons that conflict with model theory and degrade performance. Linear regression has several expectations of the data including:</p><ol><li>Linearity between the variables used for the model and the dependent variable.</li><li>Full rank — none of the features is composed of a linear combination of one or more other features.</li><li>The error terms for the model are not a function of the independent variables used to construct it.</li><li>The values of the data can’t be explained using earlier data points. IE, I can’t predict the current HPI value using last quarter’s value.</li><li>A constant variance is maintained in the data.</li></ol><p>Two important components that often lead to all other problems in macro-economic data are autocorrelation and nonstationarity which are described in points 4 and 5.</p><p>If we fit a simple model on an auto-correlated time-series we will have an important component of the the input data that we aren’t accounting for which will impact accuracy and model behavior. The theory of many models also assumes that the data is stationary; That the data has a constant variance over time with no trend.</p><p>I’ve created two charts below to show that the HPI series contains both of these elements. The first series plots the correlation between the HPI data with itself over a set of different lags. The second chart is a plot of the variance over the life of the HPI data. We can easily see that as time goes on, the variance rises.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S3I4tB_lojg0kAVnWEAgyg.png" /><figcaption>ACF plot of HPI data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gM2dr7OJAV72y1sG3jRvoA.png" /><figcaption>Plot of HPI series variance using a window of 8 quarters</figcaption></figure><p>There is a rich and well-developed literature on detecting and interpreting autocorrelation and nonstationarity so I won’t perform a deep dive on the subject. To read further, I recommend <a href="https://bookdown.org/rdpeng/timeseriesbook/stationarity.html">A Very Short Course on Time Series Analysis</a> and <a href="https://www.amazon.com/Econometric-Analysis-8th-William-Greene/dp/0134461363">Econometric Analysis</a>.</p><p>Now that we’ve explored these considerations, it’s time to determine how we can transform this data to create a more accurate and interpretable model.</p><h4>Commonly used data transformations</h4><p>We’ve discussed the impact that untransformed data can have on attempts to model data. The next step is to discuss how we can remove the structural issues in the data for use in modeling. We’ll start by reviewing the most common variable transformations and review the impact that these transformations have.</p><p>The most commonly used transformation is simple differencing of the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/0*TSzo3_tv3l2dBCsn" /></figure><p>A simple alternative would be to use the percentage change of the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/119/0*KWkdKx02TzabVXvP" /></figure><p>A common example is the transformation of the home price index into home price appreciation (HPA):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QcImTgnMEscQ7grh9ujzHw.png" /><figcaption>Comparison of HPI, differenced HPI, and HPA</figcaption></figure><p>We can see from the comparison above that there is a lower level of autocorrelation in the HPA data than in the untransformed data.</p><h4>Fractional differencing</h4><p>The transformations described in the previous section are effective at removing problematic data qualities like autocorrelation and making the data series stationary, but also have the impact of making the transformed data lose its resemblance to the original data series. This is typically not problematic but it may influence the accuracy of the model that’s ultimately generated.</p><p>To arrive at the concept of fractional differencing, we start with the formula for 1 period differencing shown earlier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/0*TSzo3_tv3l2dBCsn" /></figure><p>Let’s define a differencing operator below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/95/0*xFW2qSvBoDzNdxIy" /></figure><p>The differencing formula from earlier becomes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/0*ylzzSaE-Iuue2HfR" /></figure><p>Defining B allows us to treat it algebraically. We can utilize the binomial series with <em>d entries:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/0*PtxoxCY6ozLbDxNY" /></figure><p>This series can be used to create our list of weights for lags:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*9e465oW9RUIDJ3xE" /></figure><p>The weights can be simplified to the equation below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/0*pZ7ZUSEtXFcvspGr" /><figcaption>Recursive equation for lag weights</figcaption></figure><p>We can leverage the framework above to difference data in a manner which isn’t based only on integers (lag of 1 or 2). This strategy is referred to as fractional differencing and it allows users to balance between simply differencing the data and leaving it untransformed.</p><p>Consider a trivial example where we use a difference of 1. In this case the weights become:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/0*GP6kmn4a7NPPQ52y" /></figure><p>We apply these values as weights to transform the original data which yields:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/0*9leKsDAq_AYSVFIL" /></figure><p>A more useful example is letting d=.4 which produces the following weights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/0*5BYVArRzCF_oYSL6" /><figcaption>Weights to be used when d = .4</figcaption></figure><p>I’ve implemented this rule in the code below (adapted to R from Marcos Lopez de Prado’s Python code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4833d84a95a50293c7aa604a6b66746d/href">https://medium.com/media/4833d84a95a50293c7aa604a6b66746d/href</a></iframe><p>Using the code above, we can demonstrate the way the lag weights change as the parameter<em> d </em>is varied:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UoniJogcSoa3up6JaFPGGQ.png" /></figure><p>Note that the weights trail off as the lag values get larger. This demonstrates that data from the past is still being factored in. The lower the d value, the more weight will be given to observations in the past when compared to recent observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MCjyg9OwVzL_qyoWVIqubQ.png" /></figure><p>For practical purposes we truncate the weights. This is a fixed fractional window. You can find my code implementing the fixed fractional window, and applying it to data in below. This strategy is again adapted to R from de Prado’s book.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fc38f650b5ca13d4b31a46cac02325e/href">https://medium.com/media/2fc38f650b5ca13d4b31a46cac02325e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9354a553d15749d80aae645120e44026/href">https://medium.com/media/9354a553d15749d80aae645120e44026/href</a></iframe><h4>Procedure</h4><p>The discussion above raises the question: How do we select d? Here, I’ve implemented and presented results based off of Marcos Lopez de Prado’s book which suggests the following framework:</p><ol><li>Create alternate sets of fractionally differenced data using a range of lags, <em>d.</em></li><li>Use the Augmented Dickey-Fuller (ADF) test over each set of transformed data.</li><li>Select the lowest d-value allows us to reject the null hypothesis of non-stationarity.</li></ol><p>I’ve implemented these steps in the function below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/385b8457a2faec6ef0cc3982f62c1e4a/href">https://medium.com/media/385b8457a2faec6ef0cc3982f62c1e4a/href</a></iframe><p>While the framework shown above is straightforward, it isn’t the final word on what amount of fractional differencing to use. For instance, we could use the KPSS test instead of ADF. I’ve plotted an example of the output when used on a monthly series of data from the SP 500 below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LVXOmyGkWNAVsHg52nEo2Q.png" /></figure><p>I used this heuristic to select values for d for monthly data for HPI and SP500. The results are presented in the charts below along with the untransformed data and the data after simple differencing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POj1cf4Z89oCtBsGT918Ng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SeMMe5VLaNWG2nd371sl8A.png" /></figure><p>We can see two separate outcomes in the charts above. For HPI — it actually took a fractional difference <em>greater </em>than 1 to remove stationarity. We arrive at a value which is very similar to simple differencing. For the SP500, d = .34. When we use this value of d in the fractional differencing function we arrive at a series which is stationary, but also contains the general trend information originally present in the untransformed data.</p><h4>Conclusion</h4><p>The most important component of building any model is to ensure that quality data is used appropriately. High quality macro-economic data can easily be found, but there are practical and technical barriers to using it to directly create forecasts. Nonstationary data which has autocorrelation or other qualities can create models that are missing valuable information at best. At worst, the data can create models which give a false sense of security and aren’t fit to forecast outside of the training range.</p><p>Econometric literature provides a wide range of alternative methods that can be used by a practitioner with domain knowledge to generate features which are intuitive and effective at explaining the phenomenon. I discussed a few of the more basic strategies like differencing.</p><p>While strategies like differencing and taking the percentage change are intuitive and often effective at removing stationarity and autocorrelation, they also sacrifice information about the general information of the data series that might be useful in the model. Fortunately, Hosking introduced a strategy called fractional differencing which allows a modeler to strike a balance between preserving the general trend of the data while still removing problematic elements. In this post, I used notes from Marco Lopez de Prado’s book to create a practical implementation of this strategy and compare it to more standard strategies.</p><p>The results showed that in some cases, it may help the modeler preserve the general shape of the feature while removing autocorrelation and enforcing stationarity. Using the Augmented Dickey-Fuller test to determine what parameters balance between transforming the data and preserving memory provides a useful heuristic, but other strategies like using autocorrelation or other tests of stationarity may also be useful.</p><h4>Resources and References</h4><p>All code used to generate analysis for this post is located in this post’s <a href="https://github.com/poc1673/Fractional-Differencing-Post-Details">repository</a>.</p><p>I consulted the following books, articles, and blog posts for this article:</p><ol><li><a href="https://www.amazon.com/Advances-Financial-Machine-Learning-Marcos/dp/1119482089">Advances in Financial Machine Learning</a> — Marcos Lopez de Prado</li><li><a href="https://arxiv.org/pdf/2005.04897.pdf">Macroeconomic Forecasting with Fractional Factor <br>Models</a> — Tobias Hart</li><li><a href="https://towardsdatascience.com/preserving-memory-in-stationary-time-series-6842f7581800">Preserving Memory in Stationary Time Series</a> — Simon Kuttruf</li><li><a href="https://www.ma.imperial.ac.uk/~ejm/M3S8/Problems/hosking81.pdf">Fractional Differencing</a> — J.R.M. Hosking</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fe1dbab2be9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/risk-and-statistical-learning/data-transformations-for-modeling-loan-default-1fe1dbab2be9">Data transformations for modeling loan default</a> was originally published in <a href="https://medium.com/risk-and-statistical-learning">Peter Caya’s Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boosting performance by combining trees with GLM: A benchmarking analysis]]></title>
            <link>https://medium.com/data-science/boosting-performance-by-combining-trees-with-glm-a-benchmarking-analysis-1840cac91cb1?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/1840cac91cb1</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[machine-lear]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r-programming]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sat, 26 Jun 2021 21:34:27 GMT</pubDate>
            <atom:updated>2021-06-27T03:34:49.013Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LOtE9dfHescrvuTZ" /><figcaption>Photo by <a href="https://unsplash.com/@8moments?utm_source=medium&amp;utm_medium=referral">Simon Berger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>How much of an improvement is gained by combining trees with GLM, and how does it compare to additive models?</h4><p>A common pitfall of statistical modeling is ensuring the modeling method is appropriate to the structure of the data. linear models like logistic regression assume the existence of a linear relationship between the likelihood of the predicted event, and of the independent variables. While researching the topic recently, I came across <a href="https://stackoverflow.com/questions/41692017/decision-trees-combined-with-logistic-regression">this</a> article on StackExchange discussing the use of shallow decision trees as a feature engineering step for logistic regression. Briefly, this strategy attempts to cope with nonlinear data by using trees as a feature engineering step to transform it into dummy variables that can then be used in the logistic regression specification.</p><p>After researching this strategy, my impression was that performing this introductory step may introduce the drawback of reducing the flexibility of the model while offering a marginal increase in model results. It may be more useful to either transform the data to remove the nonlinearities, or to account for them in the specification of the model. However, I was surprised to see that there weren’t many articles or posts comparing it to other methods!</p><p>To see how decision trees combined with logistic regression (tree+GLM) performs, I’ve tested the method on three data sets and benchmarked the results against standard logistic regression and a generalized additive model (GAM) to see if there is a consistent performance difference between the two methods.</p><h3>The Tree + GLM Methodology</h3><p>Logistic regression and decision trees are generally the first two classification models one is introduced to. Each has its own pitfall. Regression models assume the dependent variable can be explained using a set of linear functions applied to the independent variables and have an equation of the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/132/0*NANTB-pQD9svLu7z" /></figure><p>Decision trees don’t make an assumption on the distribution of variables, they simply create a new branch in the decision tree based on criteria like Gini impurity. However, decision trees are prone to overfitting and can be unstable to changes in the training data.</p><p>The chart below illustrates the results for linear regression and a decision tree models over the range of a dependent variable in a one-variable model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vpJeQA1fzQXH_46sedACLQ.png" /><figcaption>Image by author</figcaption></figure><p>From the chart above, we can see that the predictions from the decision tree are discontinuous in contrast to the smooth function generated by the linear regression model. While this will be problematic for simple linear data, the ability of the decision tree strategy to change in a nonlinear fashion provides justification for its use on nonlinear data.</p><p>To try to remedy the downsides of these two methods, several sources have suggested using a decision tree as an intermediate step which helps remove potential nonlinearity in the model. In its most simple form, the process is as follows:</p><ol><li>Fit a shallow decision tree, T(X) for the training data, X. This tree will have N terminal nodes.</li><li>N categorical variables denoted C_n are included as features in the logistic regression specification.</li><li>The logistic regression is fit using the modified set of data.</li></ol><p>There are two simple alternatives to handling nonlinearity in data. The first choice is to use some other transformation step on the raw data to give it a linear relationship with the dependent variable. This strategy isn’t always an option though, and its appropriateness varies based on the domain. The second way to eliminate it is to change the model specification to a method that can cope with nonlinear data.</p><h3>Generalized additive models</h3><p>There are several methodologies that can be used to cope with nonlinear data, but the one I chose for this exercise is the generalized additive model. Generalized additive models are a framework originally introduced in “<a href="https://web.stanford.edu/~hastie/Papers/gam.pdf">Generalized Additive Models</a>” by Trevor Hastie and Robert Tibshirani. The authors went on to write <a href="https://web.stanford.edu/~hastie/ElemStatLearn/">Elements of Statistical Learning</a> which is where I first encountered it.</p><p>It takes the concept of the generalized linear model that logistic regression is based off, and relaxes the assumption of linear basis functions. In the case of the logistic regression specification, the GAM equivalent for logistic regression would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/0*sBbbab_63FTO2TS5" /></figure><p>In this case, the notation above substitutes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/37/0*j26g4MLbaBfY6w7V" /></figure><p>for</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/14/0*TpRGH_5Qzn0fRvmv" /></figure><p>in the original logistic regression equation. This is because we have changed the term operating on each dependent variable to an arbitrary smooth function. Alternatively, we could think of a logistic regression model as an additive model where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/90/0*RXNoBbYVGbZ-MORw" /></figure><p>For some regression coefficient</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/12/0*-uZfsq8jqB8FAohA" /></figure><p>It is clear that this offers us additional flexibility in how we treat the variables that we are modeling, so we will use this specification in our numerical experiment as a challenger to the tree+logistic regression specification.</p><h3>Data</h3><p>Three data sets were used for the comparisons performed in this post These were:</p><ol><li>Synthetic data set — An artificial data set containing 10,000 observations created to include linear and nonlinear variables.</li><li>Banking data — A data set generated by a banking campaign. The data utilized here was actually cleaned and formatted by Andrzej Szymanski, Ph.D. who wrote an <a href="https://towardsdatascience.com/combining-logistic-regression-and-decision-tree-1adec36a4b3f">article </a>on using logistic regression with decision trees. I wanted to include this as it could provide a valuable comparison to results that had been generated before. The original data is drawn from <a href="https://github.com/AndrzejSzymanski/TDS/blob/master/banking.csv">Andrzej’s</a> GitHub.</li><li>Adult data — Census data used to attempt to predict whether income exceeds $50,000/year. It is available at the UCI Machine Learning Repository <a href="https://archive.ics.uci.edu/ml/datasets/adult">here</a>.</li></ol><p>In the below code chunk, we collect the data that’s going to be used in our three tests and examine it. We begin by generating the synthetic data, and then downloading and formatting the Banking and Adult data sets. After this, we perform some data cleaning steps to make sure that the variables we are using are not represented as characters. Finally, we break the data into training and testing data:</p><pre>rm(list = ls())<br>library(pacman)<br>p_load(data.table,caret,ggplot2,plotly,mgcv,rpart,magrittr,precrec,MLmetrics,partykit,gam,rmarkdown,knitr,broom,rpart.plot,reactable)<br>set.seed(13)<br>with.nonlinear &lt;- data.table(twoClassSim(10000,<br>                                         linearVars = 6,ordinal = F ) )<br>data.list &lt;- list(&quot;Synthetic&quot; = with.nonlinear,<br>                  &quot;Banking&quot; =  fread(&quot;<a href="https://raw.githubusercontent.com/AndrzejSzymanski/TDS-LR-DT/master/banking_cleansed.csv">https://raw.githubusercontent.com/AndrzejSzymanski/TDS-LR-DT/master/banking_cleansed.csv</a>&quot;)  ,<br>                  &quot;Adult&quot; = fread(&quot;<a href="https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data">https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data</a>&quot;)  )<br>names(data.list[[3]]) &lt;- c(&quot;age&quot;,&quot;workclass&quot;,&quot;fnlwgt&quot;,&quot;education&quot;,&quot;education-num&quot;,&quot;marital-status&quot;,&quot;occupation&quot;, &quot;relationship&quot;, &quot;race&quot;,&quot;sex&quot;,&quot;capital-gain&quot;,&quot;capital-loss&quot;,&quot;hours-per-week&quot;,&quot;native-country&quot;,&quot;Income-class&quot;)<br># Change variables to factors for banking data-set<br>banking.names.for.factors &lt;- names(data.list$Banking)[apply(data.list$Banking,MARGIN = 2, function(x){length(unique(x))})==2]<br>data.list$Banking[,<br>                  names(data.list$Banking)[apply(data.list$Banking,MARGIN = 2, function(x){<br>                    length(unique(x))})==2] := lapply(X = .SD,FUN = factor),<br>                  .SDcols = banking.names.for.factors]<br># Change variables to factors for Adult data-set:<br>adult.names.for.factors &lt;- names(data.list$Adult)[sapply(X = 1:ncol(data.list$Adult),function(x){is.character( data.list$Adult[[x]]  )})]<br>data.list$Adult[,names(data.list$Adult)[sapply(X = 1:ncol(data.list$Adult),<br>                                               function(x){is.character( data.list$Adult[[x]]  )})]:= lapply(X = .SD,FUN = factor),<br>                .SDcols = adult.names.for.factors]<br>data.list$Adult[,names(data.list$Adult)[sapply(X = 1:ncol(data.list$Adult),<br>                                               function(x){is.integer( data.list$Adult[[x]]  )})]:= lapply(X = .SD,FUN = as.numeric ),<br>                .SDcols = names(data.list$Adult)[sapply(X = 1:ncol(data.list$Adult),<br>                                                        function(x){is.integer( data.list$Adult[[x]]  )})]]<br>training.data &lt;- list()<br>test.data &lt;- list()<br>for( i in 1:length(data.list)){<br>  train_inds &lt;- sample(x = 1:nrow(data.list[[i]]) ,size = .8*nrow(data.list[[i]]))<br>  training.data[[i]] &lt;- data.list[[i]][train_inds]<br>  test.data[[i]] &lt;-   data.list[[i]][-train_inds]  }<br>names(training.data)&lt;- names(data.list)<br>names(test.data)&lt;- names(data.list)</pre><p>Before engaging in our benchmark analysis, it’s useful to understand which variables have a nonlinear relationship with the dependent variable. A simple and intuitive way to do this is to chart the relationship between the dependent and independent variables to see if a nonlinear relationship is obvious from a visual inspection. In the case of the implementation here, I am using a GAM with one variable to describe the general shape of the relationship between the two variables. You can see the implementation below which generates assesses the nonlinearity using charts:</p><pre>nonlinear.viz &lt;- function(dt,dep.var,indep.var,bins = 100){<br>  dt$y &lt;- as.numeric(as.character(dt$y))<br>  return.plot &lt;- ggplot(dt, <br>                        aes_string( x = indep.var,y = dep.var) ) + stat_smooth(method = &quot;gam&quot;, <br>                                                                               method.args = list(family = &quot;binomial&quot;),) + theme_bw() + theme(axis.title = element_text(size = 16),axis.text = element_text(size = 11),plot.title = element_text(size = 20)   ) +ylab(indep.var)+<br>    xlab(dep.var)+<br>    ggtitle(paste(&quot;Relationship between &quot;,dep.var,&quot; and &quot;, indep.var,sep = &quot;&quot;))<br>  return(return.plot)}</pre><pre>training.data$Synthetic[,Class := ifelse(Class == &quot;Class1&quot;, 1,0)]<br>test.data$Synthetic[,Class := ifelse(Class == &quot;Class1&quot;, 1,0)]<br>training.data$Adult[,&quot;Class&quot; := ifelse(`Income-class` == &quot;&gt;50K&quot;, 1,0)]<br>test.data$Adult[,&quot;Class&quot; := ifelse(`Income-class` == &quot;&gt;50K&quot;, 1,0)]<br>synthetic.plot &lt;- lapply(X = names(training.data$Synthetic)[names(training.data$Synthetic)!=&quot;Class&quot;],<br>                         function(x){nonlinear.viz(dt = training.data$Synthetic,dep.var = &quot;Class&quot;,x)})<br>banking.plot &lt;- lapply(X = c(&quot;age&quot;,&quot;previous&quot;,&quot;euribor3m&quot;,&quot;cons_conf_idx&quot;,&quot;cons_price_idx&quot;,&quot;nr_employed&quot;, &quot;emp_var_rate&quot;   ,   &quot;pdays&quot;),<br>                       function(x){nonlinear.viz(dt = training.data$Banking,dep.var = &quot;y&quot;,x)})<br>names(training.data$Adult) &lt;- gsub(names(training.data$Adult),pattern = &quot;-&quot;,replacement = &quot;_&quot;)<br>names(test.data$Adult) &lt;- names(training.data$Adult)<br>adult.plot &lt;- lapply(X = c(&quot;age&quot;,&quot;education_num&quot;,&quot;capital_gain&quot;,&quot;capital_loss&quot;,&quot;hours_per_week&quot;),<br>                     function(x){nonlinear.viz(dt = training.data$Adult,<br>                                               dep.var = &quot;Class&quot;,x )})</pre><p>The code above will generate charts for <em>all</em> variable relationships. For the sake of brevity, I concentrate only on nonlinear data in the sections below:</p><p><strong>Synthetic data-set</strong></p><p>The synthetic data set that’s being used is generated from the <a href="https://rdrr.io/cran/caret/man/twoClassSim.html">twoClassSim </a>function from the caret library. This function is straightforward: A data-set is generated with a binary outcome and a set of variables that are either linearly related, or non-linearly related. This data set is useful for our tests because it allows us to compare the algorithms without being concerned with information regarding the domain of the problem. The relationship between variables is summarized in the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*evr-a-FVwf2O9QvgnaBYDw.png" /><figcaption>Image by author</figcaption></figure><p>I’ve plotted the nonlinear variables for the synthetic data against the likelihood of the class in the data-set using the below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*RD21UfcDm9n0zc0cJJdAzA.png" /><figcaption>Image by author</figcaption></figure><p>We can see that the variables are what we would expect: In all cases, there is a generally negative nonlinear relationship exists between the likelihood of an instance being a positive class, and the likelihood of a negative class.</p><p><strong>Banking data-set</strong></p><p>The banking data set contains 27 independent variables. Of these, 7 of these variables are continuous variables with the remaining being binary variables. The feature the relationship to the dependent variable are shown in the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q92HjtDmcC51E8Ykl8k43g.png" /><figcaption>Image by author</figcaption></figure><p>As with the synthetic data, I’ve also produced a set of charts showing the relationship between the continuous feature variables and the dependent variable (in this case, y):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oA25ItOBEv-hCvmkTac7pA.png" /><figcaption>Image by author</figcaption></figure><p>We can see that in each of the example charts above, there is a pronounced nonlinear relationship between the dependent and independent variable that would be difficult to compensate for using a linear basis function.</p><p><strong>Adult data-set</strong></p><p>As with the banking data-set, the adult data-set has many variables which are binary variables along with a set of continuous variables (in this case, five). They are summarized in the table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5wOTfjsX17_mFEvWrTqug.png" /><figcaption>Image by author</figcaption></figure><p>I’ve also created the chart below which illustrates the relationship between each variable and the dependent variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6wUd1-Dt35XUq909UCv39g.png" /><figcaption>Image by author</figcaption></figure><p>The relationship between the continuous independent variables and the dependent variable similar to the banking data-set. The sole exception is the education_num variable which represents the number of years in school. We can see that it has a clearly positive relationship and might be approximated with a linear function, despite the concavity.</p><h4>Balance of data</h4><p>It’s helpful to review the balance between classes in the data sets before fitting and experimenting with models. Class imbalance in data can introduce bias that may have to be addressed using resampling techniques.</p><p>We’ll investigate the imbalance using the code displayed below:</p><pre>sum.func &lt;- function(x,col,model.name){ <br>  dims &lt;- data.table( &quot;Measure&quot; = c(&quot;Observations&quot;,&quot;Factors&quot;)  , &quot;Count&quot; = dim(x))<br>  factors &lt;- data.table( &quot;Minority Class Percent&quot;,  min(x[,.N,by = col]$N)/sum(x[,.N,by = col]$N)  )<br>  names(factors) &lt;- c(&quot;Measure&quot;,&quot;Count&quot;)<br>  for.return &lt;- data.table(rbind(dims,factors)   )<br>  names(for.return)[2] &lt;- model.name<br>  # factors$Measure &lt;- paste(&quot;Class = &quot;,factors$Measure,sep = &quot;&quot;)<br>  return(  for.return    )  }<br>dep.vars &lt;- c(&quot;Class&quot;, &quot;y&quot;, &quot;Income_class&quot;)</pre><pre>summaries &lt;- lapply(X = 1:length(training.data),<br>                    FUN = function(x){sum.func(training.data[[x]], dep.vars[x] ,<br>                                               model.name = names(training.data)[x])   })<br>summaries[[1]]$Banking &lt;- summaries[[2]]$Banking<br>summaries[[1]]$Adult &lt;- summaries[[3]]$Adult<br>kable(summaries[[1]],digits = 3)</pre><p>I’ve formatted the results of the table to show the size of the minority class of the dependent variable compared to the rest of the data set below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*5_iHlV5PX4cUgB0y_UdLaQ.png" /><figcaption>Image by author</figcaption></figure><p>We can see from the table above that the Banking and Adult data sets contain a significant imbalance for the prediction classes. To correct this, we will use the synthetic minority oversampling technique and review the results.</p><p>A quick note: The SMOTE step may take some time.</p><pre>banking.smote &lt;- RSBID::SMOTE_NC(data = training.data$Banking , outcome = &quot;y&quot; )<br>adult.smote &lt;- RSBID::SMOTE_NC(data = training.data$Adult, outcome = &quot;Income_class&quot; )</pre><pre>p_load(data.table,rmarkdown,knitr)<br>resampled_data &lt;- list(training.data$Synthetic,banking.smote,adult.smote)<br>dep.vars &lt;- c(&quot;Class&quot;, &quot;y&quot;, &quot;Income_class&quot;)<br>names(resampled_data) &lt;- names(training.data)<br>summaries &lt;- lapply(X = 1:length(training.data),<br>                    FUN = function(x){sum.func(resampled_data[[x]], <br>                                               dep.vars[x] ,<br>                                               model.name = names(training.data)[x])   })<br>summaries[[1]]$Banking &lt;- summaries[[2]]$Banking<br>summaries[[1]]$Adult &lt;- summaries[[3]]$Adult<br>kable(summaries[[1]],digits = 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*N9Cs98iVKZ9Wb5LgWRug0w.png" /></figure><h3>Building Decision Trees</h3><p>We begin by building simple decision trees to use as a factor <em>instead</em> of the original categorical variables. These trees will have to be small in order to avoid overfitting. I’ve tested models using trees with maximum depths of two, three, and four and chosen on using a maximum depth of four in each model based on its improved accuracy in ROC and PR AUC. In the code chunk below, CART decision trees are fitted with the leaf node prediction then being used to create a new factor variable in the data. After that, the new data is used to fit a logistic regression model.</p><p>We first fit the initial decision trees (another note: I tested this with several iterations of depth and other details. The choices below seemed to perform the best):</p><pre>synthetic.tree.2 &lt;- rpart(data = training.data$Synthetic,<br>                          formula = Class~Linear1+Linear2+Linear3+Linear4+Linear5+Linear6+Nonlinear1 +Nonlinear2 +Nonlinear3,<br>                          control = rpart.control(  maxdepth = 4 ))<br>banking.tree.3 &lt;- rpart(data = banking.smote,<br>                        formula = y~.-V1, control = rpart.control(  maxdepth = 4)  )<br>adult.tree.3 &lt;- rpart(data = adult.smote,<br>                      formula = Class~age+workclass+fnlwgt+education+education_num +marital_status+occupation+relationship+race+sex+capital_gain +  capital_loss + hours_per_week+ native_country,<br>                      control = rpart.control(  maxdepth = 4))</pre><pre>synth.models &lt;- list(synthetic.tree.2)<br>banking.models &lt;- list(banking.tree.3)<br>adult.models &lt;- list(adult.tree.3)</pre><pre>tree.to.feature &lt;- function(tree.model,dt){<br>  require(partykit)<br>  tree_labels &lt;- factor( predict(as.party(tree.model), dt ,type = &quot;node&quot;) )<br>  return(tree_labels)}</pre><p>Next, we take these results and use them to fit the logistic regression models:</p><pre>synth.train.preds &lt;- lapply(X = synth.models,FUN = function(x){   tree.to.feature(tree.model = x,dt = training.data$Synthetic)  }) %&gt;% data.frame<br>banking.train.preds &lt;-lapply(X = banking.models,FUN = function(x){   tree.to.feature(tree.model = x,dt = training.data$Banking)  }) %&gt;% data.frame<br>adult.train.preds &lt;-lapply(X = adult.models,FUN = function(x){   tree.to.feature(tree.model = x,dt = training.data$Adult)  }) %&gt;% data.frame<br>names(synth.train.preds) &lt;- c(&quot;three.nodes&quot;)<br>names(banking.train.preds) &lt;- c(&quot;four.nodes&quot;)<br>names(adult.train.preds) &lt;- c(&quot;four.nodes&quot;)<br>training.data$Synthetic &lt;- cbind( training.data$Synthetic,synth.train.preds )<br>training.data$Banking &lt;- cbind(training.data$Banking ,banking.train.preds )<br>training.data$Adult &lt;- cbind(training.data$Adult ,adult.train.preds )<br>synth.model.three.deep&lt;- glm(formula = Class~Linear1+Linear2+Linear3+Linear4+Linear5+Linear6+Nonlinear1 +Nonlinear2 +Nonlinear3 + three.nodes,family = &quot;binomial&quot;,data = training.data$Synthetic)<br>banking.mode.four.deep &lt;- glm(formula = y~.-V1 ,family = &quot;binomial&quot;,data = training.data$Banking)<br>adult.mode.four.deep &lt;- glm(formula = Class~age+workclass+fnlwgt+education+education_num +marital_status+occupation+relationship+race+sex+capital_gain +  capital_loss + hours_per_week+ native_country + four.nodes,<br>                            family = &quot;binomial&quot;,<br>                            data = training.data$Adult)</pre><p>Finally, we fit the GAM and GLM models which we will use to benchmark the GLM+Tree model results:</p><pre># Create GAM models and the GLM models:<br>synth.gam &lt;- gam(data = training.data$Synthetic,formula = Class~s(Linear1)+s(Linear2)+s(Linear3)+s(Linear4)+s(Linear5)+s(Linear6)+s(Nonlinear1) +s(Nonlinear2) +s(Nonlinear3),family = binomial)</pre><pre>banking.gam &lt;- gam(formula = y~s(age)+s(previous)+s(euribor3m)+s(cons_conf_idx)+s(cons_price_idx)+s(nr_employed)+s(emp_var_rate)+s(pdays)+`job_blue-collar`+<br>                     job_management+`job_other 1`+`job_other 2`+job_services+job_technician+marital_married+marital_single+ education_high.school+education_professional.course+education_university.degree+education_unknown+default_unknown+housing_unknown+<br>                     housing_yes+loan_unknown+loan_yes+poutcome_nonexistent+poutcome_success,family = &quot;binomial&quot;,data = training.data$Banking)</pre><pre>adult.gam &lt;- gam(formula = Class~s(age)+(workclass)+s(fnlwgt)+(education)+s(education_num) +(marital_status)+(occupation)+(relationship)+(race)+(sex)+s(capital_gain) +  s(capital_loss) + s(hours_per_week)+ (native_country),<br>                 family = &quot;binomial&quot;,<br>                 data = training.data$Adult)<br># Create GLM Models</pre><pre>synth.model.glm&lt;- glm(formula = Class~Linear1+Linear2+Linear3+Linear4+Linear5+Linear6+Nonlinear1 +Nonlinear2 +Nonlinear3 ,family = &quot;binomial&quot;,data = training.data$Synthetic)</pre><pre>banking.mode.glm &lt;- glm(formula = y~.-V1-four.nodes ,family = &quot;binomial&quot;,data = training.data$Banking)</pre><pre>adult.mode.glm &lt;- glm(formula = Class~age+workclass+fnlwgt+education+education_num +marital_status+occupation+relationship+race+sex+capital_gain +  capital_loss + hours_per_week+ native_country ,<br>                      family = &quot;binomial&quot;,<br>                      data = training.data$Adult)Finally, we fit the GAM and GLM models which we will be using to benchmark the GLM+Tree model results:</pre><pre><em># Create GAM models and the GLM models:</em><br>synth.gam <strong>&lt;-</strong> gam(data <strong>=</strong> training.data<strong>$</strong>Synthetic,formula <strong>=</strong> Class<strong>~</strong>s(Linear1)<strong>+</strong>s(Linear2)<strong>+</strong>s(Linear3)<strong>+</strong>s(Linear4)<strong>+</strong>s(Linear5)<strong>+</strong>s(Linear6)<strong>+</strong>s(Nonlinear1) <strong>+</strong>s(Nonlinear2) <strong>+</strong>s(Nonlinear3),family <strong>=</strong> binomial)<br><br>banking.gam <strong>&lt;-</strong> gam(formula <strong>=</strong> y<strong>~</strong>s(age)<strong>+</strong>s(previous)<strong>+</strong>s(euribor3m)<strong>+</strong>s(cons_conf_idx)<strong>+</strong>s(cons_price_idx)<strong>+</strong>s(nr_employed)<strong>+</strong>s(emp_var_rate)<strong>+</strong>s(pdays)<strong>+</strong>`job_blue-collar`<strong>+</strong><br>                     job_management<strong>+</strong>`job_other 1`<strong>+</strong>`job_other 2`<strong>+</strong>job_services<strong>+</strong>job_technician<strong>+</strong>marital_married<strong>+</strong>marital_single<strong>+</strong> education_high.school<strong>+</strong>education_professional.course<strong>+</strong>education_university.degree<strong>+</strong>education_unknown<strong>+</strong>default_unknown<strong>+</strong>housing_unknown<strong>+</strong><br>                     housing_yes<strong>+</strong>loan_unknown<strong>+</strong>loan_yes<strong>+</strong>poutcome_nonexistent<strong>+</strong>poutcome_success,family <strong>=</strong> &quot;binomial&quot;,data <strong>=</strong> training.data<strong>$</strong>Banking)<br><br>adult.gam <strong>&lt;-</strong> gam(formula <strong>=</strong> Class<strong>~</strong>s(age)<strong>+</strong>(workclass)<strong>+</strong>s(fnlwgt)<strong>+</strong>(education)<strong>+</strong>s(education_num) <strong>+</strong>(marital_status)<strong>+</strong>(occupation)<strong>+</strong>(relationship)<strong>+</strong>(race)<strong>+</strong>(sex)<strong>+</strong>s(capital_gain) <strong>+</strong>  s(capital_loss) <strong>+</strong> s(hours_per_week)<strong>+</strong> (native_country),<br>                           family <strong>=</strong> &quot;binomial&quot;,<br>                           data <strong>=</strong> training.data<strong>$</strong>Adult)<br><em># Create GLM Models</em><br><br>synth.model.glm<strong>&lt;-</strong> glm(formula <strong>=</strong> Class<strong>~</strong>Linear1<strong>+</strong>Linear2<strong>+</strong>Linear3<strong>+</strong>Linear4<strong>+</strong>Linear5<strong>+</strong>Linear6<strong>+</strong>Nonlinear1 <strong>+</strong>Nonlinear2 <strong>+</strong>Nonlinear3 ,family <strong>=</strong> &quot;binomial&quot;,data <strong>=</strong> training.data<strong>$</strong>Synthetic)<br><br>banking.mode.glm <strong>&lt;-</strong> glm(formula <strong>=</strong> y<strong>~</strong>.<strong>-</strong>V1<strong>-</strong>four.nodes ,family <strong>=</strong> &quot;binomial&quot;,data <strong>=</strong> training.data<strong>$</strong>Banking)<br><br>adult.mode.glm <strong>&lt;-</strong> glm(formula <strong>=</strong> Class<strong>~</strong>age<strong>+</strong>workclass<strong>+</strong>fnlwgt<strong>+</strong>education<strong>+</strong>education_num <strong>+</strong>marital_status<strong>+</strong>occupation<strong>+</strong>relationship<strong>+</strong>race<strong>+</strong>sex<strong>+</strong>capital_gain <strong>+</strong>  capital_loss <strong>+</strong> hours_per_week<strong>+</strong> native_country ,<br>                           family <strong>=</strong> &quot;binomial&quot;,<br>                           data <strong>=</strong> training.data<strong>$</strong>Adult)</pre><h3>Model Testing</h3><p>Now that we’ve fit the models, we can compare their results. To do this, I will rely on ROC AUC and PR AUC. The code to evaluate the results is presented below:</p><pre>synth.test.preds &lt;- lapply(X = synth.models,FUN = function(x){  <br>  tree.to.feature(tree.model = x,dt = test.data$Synthetic)  }) %&gt;% data.frame<br>banking.test.preds &lt;-lapply(X = banking.models,FUN = function(x){ <br>  tree.to.feature(tree.model = x,dt = test.data$Banking)  }) %&gt;% data.frame<br>adult.test.preds &lt;- lapply(X = adult.models,FUN = function(x){ <br>  tree.to.feature(tree.model = x,dt = test.data$Adult)  }) %&gt;% data.frame<br>names(synth.test.preds) &lt;- c( &quot;three.nodes&quot;)<br>names(banking.test.preds) &lt;- c( &quot;four.nodes&quot;)<br>names(adult.test.preds) &lt;- c( &quot;four.nodes&quot;)<br>test.data$Synthetic &lt;- cbind( test.data$Synthetic,synth.test.preds )<br>test.data$Banking &lt;- cbind(test.data$Banking ,banking.test.preds )<br>test.data$Adult &lt;- cbind(test.data$Adult ,adult.test.preds )<br>training.for.mmdata &lt;- data.frame(predict(banking.mode.glm,newdata = training.data$Banking, type = &quot;response&quot; ),<br>                                  predict(banking.mode.four.deep, newdata = training.data$Banking,type = &quot;response&quot; ),<br>                                  predict(banking.gam,newdata = training.data$Banking, type = &quot;response&quot; )  ) <br>training.mdat &lt;- mmdata(scores = training.for.mmdata,labels = training.data$Banking$y,<br>                        modnames = c(&quot;Logistic Regression&quot;, &quot;Tree w/ GLM&quot;, &quot;GAM&quot;))<br>testing.for.mmdata &lt;- data.frame(predict(synth.model.glm,newdata = test.data$Synthetic, type = &quot;response&quot; ),<br>                                 predict(synth.model.three.deep, newdata = test.data$Synthetic,type = &quot;response&quot; ),<br>                                 predict(synth.gam,newdata = test.data$Synthetic, type = &quot;response&quot; )  ) <br>testing_mdat &lt;- mmdata(scores = testing.for.mmdata,labels = test.data$Synthetic$Class,<br>                       modnames = c(&quot;Logistic Regression&quot;, &quot;Tree w/ GLM&quot;, &quot;GAM&quot;))<br>training_ROC &lt;- autoplot(evalmod(training.mdat),curvetype = c(&quot;ROC&quot;))+theme(legend.position = &quot;bottom&quot;) +ggtitle(&quot;ROC Curve - Training Data&quot;)<br>training_PR &lt;- autoplot(evalmod(training.mdat),curvetype = c(&quot;PR&quot;))+theme(legend.position = &quot;bottom&quot;) +ggtitle(&quot;PR Curve - Training Data&quot;)<br>testing_ROC &lt;- autoplot(evalmod(testing_mdat),curvetype = c(&quot;ROC&quot;))+theme(legend.position = &quot;bottom&quot;) +ggtitle(&quot;ROC Curve - Testing Data&quot;)<br>testing_PR &lt;- autoplot(evalmod(testing_mdat),curvetype = c(&quot;PR&quot;))+theme(legend.position = &quot;bottom&quot;) +ggtitle(&quot;PR Curve - Testing Data&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Zud1mUf6rhlg1G89bmnXw.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vO25TP86HtmjhJ0f40g88A.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*6a9ZYPit__EbIEiJ7aRPIw.png" /><figcaption>Image by author</figcaption></figure><p>Each model performs well in both the training and testing data sets and there is little to no deterioration of model accuracy when switching to the out-of-sample data. In all cases, there is a small improvement of the performance metrics from using decision trees as a feature engineering step. We also see that in all cases the GAM model outperforms either model.</p><p>In several cases, the GAM outperforms the GLM+Tree model by a greater margin than the GLM+Tree outperforms the logistic regression model (see PR AUC for Banking and Synthetic and the Adult data-set). From this, we can see that including the structure of the nonlinear data in the model specification instead of transforming it allows for improved performance in all cases. This is significant given that using GAM also eliminates the investigation and comparison of potential decision trees for the model.</p><h3>Conclusion:</h3><p>I compared the results from two different approaches to coping with nonlinear data. The first was used a decision tree model as a feature engineering step to create a set of one-hot encoders which are then used in a logistic regression model. The second method was a generalized additive model which incorporates the potential nonlinearity of the data into the model specification. This eliminates the assumption of a linear relationship between the dependent and independent variables. These two methodologies were compared on three data sets to see if there was a consistent difference in their performances. The tests show that GAM outperformed the GLM+Tree models in every instead. Both models outperformed simple logistic regression models.</p><p>The fact that GAMs outperformed the Tree+GLM models is significant since it also eliminates the feature engineering step and the ambiguity about the depth of the tree being used to generate the processed variables.</p><h3>References:</h3><p>[1] T. Hastie, R. Tibshirani, J. Friedman, <a href="https://web.stanford.edu/~hastie/ElemStatLearn/">Elements of Statistical Learning</a> (2009)</p><p>[2] T. Hastie, R. Tibshirani, <a href="https://web.stanford.edu/~hastie/Papers/gam.pdf">Generalized Additive Models</a>, Statistical Science (1986)</p><p>[3]Syzmanski, Andrzej, <a href="https://towardsdatascience.com/combining-logistic-regression-and-decision-tree-1adec36a4b3f">Combining logistic regression and decision tree</a>, Towards Data Science (2020)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1840cac91cb1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/boosting-performance-by-combining-trees-with-glm-a-benchmarking-analysis-1840cac91cb1">Boosting performance by combining trees with GLM: A benchmarking analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Of Data and Dairy Farmers]]></title>
            <link>https://medium.com/pete-caya/of-data-and-dairy-farmers-d60829da0b73?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/d60829da0b73</guid>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[agriculture]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sat, 07 Apr 2018 12:53:00 GMT</pubDate>
            <atom:updated>2018-04-07T12:59:49.170Z</atom:updated>
            <content:encoded><![CDATA[<p>I don’t have a stake in agriculture past my consumption of the products, but I’ve heard rumblings about the state of the dairy industry in the US. What I’ve attempted to do here is come to a conclusion about the cause of closures in US dairy farms using publicly available data to see if I could better understand the claim.</p><p>My conclusion based on the skimming I’ve done is that the cause of dairy farm closures relates to the increase efficiency for milk production and the drop in international demand for US dairy products. This first cause isn’t surprising given that the historical trend has been towards fewer farmers as farming practices become more efficient. The second cause is more interesting, and I think, explains the reason price drop better. However, a full discussion of international trade would get too involved so in this case, I’ve aggregated the analysis of others.</p><h3>What data do we have?</h3><p>The data I used from this article is largely drawn from the USDA which has data available back to 1975. I examined it using Excel and R. Other information came from FRED, the WSJ, and a paper from the USDA ERS. These items are cited at the bottom.</p><h3>Is the way people are consuming dairy changing?</h3><p>Ultimately, any goods produced by farmers have to be taken to market — with this is mind, it would be helpful to know what the actual consumption of dairy products in the US is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8zkt0LSU95ei64--kdcNYg.png" /></figure><p>Since 1975, the per capita consumption of dairy products has dropped 21%. While this per capita estimate is not a country-wide total consumption number (which would have grown given the increase of the US population over the same time), it does provide evidence that these products are not viewed as necessary as they used to be.</p><h3>What is happening to dairy production?</h3><p>The total number of dairy cows has actually dropped by 14% while the number of gallons of milks each produces has went up 91% relative to 1980. This implies that dairy farmers are able to produce “more with less”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tunn6lOaCA-czlBWim31A.png" /></figure><h3>How are prices controlled by the dairy industry?</h3><p>The trend as of 2016 has been to increase the dumping of milk in order to lower supply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*WL2zNGa18A8oLHTHNMf1ZA.png" /><figcaption>Chart taken from the WSJ</figcaption></figure><h3>How has international trade changed and how does it impact the US dairy industry?</h3><p>Only looking at domestic factors on milk productions would be short sighted. As discussed in this <a href="https://econlife.com/2017/05/milk-glut-2/">blog post</a>, a large impact was felt by the dairy industry because of the closing of exports to Russia, and the EU requiring fewer dairy products from the US. Interestingly, the biggest single factor impacting these dairy prices might right now might actually be the state of international trade policies!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNe3zR_nr8FAi7-2sXDq1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIm4jnf04TSqbs7II2JaIg.png" /></figure><blockquote>In 2015, U.S. dairy exports declined for several reasons:<br> • Global demand for dairy products was relatively weak, especially from China.<br> • Since August 2014, Russia has banned imports of most dairy products from the United States, the European Union (EU), Canada, Australia, and Norway. Although the United States is not a major supplier for Russia, the ban caused the EU to export to alternative markets in competition with the United States. Russian dairy imports declined due to the ban and recession in the country, contributing to lower global dairy prices (see box, “Russian Trade Ban”).<br>• The value of the U.S. dollar was strong relative to other currencies, causing U.S. exports to be less attractive.<br>• In April 2015, the EU discontinued its milk supply quotas. EU dairy farmers thereafter increased their level of milk production, boosting their exports and crowding out dairy product imports from the United States.</blockquote><blockquote>The 2015 decline in U.S. dairy exports contributed to a substantial fall in milk prices received by dairy farmers. The all-milk price fell from a record high of $24.00 per cwt in 2014 to $17.10 per cwt in 2015 (USDA NASS, 2016), a 29-percent fall — “Growth of US Dairy Exports,” USDA ERS</blockquote><h3>What can we say about the dairy industry based off the evidence we have?</h3><p>A major change has occurred in the way that the United State dairy industry operates: The person that ultimately buys the product has changed. Whereas the industry previously has been domestic in scope, falling per-capita consumption by people in the US and an increase in exports means the price a dairy farmer is paid may well be more affected by trade policy and subsidies than anything else.</p><p>As discussed in [3] and [4] the recent volatility in the prices can be explained by the rise in demand from increased exports, and then the sudden shutting off of that demand due to greater competition in the EU and Russia’s trade ban.</p><p>A final note on price controls for dairy products. While beginning to write this article I noted several people I knew had substituted milk for one of the alternatives (for instance, almond milk). Throughout history, dairy products have had no competition and were viewed as a staple. Recently, more people have ceased purchasing dairy, while we have seen the entry of a wide range of alternatives to dairy products. With greater competition in mind, it seems like strategies to artificially raise the price of dairy products may do more harm than good since a consumer can easily replace milk with a substitute.</p><h3>References:</h3><p>[1] <a href="https://www.ers.usda.gov/data-products">USDA ERS Dairy Product Data</a></p><p>[2] <a href="https://fred.stlouisfed.org/series/WPU01610102#0">FRED</a></p><p>[3] <a href="https://www.ers.usda.gov/webdocs/publications/81255/ldpm-270-01.pdf?v=42703">Growth of U.S. Dairy Exports</a></p><p>[4] <a href="https://econlife.com/2017/05/milk-glut-2/">Why Happy Cows Make Farmers Sad</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d60829da0b73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pete-caya/of-data-and-dairy-farmers-d60829da0b73">Of Data and Dairy Farmers</a> was originally published in <a href="https://medium.com/pete-caya">Pete Caya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Looking back on my first 50K]]></title>
            <link>https://peterolivercaya.medium.com/looking-back-on-my-first-50k-50f053ecc582?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/50f053ecc582</guid>
            <category><![CDATA[physical-fitness]]></category>
            <category><![CDATA[stoicism]]></category>
            <category><![CDATA[bucketlist]]></category>
            <category><![CDATA[ultramarathon]]></category>
            <category><![CDATA[running]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sun, 25 Mar 2018 12:55:38 GMT</pubDate>
            <atom:updated>2018-03-25T12:55:38.969Z</atom:updated>
            <content:encoded><![CDATA[<p>Last Saturday I knocked another item off my bucket list: Running a 50K!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oKBoCuEIl-40wt-F-6AJsw.jpeg" /></figure><p>I decided to do the <a href="http://www.jcstone50k.com/">JC Stone 50K Road Race </a>back in October after I did the half-marathon trail-run. Doing an ultra had been on my bucket list for a while, and after managing to do marathon distances a few times, I thought I would give it a go:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*0btluHMnoIIUU7OjNtR6Zw.png" /><figcaption>The race consisted of a five mile loop around this lake in North Park near Pittsburgh.</figcaption></figure><p>Training in the winter and coordinating my career with the amount of mileage necessary for this race was a challenge. While things were slow going in the beginning, I found the best remedy for doubts about something like this was to just start doing it and assume that things would get easier as I went along. As it turns out, they did! I adapted to running in 10 degree weather, dealing with ice, and sometimes doing two training runs on the same day. At the same time, Pittsburgh offered me some great views while I was preparing!</p><p>After my fourth lap around the lake I did a quick inventory of how far I had gone: 21 miles with ten more to go. Right then the only thing I wanted to do was quit. Another lap more and I could feel an aching sensation in my hips which came and went for the rest of the race. If I stopped, my muscles would sometimes begin to cramp. I kept conscious of one thing though: Even if I was uncomfortable or in pain, nothing was forcing me quit.</p><p>Looking back now, I believe even more that the exhaustion is almost completely a mental product. It isn’t the result of your body or will being used up, it’s an accumulation of inconveniences and excuses. If one doesn’t recognize them as they come, and take an accounting of them, it’s possible to start taking them seriously and quit.</p><p>While there is a physical barrier, one generally doesn’t encounter it, and when one does, it’s almost always possible to find an alternate path around it. Not everything is possible through discipline alone, but resolution and time can bring most things into reach.</p><p>In a lot of ways, as a Stoic, this should have been self-evident but often reading something and practicing it is very different.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50f053ecc582" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Winter Running]]></title>
            <link>https://peterolivercaya.medium.com/winter-running-6a7d234b2e0c?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a7d234b2e0c</guid>
            <category><![CDATA[running]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sat, 06 Jan 2018 22:14:23 GMT</pubDate>
            <atom:updated>2018-01-06T22:14:23.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rZOT9AJnhJoiUrlT93Z0Q.jpeg" /><figcaption>Getting into winter running has been a real challenge for me the past two weeks! Never one to get discouraged, here are some thoughts on it.</figcaption></figure><p>The holidays are finally over and the last bit of good weather left with 2017. The result: Getting out and training for my 50K has become that much harder. I haven’t had a good distance run in about three weeks due to cold and travel.</p><p>I try to view the difficult that the cold brings as an advantage though. Running, in a lot of ways, is an activity based off enduring and doing something in spite of it being uncomfortable. That’s the way I’ve tried to frame it, at least.</p><h3>Coping with the Cold</h3><p>The main strategy I’ve used has been to break up longer runs into two pieces. This has allowed me to keep my mileage up, but I have some concerns about my performance at really long distances. The trouble is that much of training has to do with how you feel <em>after</em> mile twenty when your body feels spent.</p><p>One idea ( which I haven’t tried) was inspired by <a href="https://runnersconnect.net/cience-of-bonking-and-glycogen-depletion/">this </a>post which describes people running in a fasted state. One of the basics of endurance running is how to train yourself to keep going once you’ve depleted your glycogen reserves, and it seems like a period of fasting could be a way to do this. I could run ten miles one night, and fast until twelve miles is run the next day. While this might not be perfect, it strikes me as a way to get by and I may do it.</p><h3>Weightlifting on the Side</h3><p>I’ve also been compensating for the decreased time running by spending more time on weightlifting. I’m not really a specialist in terms of fitness, so weightlifting has always been a big part of my training. Being stuck indoors has made me spend more time working on my deadlift, squats, and other strength-building movements that complement distance running.</p><h3>Just Going the Distance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5cvY9NlD7ekZH0gXNxvECg.jpeg" /></figure><p>Coping with the cold, and maybe finding some ways to cross-train are all well and good, but it remains necessary embrace the fact that it’s uncomfortable and do it anyway. With that in mind, I turned today’s run into a Stoic exercise:</p><h4>Stoic Thoughts</h4><ol><li>My scarf makes it harder to breath — At least it won’t hurt me in the long run, and it will probably improve my cardio.</li><li>I’m miserable with how cold it is — Two months from now it will be warm again and running in the cold will toughen me up while other people are cozy indoors.</li><li>These clothes are so stiff and heavy — When I am back to shorts and a t-shirt everything is going to feel so light and flexible!</li><li>Nobody else is out. Why should I be? — I am getting to see the city in a way that very few people get to this time of the year.</li></ol><p>Anyway, I’ll leave off here. I’m still on track to run my first 50K on March 17, and I’m really looking forward to it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a7d234b2e0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Looking Back on Five Weeks of Training]]></title>
            <link>https://medium.com/pete-caya/looking-back-on-five-weeks-of-training-96e3fcd1b1ec?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/96e3fcd1b1ec</guid>
            <category><![CDATA[inner-strength]]></category>
            <category><![CDATA[marathon]]></category>
            <category><![CDATA[running]]></category>
            <category><![CDATA[ultramarathon]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sat, 09 Dec 2017 02:47:36 GMT</pubDate>
            <atom:updated>2017-12-09T02:47:36.766Z</atom:updated>
            <content:encoded><![CDATA[<p>Right now, I’m five weeks into training for the <a href="http://www.jcstone50k.com/">JC Stone 50K </a>on March 17, 2017. I’m still in the initial ramp-up phase before a weekly distance run become something too routine to be novel. Pittsburgh’s terrain is more challenging than Rochester’s, but I’ve finally found my way from the little burgh of Dormont into the city:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*whFLih_JeodvO3CefuESwg.png" /></figure><p>As an update, I thought I’d write about the ultra-marathon training plan that more or less describes what I’ll be doing during the next thirteen weeks. I don’t plan on holding to this strictly but it is something to shoot for:</p><p>Something I noticed when putting together this training plan based on what other people have posted is that training for distances greater than a marathon does not entail a massive jump in the number of miles run in one specific day. Instead, longer runs are “stacked” closer together.</p><p>This goes back to the heuristic that the recovery time required from a run increases super-linearly with the number of miles run. Running a marathon, for example, takes more than twice the time to recover from than running a half-marathon. Stacking longer runs allows you to build up the endurance to do a longer distance without need long periods of time to recover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*puP5P7LUSymD8R37CAxVtw.png" /></figure><p>Part of the training I have been doing involves weight training several times a week. I like running, but I am willing to sacrifice a modest amount of speed or endurance in order to keep some extra muscle tone. My favorites are dead-lifts, squats, and functional fitness exercises in order to improve body composition and to give me a break from just running all the time.</p><p>Diet has also been a major concern for this. When I was training for my first marathon I found the difficult part to be recovering and feeling well around the distance running days, not the distance running itself.</p><p>In the past month I’ve let my diet become less strict and though this hasn’t caused any sorts of health issue or weight gain like you would expect, it is a weakness that can be quickly fixed. I’m going to brush the dust off my diet spreadsheet and use that to plan my meals for each day.</p><p>Tomorrow I’ll be doing by 20 miler, which will be the longest distance I’ve done since August, but I’m looking forward to a brisk (though not quick) venture on Pittsburgh’s north trail.</p><p>À la prochaine!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96e3fcd1b1ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pete-caya/looking-back-on-five-weeks-of-training-96e3fcd1b1ec">Looking Back on Five Weeks of Training</a> was originally published in <a href="https://medium.com/pete-caya">Pete Caya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Quick Explanation of Single Firm Event Studies]]></title>
            <link>https://medium.com/pete-caya/a-quick-explanation-of-single-firm-event-studies-d4f7a9f47ea7?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4f7a9f47ea7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sat, 02 Dec 2017 21:33:28 GMT</pubDate>
            <atom:updated>2017-12-02T21:33:28.225Z</atom:updated>
            <content:encoded><![CDATA[<p>Economists attempting to discern the way that some events impact a the value of a company’s stocks or bonds must find a way to measure how abnormal a change in the security’s value is. Often, these events can be something extraordinary (instances of fraud for example) but these impacts may involve something much more mundane such as the effect of stock splits on a company’s value. From here, cue a discussion on financial news networks like CNBC about the way that the news impacts the value of the company and the way that the company behaves as a result of the disclosure.</p><p>One way to determine the degree to which returns for a security are abnormal is to construct a model of the way that security typically behaves and then compare those predicted results to what actually happened. This methodology is referred to as an event study and has been used for decades in academia and industry. For example, the realm of legal cases for securities fraud deals with whether real harm was caused and what damages are. Academics have also used event study on multiple firms to understand how one specific type of event (like stock splits) impacts a collection of companies. My goal in this little article is to describe the underpinnings of event studies, and to show the basic steps in starting a single firm event study with a real example.</p><p>Before we try to determine whether something unusual has happened with stocks or bonds though, let’s try to use some basic principles to see if we can spot something unusual in the more simple context of coin-flipping.</p><h3>Which Coins are Fair?</h3><p>To determine what unusual impact an event has, we first have to come to some kind of understanding of what is normal. Let’s say that we have 1000 coins and that we want to determine which of the coins is fair and which are unfair. One possible way to accomplish our objective would be to flip each coin 100 times to see if the number of heads and tails falls within an acceptable distance from our expectation (50–50).</p><p>To determine what is acceptable, we have to pose a probability distribution function which describes the number of heads and tails. This simple model can be described using the binomial probability distribution function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*GlYfAsNc46l1HNArZrxiXw.png" /></figure><pre>set.seed(13)<br>sample_size &lt;- 1000<br>trials &lt;- rbinom(1000,size = 100,prob = .5)<br>trials[c(22,516,843)] &lt;- c(9,88,31)<br>trial_probs &lt;- pbinom(trials,size = 100,p=.5)<br><br>stdev &lt;- sqrt(100*.5*(1-.5))<br><br>plot_data &lt;- data.frame(1:1000, trials)<br><br>names(plot_data ) &lt;- c(&quot;Ind&quot;,&quot;value&quot;)<br>freaks &lt;- subset(x = plot_data,subset = plot_data$Ind %in% c(22,516,843))<br><br>ggplot(plot_data) + geom_point(aes(x = Ind,y= value))+<br>   geom_point(data = freaks,aes(x = Ind,y= value),colour = &quot;red&quot;)+<br>   geom_hline(yintercept = qbinom(p = .99,size = 100,prob = .5,lower.tail = FALSE))+<br>   geom_hline(yintercept = qbinom(p = .99,size = 100,prob = .5,lower.tail = TRUE))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BwQnZ3MOu2cbuoqO0-M_VQ.png" /></figure><p>Most of the data falls nicely within the top and bottom most lines of the graph which mark where 99% of our outcomes should fall. I also added a few completely unfair results to the data that was plotted — we can recognize these by finding the likelihood that we would obtain each of these observations from a binomial probability distribution:</p><p>Additionally, we do see that we have 19 observations which lie outside of the 2 standard deviations, so we may want a stronger measure depending on what our application is.</p><h3>Anomaly Detection with a Market Model</h3><p>The previous example is simple: We use the mean and some simple properties of the probability distribution function to identify cheating in our 1000 trials. As it turns out, we can identify suspicious games by simply looking for the values which lie far enough from the mean as to be extremely unlikely. In the case above, our artificial “cheater” experiments have the probabilities of 1.66^-18}, 1.27^-16, and 9.15^-5.</p><p>Let’s add one other detail to make things slightly more complicated: Let’s say that instead of doing simple binomial trials like above, we perform a set of trials where the mean of the distribution changes over time. If we do this, the strategy above won’t work as well since the central measure that we are using to determine outliers is changing. In this case, we need to instead define some expected value for the company’s returns using a market model. We will use a OLS regression model to calculate the returns in order to be consistent with finance literature regarding the distribution of security returns.</p><h3>Abnormal Returns/Residuals</h3><p>As it turns out, employing a regression model employs similar reasoning as this second numerical example above. In the same way, we attempt to identify what the center of the distribution <em>should</em> be and the size of the standard deviation for the distribution. In this case, the statistical model we use to describe the returns tends to be ordinary least squares:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/91/1*Q2Wu7pZ_tS9D_i3xYMrKTA.png" /></figure><p>We find the difference between the actual returns versus the model returns in the same manner as example 2, first by calculating the abnormal returns (residuals):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/101/1*VTcBl1IcsHO9piVQ8obg_w.png" /></figure><h3>Calcuating the T-stats</h3><p>In order to create a meaningful measure of how far our residual lies from the expectation defined by our regression model. This turns out to be a fairly simple procedure: It turns out that we can identify meaninful deviations from our model using the fact that we assume that the errors are assumed to be distributed normally. If we do this, we can apply a simple t-test to the residuals using the standard error the regression model, <em>σ</em>σ, as the denominator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*9IGvdvxvg33xOmVCAncg2A.png" /></figure><h3>Cumulative Access Returns</h3><pre>symb_eval &lt;- <strong>function</strong>(x){as.data.table(eval(parse(text = x)))}<br><br>pct &lt;- <strong>function</strong>(x){<br>  inds &lt;- (is.na(x))<br>  returns &lt;-  x[!inds]/Lag(x[!inds])-1<br>  ret_x &lt;- x<br>  ret_x[!inds] &lt;- returns<br>  <strong>return</strong>(ret_x)<br>}<br><br>tickers &lt;- c(&quot;CHK&quot;,&quot;SPY&quot;,&quot;SLB&quot;)<br>getSymbols(tickers,src = &quot;google&quot;)</pre><pre>Company &lt;- symb_eval(tickers[1])<br>Company[,`Company Returns` := lapply(X = .SD,FUN = pct),.SDcols = grep(x = names(Company),pattern = &quot;Close&quot;)]<br><br>MKT &lt;- symb_eval(tickers[2])<br>MKT[,`Market Returns` := lapply(X = .SD,FUN = pct),.SDcols = grep(x = names(MKT),pattern = &quot;Close&quot;)]<br><br>IND &lt;- symb_eval(tickers[3])<br>IND[,`Index Returns` := lapply(X = .SD,FUN = pct),.SDcols = grep(x = names(IND),pattern = &quot;Close&quot;)]<br><br>Model_Data &lt;- Reduce(f = <strong>function</strong>(x,y){ base::merge(x,y,by=&quot;index&quot;)}, x = list(Company,MKT,IND))<br><br><em># It looks like we are missing a few days.  There isn&#39;t much to be done about this aside from being aware of it:</em><br>Company$index[!Company$index %in% Model_Data$index]</pre><pre><em># Train the regression model from the period 1/1/2013 to 12/31/2014:</em><br><br>mkt_model &lt;- lm(data =Model_Data,formula = `Company Returns`~`Market Returns`+`Index Returns`,<br>                subset = (Model_Data$index &gt;= as.Date(&quot;2015-06-01&quot;) &amp;  Model_Data$index &lt;= as.Date(&quot;2015-12-31&quot;)))<br><br>evt_study &lt;- data.frame(Model_Data[,grep(x = names(Model_Data),pattern = &quot;Return|index&quot;),with=FALSE],<br>                predict(object = mkt_model,newdata = Model_Data))<br><br>names(evt_study)[c(1,5)] &lt;- c(&quot;Date&quot;,&quot;Predicted Returns&quot;)<br>SE &lt;- summary(mkt_model)$sigma<br><br>evt_study$Residuals &lt;- evt_study$Company.Returns-evt_study$`Predicted Returns`<br>evt_study$`T-Stat` &lt;- evt_study$Residuals/SE<br><br>evt_study &lt;- as.data.table(evt_study)<br><br>test_begin &lt;- as.Date(&quot;2016-01-01&quot;)<br>test_end &lt;- as.Date(&quot;2016-06-01&quot;)<br><br>evt_subset &lt;- evt_study[Date &gt;= test_begin &amp; Date &lt;= test_end]<br>g1 &lt;- subset(evt_subset,abs(`T-Stat`) &gt;6)<br><br>ggplot(data = evt_subset) + geom_point(aes(x = Date,y = `T-Stat`))+<br>  geom_point(data=g1,aes(x = Date,y = `T-Stat`), colour=&quot;red&quot;)+<br>  geom_hline(yintercept = 1.96)+geom_hline(yintercept = -1.96)+<br>  geom_hline(yintercept = 2.58)+geom_hline(yintercept = -2.58)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZOSaiqU89JFiUBI1EkxKg.png" /></figure><pre>feb_stat &lt;-evt_study[Date==as.Date(&quot;2016-02-08&quot;),`T-Stat`]<br>apr_stat &lt;-evt_study[Date==as.Date(&quot;2016-04-12&quot;),`T-Stat`]</pre><p>I picked Chesapeake because it’s a company that has had a lot of interesting news in the past year. I we look at the plot I posted above we see that 2/8/2016 has a t-stat of -7.42 . This makes sense given that on that day there were <a href="http://www.cnbc.com/2016/02/08/energy-shares-plunge-in-early-rout-chesapeake-williams-both-off.html">rumours that the company was hiring restructuring attorneys</a>. 4/12/2016 has a positive return which is almost as significant (6.97) after it was revealed that the banks financing the company <a href="http://money.cnn.com/2016/04/12/investing/chesapeake-energy-cash-crunch-stock-surge/index.html">would not take away their line of credit</a>. Obviously Chesapeake Energy was having a bump ride at the time (hence the other significant days) but since this is a post on performing the calculations for an event study, I’ll leave off on an in-depth analysis of the company.</p><h3>Issues Involved with Event Study Analysis</h3><p>This was a very brief and mostly numerical treatment of event studies. While this example concerns only one company, in academia an event study is often performed on a large group of companies. Another example could be factors in the business environment which contribute to abnormal returns, but are not related to the events we are trying to study (these are called confounding events). This departs from the statistical properties of the model and concerns the researcher’s knowledge of the circumstances of the company they are investigating however.</p><h3>References:</h3><p>[1] The Econometrics of Financial Markets: Campbell, Lo, MacKinlay at <a href="https://pdfs.semanticscholar.org/aac6/83a678a12a3dcd73389aac7289868847ea73.pdf">https://pdfs.semanticscholar.org/aac6/83a678a12a3dcd73389aac7289868847ea73.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4f7a9f47ea7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pete-caya/a-quick-explanation-of-single-firm-event-studies-d4f7a9f47ea7">A Quick Explanation of Single Firm Event Studies</a> was originally published in <a href="https://medium.com/pete-caya">Pete Caya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Analysis for Ultramarathon Training]]></title>
            <link>https://medium.com/pete-caya/data-analysis-for-ultramarathon-training-a0f8cd841aa8?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0f8cd841aa8</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[running]]></category>
            <category><![CDATA[marathon]]></category>
            <category><![CDATA[ultramarathon]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Mon, 06 Nov 2017 11:45:08 GMT</pubDate>
            <atom:updated>2017-11-06T11:45:08.135Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*e4yDHt4N7KpkFAg-U8Fl-g.jpeg" /><figcaption>Dehydrated and tired. Not my best picture, but I’m looking forward to doing the 50K version of this race in March!</figcaption></figure><p>Ultramarathon training is an area where I was able to find a limited amount of actual information on what works, and what doesn’t work.</p><p>About two weeks ago I decided that in March I would be running my first ultramarathon. This was motivated by a few things. First, I’ve been doing some light marathon running for about two years now and this seems like a nice step up to 50K (31.2 miles). In part, it’s an act of contrition — I missed running in the Rochester marathon this year because of a move, and wanted to pick up something to make up for it.</p><p>About the same time that it occurred to me to run this 50K, it also occurred to me to do a bit extra: Curate and analyze a data-set concerning my training.</p><h4>Why Bother?</h4><p>The number one heuristic for amateur runners is to just run more miles. However, it’s easy to go out of control with this simple strategy. I’ve fallen into the habit of over-training in the past which has caused the normal collection of symptoms like sleep disruptions, overuse injuries, and fatigue. Have a better track of my methods and progress will also mean that I can also have a better idea of what has a positive impact on my performance versus what just sounds like it would be effective which often seems to be the strategy I see people use.</p><h4>What Am I Doing Differently?</h4><p>Keeping track of training is nothing — runners are always told to keep running journals, and we were always told how important it was to keep track of our progress when I was into lifting. I plan on doing a few things differently.</p><p>First, I’m going to attempt to keep more expansive and accurate records for what I am doing. This includes the normal information on training frequency, speed, and distance, as well as metrics related to body composition, heart rate during runs, and resting heart rate.</p><p>Second, I plan to conduct statistical analysis of my training in order to come to a better conclusion on what does an does not work. In the near term, this will be nothing more than just inferential and summary statistics. In the longer term, I would like to try to use more sophisticated methods like regression analysis and PCA to discern relationships between things like average speed on a distance run and caloric intake.</p><h4>Data Collection</h4><p>I have a few different tools to help me work on my goals:</p><p><strong>Garmin ForeRunner</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*80goIBcAu-qbGm0-lhVSrQ.jpeg" /></figure><p>This has actually been a tool I’ve used for a few years now. Before that, I just plotted my route on Google Maps, but this gadget made my life so much easier.</p><p><strong>Heart-Rate Monitor</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*p_u2fu3wQd0gAkqmtqOVsQ.jpeg" /></figure><p>This is actually still a work in progress. I’ve been shopping for devices which will be able to provide an idea of my heart rate during runs, and will give me direct access to the data so that I can save it as a CSV file. Unfortunately, some of the choices are around $300, so depending on how things work out, I may end up making this a <a href="https://tutorials-raspberrypi.com/raspberry-pi-heartbeat-pulse-measuring/">DIY project</a>.</p><p><strong>Software Accoutrements:</strong></p><p>Data collection and basic summary statistics will be in Excel. Any heavy duty analysis will be performed in R.</p><h3>Long-Term Goals:</h3><p>The <a href="http://www.jcstone50k.com/">Lt. JC Stone 50K Road Race</a> is in about sixteen weeks so I am following this <a href="http://running.competitor.com/files/2012/11/46_nat_r1.pdf">50K training schedule</a>. I’m actually starting around the third week in the training plan, but I plan on drawing it out a little bit just to add in some extra recovery time into the mix. After this race, I plan on running in the <a href="http://www.pittsburghmarathon.com/">Pittsburgh Marathon</a> in May, and then to pick out a trail run in June.</p><p>Keeping records on my training during the next 7–8 months will undoubtedly yield some very useful information for the future and I’m looking forward to this new challenge!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0f8cd841aa8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pete-caya/data-analysis-for-ultramarathon-training-a0f8cd841aa8">Data Analysis for Ultramarathon Training</a> was originally published in <a href="https://medium.com/pete-caya">Pete Caya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Après Trois Mois J’ecrire un Nouveau Poste Français]]></title>
            <link>https://medium.com/pete-caya/apr%C3%A8s-trois-mois-jecrire-un-nouveau-poste-fran%C3%A7ais-bf077c01ce9a?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf077c01ce9a</guid>
            <category><![CDATA[language-learning]]></category>
            <category><![CDATA[french]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sun, 04 Jun 2017 01:08:05 GMT</pubDate>
            <atom:updated>2017-06-04T02:16:14.623Z</atom:updated>
            <content:encoded><![CDATA[<p>Salut! Il y a trois mois depuis mon dernier poste en Francais donc j’ai décidé qu’il est temps à écrire un nouveau article pour mon blog.</p><h3>Qu’est-ce Que J’etudiais Depuis Mars?</h3><p>Mon premier objectif a été étudier la conjugaison du verbs en Francais — notamment les différences entre passé composé et imparfait qui est très intéressante car il est plus précis que l’anglais. En plus, j’apprenais les conjugaisons pour présent et passé conditionnel qui ont plus simple que imparfait et passe compose. Enfin, je commencais etudier la conjugaison pour futur simple.</p><p>Quand j’ai commencé étudier Francais j’ai été un autodidact. J’utilisais seulement Duolingo et quelque autres sites et livres. Pour la commencement il est un bon strategie car tu n’as pas besoin un direction pour tes études: juste trouves un topic et étudier ca! Après quelques semaines j’ai voulu orientation en mon Francais donc j’ai engagé un enseignant — il est une chose a juste lire un livre mais si tu vois tu améliores ton nouveau langue plus vite un enseignant est très utile.</p><p>Maintenant, je recountres avec ma enseignante les mardis, et les autres jours j’utilise quelques livres: “Practice Makes Perfect French”. La livre “French Conversation” est tres bien pour apprendre les phrases comme “je pige” ou “pas de quoi” que n’est pas en Duolingo ou autres sites.</p><h3>Quoi D’autre?</h3><p>J’aime l’exercices dans mes livres et ma “classe” mais une grande partie de apprendre un nouveau langue est l’utilise. En la futur je voudrais écrire en cette blog plus souvent. En plus, je dois lire quelque simple livres comme la classique “Le Petite Prince”. Aussi, j’ai une copie de “Le Pere Goriot” de Honore Balzac mais cette livre est plus difficil, je pense!</p><h3>Back in Anglais</h3><p>I think I’ve exhausted my French and a reader will see my use of imparfait is, well, imperfect. Still, I though this was a fun little exercise for the evening. I’m looking forward to continuing my studies in the future and maybe exercising this a little bit in another trip to Quebec.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf077c01ce9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pete-caya/apr%C3%A8s-trois-mois-jecrire-un-nouveau-poste-fran%C3%A7ais-bf077c01ce9a">Après Trois Mois J’ecrire un Nouveau Poste Français</a> was originally published in <a href="https://medium.com/pete-caya">Pete Caya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Class Logistic Regression]]></title>
            <link>https://medium.com/pete-caya/multi-class-logistic-regression-75e04bea31dc?source=rss-3953329470ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/75e04bea31dc</guid>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Peter Oliver Caya]]></dc:creator>
            <pubDate>Sat, 22 Apr 2017 20:01:36 GMT</pubDate>
            <atom:updated>2017-04-22T20:01:36.856Z</atom:updated>
            <content:encoded><![CDATA[<p>A few weeks ago I wrote <a href="https://medium.com/pete-caya/implementing-binary-logistic-regression-in-r-e3a6f59ae294">this</a> blog post where I tasked myself with implementing two-class logistic regression from scratch. Since I couldn’t find any guides for implementing multi-class logistic regression online, I decided I would implement the multi-class version as well and write about it. As I finished though it was clear that the post was going to be quite long so I’ve decided to break it into two parts.</p><p>A lot of this material was learned and implemented using <a href="http://infolab.stanford.edu/~jiali/">Jia Li’s logistic regression presentation </a>in addition to ESL. These two sources really provided a well-rounded discussion of what logistic is and how to implement it.</p><h4>What Changes When Using &gt;2 Classes?</h4><p>The principle underlying logistic-regression doesn’t change but increasing the classes means that we must calculate odds ratios for each of the K classes. Consider the odds-ratio for the binary case: We take the ratio of the probability of class <strong>A</strong> to the probability of the Kth class which would be the second class (B).</p><p>K classes means we must now consider the calculated probability, p, of class i. Each class has its own specific vector of coefficients (represented as a vector of coefficients β with a subscript signifying its class):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*Vj_BU16vzOh-7T8DOXDnXg.png" /></figure><p>Note that instead of just trying to fit one set of β parameters, we now have (K-1) sets of variables which we are trying to fit!</p><p>Going through the requisite algebra to solve for the probability values yields the equations shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*_S2VDmCoCqsnK6qvkpgLdQ.png" /></figure><p>I implemented the calculation of the class probabilities as its own separate function which I have copied below:</p><pre># I implemented the multi-class version of the probability function to produce a matrix of the class probabilities.  Let K be the number of classes.<br># Inputs:<br># [1]  X - A N*(K-1) by M design matrix.<br># [2]  beta - A N*(K-1) length design matrix.<br># [3]  classes - The number of classes we are predicting.</pre><pre># Outputs:<br># A K by N matrix containing the class probabilities calculated as:</pre><pre>find_pi_multi &lt;- function(X,beta,classes){<br>  vars &lt;- ncol(X)<br>  prob_mat &lt;- matrix(rep(0,nrow(X)*(classes-1)),nrow = nrow(X),ncol = (classes-1))<br>  for(k in 1:(classes-1)){<br>    rel_indices &lt;- 1:vars+(k-1)*vars<br>    prob_mat[,k] &lt;- exp(X%*%beta[rel_indices])<br>  }<br>  denom &lt;- apply(X = prob_mat,MARGIN = 1,FUN = sum)+1<br>  prob_mat2 &lt;- cbind(prob_mat,as.matrix(rep(1,nrow(X))))/denom<br>  return(prob_mat2)<br>}</pre><h4>The Maximum Likelihood Function</h4><p>Since we now are using more than two classes the log of the maximum likelihood function becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*vVN9QPiMISK8tYu7wZ2Q9w.png" /></figure><h4>The Gradient</h4><p>Just for convenience, I’m copying the derivation of the gradient of the maximum likelihood function below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*8EWXVZZmEusU8cuC7gLQsg.png" /></figure><p>Turning this into a matrix equation is more complicated than in the two-class example — we need to form a N(K −1)×(p +1)(K −1) block-diagonal matrix with copies of X in each diagonal block matrix. Let’s denote this block matrix as X-tilde.</p><h4><strong>Forming X-Tilde</strong></h4><pre># I&#39;m sure R has a better way to form a block matrix.  Pardon my MATLABese here.</pre><pre>X_tilde &lt;- function(X,classes){<br>  r &lt;- nrow(X)<br>  N &lt;- r*(classes -1)<br>  X_tilde &lt;- matrix(rep(0,N*(classes-1)*ncol(X)),nrow = N, ncol = (classes-1)*ncol(X) )<br>  m &lt;- 1<br>  n &lt;- 1<br>  <br>  for(i in 1:(classes-1)){<br>    # block_diags[[i]] &lt;- X<br>    X_tilde[n:(n+r-1),m:(m+(ncol(X)-1)  )] &lt;- X<br>    n &lt;- n + r<br>    m &lt;- m + (ncol(X)) <br>    n:(n+r-1)<br>    m:(m+(ncol(X)-1)  )<br>  }<br>  return(X_tilde)<br>}</pre><p>We also need to form a N(K-1) length vector in with entries 1 where class i is observed, and zero otherwise. The function to construct this vector is displayed below:</p><pre># Generate a N(K-1) length vector of indicator functions based on class.</pre><pre>form_y &lt;- function(classes,y, obs){<br>  y0 &lt;- as.matrix(rep(0,obs))<br>  <br>  for(i in 1:(length(classes)-1)){<br>    # print(i)<br>    if(i==1){yp &lt;- y0<br>    yp[which(y==classes[i])] &lt;- 1<br>    }else{y_temp &lt;- y0<br>    y_temp[which(y==classes[i])] &lt;- 1<br>    yp &lt;- rbind(yp,y_temp)<br>    }<br>  }<br>  return(yp)<br>}</pre><p>All this being completed, the gradient for the multi-class version of the maximum likelihood function becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*h81kzBqrhsjTFnqlR7ldDg.png" /></figure><h4>The Hessian</h4><p>The derivation of the Hessian matrix doesn’t change:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*ogxxatqtCVAcSBDXn41VKg.png" /></figure><p>Again, our multi-class implementation makes producing the Hessian more involved. We have to form another block matrix summarizing the class probabilities. This is better summarized in Jia Li’s presentation which you can find <a href="http://infolab.stanford.edu/~jiali/">here</a>, so I won’t go into in this blog post. Instead, here is my implementation in R:</p><p><strong>The W Matrix</strong></p><pre>form_W &lt;- function(p){<br>  # I&#39;m not too worried about speed with this seeing as I am just forming block matrices.<br>  # Form the block diagonals in the W matrix:<br>  N &lt;- nrow(p)*(ncol(p)  -1)<br>  r &lt;- nrow(p)<br>  W &lt;- matrix(rep(0,N^2),nrow = N, ncol = N )<br>  block_diags &lt;- list()<br>  upper_diag &lt;- list() <br>  m &lt;- 1<br>  n &lt;- 1<br>  for(i in 1:(ncol(p)-1)){<br>    block_diags[[i]] &lt;- diag((p[,i])*(1-p[,i]))<br>    W[n:(n+r-1),m:(m+r-1)] &lt;- block_diags[[i]]<br>    n &lt;- n +r<br>    m &lt;- m + r<br>  }<br>  <br>  # Form the off-diagonals:<br>  # W_ii = -p_k*p_m<br> <br>  k &lt;- 1<br>  m &lt;- 1<br>  n &lt;- 1<br>  <br>      if(ncol(p)&gt;2){<br>      for(i in 1:(ncol(p)-2)){ <br>        # print(W)<br>         # print(paste(&quot;i is &quot;,i))<br>         for(j in  c((i+1):(ncol(p)-1)))  {<br>           m &lt;- r*(j-1)+1<br>           upper_diag[[k]] &lt;- diag(-p[,i]*p[,j])<br>           W[(n:(n+r-1)),m:(m+r-1)] &lt;- diag(-p[,i]*p[,j])<br>           k &lt;- k+1<br>         }</pre><pre>}<br>    W &lt;- W + t(W)<br>    diag(W) &lt;- diag(W)/2<br>  }<br> <br> # The block matrices should be the same across the diagonal so I&#39;m just adding the transpose<br>  # to the original W and correcting the main diagonal.<br>  return(&quot;W&quot; = W) <br>}</pre><p>Thankfully, this will be the end of our use of block-matrices for this project. The matrix form of the Hessian for the maximum likelihood function is displayed below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*WtqitJVQHAxx-n9zS3iPxg.png" /></figure><p>The rest of my implementation of the multi-class version of the log-likelihood function is displayed below:</p><pre># Multi-class Regression -----------------------------------------------------<br># Implementation of multi-class logistic regression:<br># Inputs:<br># [1]  Y - a column vector of N elements containing the actual classification results.<br># [2]  X - a (p+1) by N design matrix.<br># [3]  beta - a p-length column vector of coefficients.</pre><pre># Outputs:<br># [1]  f - The log-likelihood function.<br># [2]  df - (p+1) length column vector representing the log-likelihood&#39;s gradient.<br># [3]  ddf - a (p+1) by p Hessian matrix for the log-likelihood function.</pre><pre>LL_Multi &lt;- function(Y,X,beta){<br>  class_types &lt;- unique(Y)[order(-unique(Y))]<br>  classes &lt;- length(class_types)<br>  class_probs &lt;- as.matrix(rep(0,nrow(X))) <br>  p &lt;- find_pi_multi(X,beta,classes)<br>  # Get the estimated probability of the observed class.<br>  p2 &lt;- as.matrix(unlist(as.data.frame(p[,1:(dim(p)[2]-1)])))<br>  y &lt;- form_y(classes = class_types,y = Y,obs = length(Y))<br>  <br>  for(k in 1:length(class_types)){<br>    class_probs[which(Y==class_types[k])] &lt;- p[which(Y==class_types[k]),k]<br>  }<br>  XT &lt;- X_tilde(X,ncol(p))<br>  ws &lt;- form_W(p = p)<br>  f &lt;- sum(class_probs)<br>  # Calculate the gradient:<br>  df &lt;- t(XT)%*%(y-p2)<br>  # Calculate the Hessian:<br>  ddf &lt;- -t(XT)%*%ws%*%XT<br>  return(list(&quot;f&quot; = f,&quot;df&quot; = df, &quot;ddf&quot; = ddf))<br>  # return(list(&quot;f&quot; = f,&quot;df&quot; = df))<br>}</pre><h3>Testing the Algorithm</h3><p>I first compared my implementation with the glm function to try to generate consistent results. The difference between my results and glm was ~1e-16 at most.</p><h4>Comparison to the Two-Class Implementation:</h4><pre>set.seed(13)<br>mydata &lt;- read.csv(&quot;<a href="http://www.ats.ucla.edu/stat/data/binary.csv">http://www.ats.ucla.edu/stat/data/binary.csv</a>&quot;)<br>admission &lt;- as.matrix(mydata$admit)<br>beta &lt;- as.matrix(rep(0,3))<br>vars &lt;- as.matrix(mydata[,2:dim(mydata)[2]])</pre><pre># My two-class algorithm versus the multiple class algorithm:<br>L_L(y = admission,X = vars,beta = beta)<br>LL_Multi(Y = admission,X = vars,beta = beta)</pre><pre># Using Newton&#39;s method:<br>fromGLM &lt;- glm(formula = admission~vars-1,family = binomial())<br>binom_algo &lt;- newton(function(input){L_L(y = admission,X = vars,beta = input)},initial_values = beta,tol = 1e-13)<br>multi_algo &lt;- newton(function(input){LL_Multi(Y = admission,X = vars,beta = input)},initial_values = beta,tol = 1e-13)</pre><pre>data.frame(fromGLM$coefficients,multi_algo, fromGLM$coefficients-multi_algo)</pre><h4>Get the Code</h4><p>My full code for implementing two-class and multiclass logistic regression can be found at my Github repository <a href="https://github.com/poc1673/Logistic_Regression">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75e04bea31dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pete-caya/multi-class-logistic-regression-75e04bea31dc">Multi-Class Logistic Regression</a> was originally published in <a href="https://medium.com/pete-caya">Pete Caya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>