<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manuel Hupperich on Medium]]></title>
        <description><![CDATA[Stories by Manuel Hupperich on Medium]]></description>
        <link>https://medium.com/@mahupperich?source=rss-31491333ba59------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EwmRkN0kbMO3prUx1Rvorg.jpeg</url>
            <title>Stories by Manuel Hupperich on Medium</title>
            <link>https://medium.com/@mahupperich?source=rss-31491333ba59------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:58:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mahupperich/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tackling a Kaggle Competition]]></title>
            <link>https://medium.com/@mahupperich/forest-cover-type-classification-5b7d3e945144?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b7d3e945144</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Fri, 01 Jul 2022 11:30:24 GMT</pubDate>
            <atom:updated>2022-07-01T11:32:44.870Z</atom:updated>
            <content:encoded><![CDATA[<p>The following blog post intends to guide you through the resolution of the Kaggle Forest Cover Type-Classification Competition.</p><p>Throughout the study, Exploratory Data Analysis, Feature Engineering, and Modelling will be performed on the data. However, in case you want to dive into the code of this problem feel free to look it up in my <a href="https://github.com/Hupperich-Manuel/Tree-Type-Prediction"><strong>repository</strong></a><strong>.</strong></p><p>Feel free to connect with me on <a href="https://www.linkedin.com/in/manuel-hupperich/">LinkedIn</a>, and have a look at my Data Engineering projects for full ML model deployment on websites or algorithmic trading with Spark on m,y <a href="https://github.com/Hupperich-Manuel">GitHub</a></p><p>ENJOY!!!</p><h3>Data</h3><p>The dataset was obtained from the Kaggle Website. The data is in raw format and contains numerical and dummy variables. The training set contains 54 features and the target Forest Cover type, it has 15,120 observations. The test set contains all the features except the target, in total it includes 565,892 observations.</p><p><strong>The seven forest cover types are:</strong></p><p>1. Spruce/Fir</p><p>2. Lodgepole Pine</p><p>3. Ponderosa Pine</p><p>4. Cottonwood/Willow</p><p>5. Aspen</p><p>6. Douglas-fir</p><p>7. Krummholz</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*bArNmxK6FtjA1D9F2T5sPg.png" /></figure><h3>Topographic Analysis</h3><p>Topographic and satellite maps of the zone representing the Roosevelt National Forest of northern Colorado have been looked at to better understand the problem.</p><p>After examining the map, preliminary findings were established, which help to deeply understand the features of the dataset:</p><p><strong>§</strong> The four wilderness zones are separated from each other in terms of location, mainly covering east to west zones.</p><p><strong>§</strong> Among the forest, there are several mountains covering from north to south, which could indicate that height could be an insightful feature to take into consideration for the classification of the forest cover types.</p><p><strong>§</strong> From a satellite perspective, the colours of the forest in each wilderness zone are similar, which preliminary means, there is no predominant cover type in a specific wilderness zone.</p><h3>Exploratory Data Analysis</h3><p>Firstly, EDA was performed to find patterns, relationships, or anomalies in the training dataset.</p><h4><strong>Numerical Features</strong></h4><p><strong>Distribution</strong></p><p>The numerical variables seem to not have a Gaussian behavior. The features that measure distance are positively skewed. Hillshade 9 am and Noon has a similar left-skewed distribution. None of the features follow a normal distribution, which is confirmed by the statistical Shapiro test for normality (see Appendix 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*Z22l8ci7iJx059aYbcJI7A.png" /></figure><p><strong>Relationship with the target</strong></p><p>The relationship of the numerical variables against the target variable offers insights into which features will be useful in predicting the Cover Type:</p><p>§ <strong>Cover Types 1 &amp; 2:</strong> are alike: they are amongst the taller trees and have resemblance in appearance. They have the smallest slope and are fairly close to hydrology but further from roadways.</p><p>§ <strong>Cover Types 3 &amp; 4:</strong> are the closest to human infrastructures such as roadways or fire points. They are also the shortest trees.</p><p>§ <strong>Cover Types 5 &amp; 6: </strong>have a range of different characteristics.</p><p>§ <strong>Cover Types 7:</strong> is the tallest tree and is the furthest away from both roadways and hydrology, indicating this could be a wilder type of tree.</p><p><strong>Elevation, horizontal distance to roadways, and to hydrology are the features with the highest variance </strong>and therefore better explain the Cover Types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*ZGIS5AWV6CDiLCGwf8kktw.png" /></figure><p><strong>Correlation Matrix</strong></p><p>Considering highly correlated features as those &gt; 0.6 the matrix (see Appendix 2) shows that the highly correlated ones are:</p><p><strong>§</strong> Vertical and horizontal distance to hydrology</p><p><strong>§</strong> Aspect and hillshade_3pm</p><p><strong>§</strong> Hillshade noon and hillshade 3 pm</p><p><strong>§</strong> Hillshade 3 pm and hillshade 9 am (negative correlation)</p><p><strong>§</strong> Hillshade noon and slope (negative correlation)</p><h4>Categorical Features</h4><p><strong>Train and Test Comparison</strong></p><p>The categorical features are wilderness area and soil type, each observation contains only one area and type. These have been reverse-encoded for easier exploration.</p><p><strong>§</strong> Wilderness Area 3 is the most common in the train set while in the test set the most common is Area 1. The train set contains few observations of area 2, however, as this is also the case in the test set it is not a concern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*q3Tg3TMym8JtHrT3cLA4Cg.png" /></figure><p>Soil types 7 and 15 do not appear in the train set but also barely appear in the test set. Overall, the most common Soils in the test set are also frequent in the train set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*XO7BvyZTnlv-iXli2eo7MQ.png" /></figure><p><strong>Relationship with the Target</strong></p><p>The graph below demonstrates that certain soil types appear more frequently in specific Cover Types. For instance, Soil type 10 could be a strong indicator of Cover type 6 or 3.</p><p>Moreover, Soil types close in number appear to be closely related in their characteristics too. For example, we can see that Soil Types under 20 appear more often in Cover types 3, 4, and 6 whereas Cover Types above 20 are more frequent in Cover Types 1 and 2.</p><p>Lastly, the Soil type also <strong>indicates that Cover Types 1 &amp; 2 and 3 &amp; 4 are alike.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*nd9cMZyIpALmA69mhgDBdQ.png" /></figure><p>The Wilderness Area category, on the other hand, is less characteristic per Cover Type.</p><p>The most distinguishing observation is that Wilderness Area 4 only appears in Cover Type 4 and therefore could be a strong indicator of this type only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*kk26DtIzmrnbWfcoApGKaw.png" /><figcaption>b</figcaption></figure><h4>Target Variable</h4><p>There is an equal number of all cover types in the training set.</p><h3><strong>Benchmark</strong></h3><p>In order to later evaluate the feature engineering, a benchmark was set using logistic regression as it is the simplest classification algorithm.</p><p>The benchmark was established by fitting a logistic regression to the initial dataset. <strong>The accuracy obtained was 47.9%.</strong></p><p>Since logistic regression is a linear-based algorithm and relies on the absence of multicollinearity, a high accuracy level <strong>was not</strong> expected.</p><h3>Feature Engineering</h3><p>In this section we will look at null values and outliers, create new features and decide which transformations to apply to the data.</p><h4>Null Values</h4><p>There are no null values in the training set therefore no further action was needed.</p><h4>Creation of New Features</h4><p>New features were created from the information in the initial dataset, in order to feed the model with more valuable information.</p><p>The initial dataset contained 40 different Soil Types that shared many characteristics in their Kaggle description. This allowed us to group them into new features. See Appendix 3 for the technical definitions of the new Soil features and Appendix 4 for the frequency count of the new categorical features created.</p><p>The features created were as follow:</p><ul><li><strong>Soil Family:</strong></li></ul><p>30 Soil types into 11 families, and there were 10 soil types that did not belong to any of the Soil Families.</p><ul><li><strong>Soil Complex</strong></li></ul><p>The soil types were categorized into 5 different complexes, including the rock complex and the aqueous complex.</p><ul><li><strong>Soil Stoneness</strong></li></ul><p>The soil types were assigned to 5 different stoneness categories, ranging from extremely stony to extremely bouldery.</p><ul><li><strong>Water Altitude</strong></li></ul><p>The water altitude feature was calculated by taking the mean of the vertical distance to hydrology and the elevation.</p><ul><li><strong>Hydrology</strong></li></ul><p>The diagonal distance to the water. Calculated using Pythagoras theorem to the horizontal and vertical distance.</p><ul><li><strong>Cumulative Sunlight</strong></li></ul><p>Adds the total surface illumination that was provided in the initial dataset with three features the hillshade at noon, 3 pm, and 9 pm.</p><ul><li><strong>Elevation Bin</strong></li></ul><p>Binned the values of the elevation feature into 4 ordinal categories with equal amounts of observations.</p><ul><li><strong>Mean Fire Roads</strong></li></ul><p>Mean distance to both roadways and fire points.</p><ul><li><strong>Polynomial Features</strong></li></ul><p>The square root of the values.</p><ul><li><strong>Probability of Cover Type per Soil Type</strong></li></ul><p>The Soil Types were target encoded: replaced with a probability of the target given the Soil Type.</p><p>The polynomial features were created, however, did not significantly increase accuracy and did reduce the explainability. The Probability of Cover Type per Soil Type also did not increase accuracy and notably increased the number of features to deal with. As such these were not included in the model.</p><h4>Outliers</h4><p>The outliers were calculated after the normalization in order to only capture the most extreme outliers, using the Inter Quartile Method. There were c. 2200 outliers present in the features that calculate the distance to hydrology, roadways, and fire points (See Appendix 5). The outliers could contain some valuable information, for example, it appears that some trees are especially isolated. Therefore, a <strong>new binary variable with the outliers column</strong> was created in the training and test set to capture this information.</p><h4>Transformations</h4><p>The numerical variables are <strong>scaled using a Robust Scaler</strong>, as this method is robust to outliers. The categorical variables will be re-converted into <strong>dummy variables using an OneHotEncoder</strong>, so that they can be fed to linear models.</p><p>Having scaled the data will allow performing some further transformations such as Principal Component Analysis (PCA). <strong>PCA</strong> is a technique to <strong>reduce the number of features, by getting rid of features that are already explained by other features.</strong></p><p>This will be particularly useful given a large number of features and as the correlation between the features has not been removed (See Appendix 6).</p><p>The abovementioned transformations are implemented in a Pipeline to evaluate the feature engineering benchmark.</p><h3>Feature Engineering Benchmark</h3><p>The logistic regression fitted to the transformed data yields an accuracy of 74%, meaning 74% of the predictions were correct. This is a 27% increase compared to the initial benchmark demonstrating that the feature engineering has been successful.</p><p>The regression provides the importance of each feature. The most determinant features were horizontal distance to roadways, followed by elevation and aspect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*6TTog9mPfzwPeDuQuirUIw.png" /></figure><h3>Machine Learning</h3><h4>Train Validation Split</h4><p>The training dataset was split into a train and a validation set, with a split strategy of 75% — 25%.</p><h4>Construct the Pipeline</h4><p>Two different preprocessors were created, one for the tree-based algorithms and the other for the linear as they rely on different assumptions.</p><p>The Tree-based preprocessor applies the following:</p><p>1. Categorical pipeline: OneHotEncoding to re-convert them into dummy variables.</p><p>The Linear and Distance preprocessor applies the following:</p><p><strong>§</strong> Categorical pipeline: OneHotEncoding</p><p><strong>§</strong> Numerical pipeline:</p><p><strong>§</strong> PowerTransformer to normalize the data</p><p><strong>§</strong> Robust Scaling as the data contains outliers</p><p><strong>§</strong> PCA to reduce complexity</p><h4>Algorithms</h4><p>To evaluate which algorithm has better predicting performance, six different algorithms were run. The following algorithms used were:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*Aeq3sc-2skYmFCRZI77sBg.png" /></figure><p>Each pre-processor is applied to the data with its relevant algorithm (tree or non-tree). The algorithms are run with different proposed hyperparameters. The Grid Search Cross-Validation function selects the best hyperparameters based on their accuracy whilst doing cross-validation.</p><p>The algorithm is then fit to the training set and predicts using the validation data.</p><p>The predictions are compared to the actual Cover Types in the validation data and based on the results several metrics are calculated. (See Appendix 7 for the metrics of all the algorithms).</p><p>The three most accurate algorithms on the validation set were:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*bCoMpcd86yBqcmrPCPiMwA.png" /></figure><p>Tree-based algorithms yield the best performance, with RandomForest having the highest accuracy, followed closely by XGBoost.</p><h4>Hyperparameters</h4><p>The selected algorithm for the final model was the Random Forest. When running the model on the training set accuracy of 100% was obtained indicating that the model is overfitted. Therefore, more hyperparameters were introduced to regularize the model. These included a minimum number of samples to split per node and a minimum number of observations at the end node (also called the leaf).</p><h3><strong>Model Evaluation</strong></h3><h4>Overall Evaluation</h4><p>The hyperparameters did not significantly change the accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*9bTUzYNFNeNp2Vo-GNBPCw.png" /></figure><p>On average, the model predicts <strong>accurately 86.5% of the time</strong>. Precision indicates that when predicting for a certain type, 86.2 % of the instances are correctly classified. Recall means that out of all the actual observations for each Cover Type, the algorithm will find 86.5% of them.</p><h4>Cover Type-Specific Evaluation</h4><p>The model is very accurate at predicting types 4, 5, and 7 and struggles to correctly predict types 1, 2, and 3.</p><p>The Cover Types it confuses the most are 1 &amp; 2. It also confuses Types 1 &amp; 7 and it incorrectly predicts many of Types 3 as 4. As previously mentioned in the exploratory data analysis, these types are the most similar in their characteristics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*eYeiqg-aaBFcXfz5miEInw.png" /></figure><p>The ROC curve plots the relationship between the True Positive Rate and the False Positive Rate for each type. All areas under the curve are &gt; 0.95 indicating that despite being more accurate for certain Cover types, the algorithm can maximize True Positives whilst keeping low False Positives for all Types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*3fj1SQGyYgGYnrl9HkFchw.png" /></figure><p>The precision-recall curve shows that Types 1 and 2 have a lower recall rate, meaning that the algorithm does not predict enough of these types but when it does it is mostly correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*ysgli2dwKvVJqu0yPHu4bA.png" /></figure><p>Overall, the predictions are very close to the actual types in the training set as shown by the real vs. the predicted density plots (See Appendix 8).</p><h3>Limitations</h3><p>The source of the data is the Kaggle website and as such the data could be outdated and of poor quality.</p><p>The model confuses cover types 1 and 2, and 3 and 4; therefore, caution needs to be kept with these predictions.</p><p>Tree-based algorithms tend to overfit therefore further cross-validation on the testing test is necessary before deploying into production.</p><h3>Recommendations</h3><p>1. Retrain the model periodically with recent data, given some of the characteristics in the trees could be altered for example due to climate change.</p><p>2. Gather more information to differentiate the types that the algorithm confuses.</p><p>3. Depending on the purpose, Cover Types 1 and 2 could be predicted as one type as they are very similar, as well as 3 and 4. This could increase the accuracy of the model and reduce the costs of the project.</p><p>4. Extend the scope of the area in the dataset, which allows to train, test, and predict cover types of other closer wilderness areas of forest such as Mount Zirkel or Savis Creek.</p><p>Incorporate other features, such as images and videos, that improve the model’s power prediction.</p><h3>Appendix</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*iQ-TGT7Ro7e4Gy6cSeWSKg.png" /><figcaption>Appendix 1: Shapiro Test for normality of initial Features</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*pYcopJUB8zMNwYvrbeEBoQ.png" /><figcaption>Appendix 2: Correlation Matrix</figcaption></figure><p><strong>Appendix 3: Technical definitions of new Soil features</strong></p><p>1. Soil Family: group of soils that share the same physical and chemical properties.</p><p>2. Soil Complex: consists of two or more recognized classifications that are so intermixed geographically that separating them is impossible.</p><p>Soil Stoniness: refers to how hard the surface is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*TXM4tNcu0-zFFCEiSgZZnQ.png" /><figcaption>Appendix 4: Categorical created features</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*m2nY_Dia-sEZPAWELdgpVw.png" /><figcaption>Appendix 5: Boxplot for Outliers on normalised data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*QA2yXeCi2wcqWKVC4XmOqQ.png" /><figcaption>Appendix 6: Correlation Matrix including created features</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*8v2-lsdsOMyJHSdpF_qAtA.png" /><figcaption>Appendix 7: Prediction metrics of the different algorithms</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*zmSOIZkYLcuuXM0YxLXHGQ.png" /><figcaption>Appendix 8: Density plots of real values vs. predictions per Cover Type</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b7d3e945144" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LSTM-XGBoost Hybrid Stock Forecasting]]></title>
            <link>https://medium.com/@mahupperich/lstm-xgboost-hybrid-stock-forecasting-2ff9e419c020?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ff9e419c020</guid>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Tue, 08 Feb 2022 13:12:08 GMT</pubDate>
            <atom:updated>2022-02-08T20:47:10.590Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*N3eCAgIBlPN_xN8Z" /></figure><p>Whom of you has not thought about being a step ahead of the stock market, using the information to gain accurate predictions for the next trading day??. Well, I am sorry to say that I am not going to provide the new cheat sheet of how to gain millions in the stock market through this strategy. Nonetheless, you might gain some ideas that get you on the right path in the usage of the newest technologies in time series data.</p><p>The core idea behind this work was to take my knowledge one step further. The hybrid combinations of Deep Learning models together with Decision trees or Linear Regressions are fascinating new ways to extract much more information from raw inputs. Therefore, I compiled the things learned throughout the past years related to coding, statistics, ML models, DL models, Business perspectives and squeezed those into an actual <em>deployable</em> model for real stock price predictions.</p><p>Note, consider this to be the final draft of what has been very intense research on every topic treated in this work. Since it would be boring to explain every drawback and handicap faced during this time, send me a message and we can have a nice conversation, sharing our experiences on ML or DL deployments.</p><h3>Abstract</h3><p>This work contains an overall analysis of the takeaways on applying a hybrid Long Short Term Memory Deep Learning Model together with an XGBoost Regressor to predict the price of the AAPL (Apple Inc.) stock for the next day (t+1). Throughout this work, some assumptions are made regarding the optimal number of features, and some tunings (even though backtesting and hyperparameter tuning were done, till a certain point). Consider that the expected outcome of this model should not be used as an indicator for investment decisions since the model could be refined much more. I keep the privilege to hold the full/optimized version of this trading strategy 😉.</p><p>Keywords: XGBoost, LSTM, Windowing, Feature Engineering, Window Optimization, Hyperparameter Tuning, Mean Absolute Error, Predictions.</p><h3>Table of Contents</h3><ol><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#introduction">Introduction</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#data">Data</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#feature_engineering">Feature Engineering</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#lstm-xgboost">LSTM-XGBoost</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#alternatives">Alternatives</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#conclusion">Conclusion</a></li></ol><h3>Introduction</h3><h4>Introduction</h4><p>Is there a way to predict the unpredictable?. Certainly not, either if stock data is discrete random, the probability of exactly predicting the correct price in the future is near 0%. Nonetheless, the spread of the future price can be shrunk down into a <em>confidence interval</em> that tries to reduce the exposure to the risk (volatility) of the price.</p><h3>Data</h3><h4>Data</h4><p>Apple Inc. is a publicly-traded company on the tech index NASDAQ 100. Nowadays, it is the highest valued company worldwide, with a capitalization of over 3 Billion $. To understand the selection of this asset, there is a need to point out what role the volatility of an asset plays when it comes to trading. Volatility is the standard deviation of stock prices, so if someone is pretending to trade, a higher fluctuation increases the probability of gaining more opportunities in the market, either <em>in the market</em> or <em>out of the market</em>. Since this work faces a technical analysis, the risk of deviation of a stock (volatility) needs to be a bit higher than normal. However, this strategy could also be applied to assets that do not suffer from such high fluctuations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/0*22Okt6iOxGOYlhIA.png" /></figure><h4>Cumulative Distribution Functions in and out of a crash period (i.e. 2008)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*80pXEhOBNTWb-247.png" /></figure><p>To better understand the behavior of stock price data, it is necessary to get some plottings, so that we can face how the returns, the price, and the outliers perform for this specific stock.</p><p>As seen in the histogram and in the cumulative distribution function, we can observe that the distribution of the returns does not follow a normal distribution, represented as a black line in the plot, even though it might seem to be one (a revealing indication is the higher kurtosis and fatter tails). The good thing is that the algorithms that are going to be used in this work make no assumptions according to the <a href="https://codowd.com/bigdata/misc/Taleb_Statistical_Consequences_of_Fat_Tails.pdf">distribution of the data</a>. Regarding the Box Plot, we can observe a significant amount of outliers that might harm our model. Since this can later turn into an issue, it must be considered while dealing with the features. Finally, it is also interesting how the stock performed in terms of cumulative returns, as seen in the line chart, where we can observe the evolution of the stock concerning other tech giants (appended you find the annualized returns).</p><p>Finally, it is always interesting to see how correlation behaves between assets and their sectors (Technology: <em>Orange/Brown</em>, Energy: <em>Green</em>, Health: <em>Blue</em>), in case one wanted to dig deeper into the fascinating world of portfolio management. See hoo for the period 2010 to 2020 tech stocks are hugely correlated among them (Apple, Microsoft, Alphabet), likewise, the Energy sector follows a similar pattern to tech. Moreover, you can also observe how some companies, even if not in the same sector, but as they are <em>blue chips</em> (biggest publicly traded companies in terms of capitalization) are slightly correlated.</p><h4>Correlation between Technology | Health | Energy Sector &amp; Correlation between companies (2010–2020)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*havgjxDhpWmXUkK0.png" /></figure><h3>Feature Engineering</h3><h4>Feature_Engineering</h4><p>In this section, we will discuss the new features created in order to tackle a good performance in our model. However, this shall be the densest section of the whole work, not only because the optimization of a model follows a cycle where you continuously adjust the features and see which one really does add value to it (entropy), this part will only cover the final outcome of what has been numerous hours of trying to optimize the input for my algorithm, avoiding so GIGO.</p><p>Since stock prices behave also according to the time of the year, a focus of interest while generating new features was to include the day, month, quarter, etc of that specific moment as there could have been patterns in the past. A simple example is that, by the end of January, a lot of <em>blue chips</em> release their quarterly earnings and since AAPL consistently does a good job on this, beating the expectations of the analysts, the stock tends to rise in a short time period (for one day to another). This is quite interesting since depending on the window optimization used for the analysis this pattern was captured or not.</p><p>Another focus of interest is the rolling window. These are useful methods when estimating the historical mean, standard deviation, quantile, or even the maximum/minimum. All these can be found in the code below.</p><pre>def feature_engineering(data, SPY, predictions=np.array([None])):<br>    <br>    &quot;&quot;&quot;<br>    The function applies future engineering to the data in order to get more information out of the inserted data. <br>    The commented code below is used when we are trying to append the predictions of the model as a new input feature to train it again. In this case it performed slightli better, however depending on the parameter optimization this gain can be vanished. <br>    &quot;&quot;&quot;<br>    assert type(data) == pd.core.frame.DataFrame, &quot;data musst be a dataframe&quot;<br>    assert type(SPY) == pd.core.series.Series, &quot;SPY musst be a dataframe&quot;<br>    assert type(predictions) == np.ndarray, &quot;predictions musst be an array&quot;<br>       <br>    #if predictions.any() ==  True:<br>        #data = yf.download(&quot;AAPL&quot;, start=&quot;2001-11-30&quot;)<br>        #SPY = yf.download(&quot;SPY&quot;, start=&quot;2001-11-30&quot;)[&quot;Close&quot;]<br>        #data = features(data, SPY)<br>        #print(data.shape)<br>        #data[&quot;Predictions&quot;] = predictions<br>        #data[&quot;Close&quot;] = data[&quot;Close_y&quot;]<br>        #data.drop(&quot;Close_y&quot;,1,  inplace=True)<br>        #data.dropna(0, inplace=True)<br>    else:<br>        print(&quot;No model yet&quot;)<br>        data = features(data, SPY)<br>    return data</pre><pre>def features(data, SPY):<br>    <br>    for i in [2, 3, 4, 5, 6, 7]:<br>                            <br>        # Rolling Mean<br>        data[f&quot;Adj_Close{i}&quot;] = data[&quot;Adj Close&quot;].rolling(i).mean()<br>        data[f&quot;Volume{i}&quot;] = data[&quot;Volume&quot;].rolling(i).mean()<br>        <br>        # Rolling Standart Deviation                               <br>        data[f&quot;Low_std{i}&quot;] = data[&quot;Low&quot;].rolling(i).std()<br>        data[f&quot;High_std{i}&quot;] = data[&quot;High&quot;].rolling(i).std()<br>        data[f&quot;Adj_CLose{i}&quot;] = data[&quot;Adj Close&quot;].rolling(i).std()<br>        <br>        # Stock return for the next i days<br>        data[f&quot;Close{i}&quot;] = data[&quot;Close&quot;].shift(i)<br>        <br>        # Rolling Maximum and Minimum<br>        data[f&quot;Adj_Close{i}&quot;] = data[&quot;Adj Close&quot;].rolling(i).max()<br>        data[f&quot;Adj_Close{i}&quot;] = data[&quot;Adj Close&quot;].rolling(i).min()<br>        <br>        # Rolling Quantile<br>        data[f&quot;Adj_Close{i}&quot;] = data[&quot;Adj Close&quot;].rolling(i).quantile(1)<br>    <br>    <br>    <br>    data[&quot;SPY&quot;] = SPY<br>    #Decoding the time of the year<br>    data[&quot;Day&quot;] = data.index.day<br>    data[&quot;Month&quot;] = data.index.month<br>    data[&quot;Year&quot;] = data.index.year<br>    data[&quot;day_year&quot;] = data.index.day_of_year<br>    data[&quot;Weekday&quot;] = data.index.weekday<br>                  <br>    #Upper and Lower shade<br>    data[&quot;Upper_Shape&quot;] = data[&quot;High&quot;]-np.maximum(data[&quot;Open&quot;], data[&quot;Close&quot;])<br>    data[&quot;Lower_Shape&quot;] = np.minimum(data[&quot;Open&quot;], data[&quot;Close&quot;])-data[&quot;Low&quot;]<br>    <br>                                                                            <br>    data[&quot;Close_y&quot;] = data[&quot;Close&quot;]<br>    data.drop(&quot;Close&quot;,1,  inplace=True)<br>    data.dropna(0, inplace=True)<br>    return data</pre><p>Notice that even though this is a very small amount of features, there was some filtering applied to it. Of course, when deploying this method in a real case scenario, it is recomendable to regularize the features and observe which ones clearly add value to the model.</p><h3>LSTM-XGBoost</h3><h4>LSTM-XGBoost</h4><p>Said this, lets deep into the core part of this project, where the combination of algorithms will (hopefully) provide us with reliable estimations of the Apple stock price for tomorrow.</p><p>The way to go went through an LSTM Network ensemble with an XGBoost Regressor.</p><h3>Content:</h3><ul><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#udf">UDF</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#xgboost">XGBoost</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#lstm">LSTM</a></li><li><a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#hybrid_approach">Hybrid Approach</a></li></ul><h3>UDF</h3><p>Main user-defined functions:</p><p>The first one is used for windowing the data. Although it is explained as comments inside the function, it could be fine to go over the main functionality of this function. Basically, this function slices the data into windows. This means that starting from a two-dimensional table having time as rows and the features as columns, this function enables to divide the data into fractions (3 dimensions). These fractions are considered windows.</p><p>Imagine you want to use the information of the last 7 days to see if they are able to predict accurately the future, so you will need to train your regressor using the input features to get the prediction for t+1.</p><p><em>Windowing</em> does exactly this:</p><h4>Windowing Procedure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*shf5wQ6gD30utG1d.gif" /></figure><p>Where the larger rectangle represents the input data, using a window of two, and the smaller rectangle is the output data that we are trying to predict. Notice that in this study the test set will be the green big rectangle since we want to estimate the unknown future value.</p><p>The other functions basically fulfill the need of splitting the data into the train and test set. However this could have been done using scikit-learn’s predefined function, in order to implement the window, it was easier to code a UDF. This is also the same case for the validation split.</p><p>Moreover, since the human brain feels much more comfortable when visualizing things, it was good practice to develop a function that is able to plot, the validation set, the test set, and the prediction for t+? (? since you can predict for more than one period ahead). Additionally, this plot includes written conclusions, the expected price of the stock, and the spread intervals, taken from the validation performance.</p><pre>def windowing(train, val, WINDOW, PREDICTION_SCOPE):<br>    <br>    &quot;&quot;&quot;<br>    Divides the inserted data into a list of lists. Where the shape of the data becomes and additional axe, which is time.<br>    Basically gets as an input shape of (X, Y) and gets returned a list which contains 3 dimensions (X, Z, Y) being Z, time.<br>    <br>    Input:<br>        - Train Set<br>        - Validation Set<br>        - WINDOW: the desired window<br>        - PREDICTION_SCOPE: The period in the future you want to analyze<br>        <br>    Output:<br>        - X_train: Explanatory variables for training set<br>        - y_train: Target variable training set<br>        - X_test: Explanatory variables for validation set<br>        - y_test:  Target variable validation set<br>    &quot;&quot;&quot;  <br>    <br>    assert type(train) == np.ndarray, &quot;train musst be passed as an array&quot;<br>    assert type(val) == np.ndarray, &quot;validation musst be passed as an array&quot;<br>    assert type(WINDOW) == int, &quot;Window musst be an integer&quot;<br>    assert type(PREDICTION_SCOPE) == int, &quot;Prediction scope musst be an integer&quot;<br>    <br>    X_train = []<br>    y_train = []<br>    X_test = []<br>    y_test = []</pre><pre>    for i in range(len(train)-(WINDOW+PREDICTION_SCOPE)):<br>        X, y = np.array(train[i:i+WINDOW, :-1]), np.array(train[i+WINDOW+PREDICTION_SCOPE, -1])<br>        X_train.append(X)<br>        y_train.append(y)</pre><pre>    for i in range(len(val)-(WINDOW+PREDICTION_SCOPE)):<br>        X, y = np.array(val[i:i+WINDOW, :-1]), np.array(val[i+WINDOW+PREDICTION_SCOPE, -1])<br>        X_test.append(X)<br>        y_test.append(y)<br>        <br>    return X_train, y_train, X_test, y_test<br>#-------------------------------------------------------------------------------------------------------------<br>#-------------------------------------------------------------------------------------------------------------<br>def train_test_split(data, WINDOW):<br>    &quot;&quot;&quot;<br>    Divides the training set into train and validation set depending on the percentage indicated.<br>    Note this could also be done through the sklearn traintestsplit() function.<br>    <br>    Input:<br>        - The data to be splitted (stock data in this case)<br>        - The size of the window used that will be taken as an input in order to predict the t+1<br>        <br>    Output:<br>        - Train/Validation Set<br>        - Test Set<br>    &quot;&quot;&quot;<br>    <br>    assert type(data) == pd.core.frame.DataFrame, &quot;data musst be a dataframe&quot;<br>    assert type(WINDOW) == int, &quot;Window musst be an integer&quot;<br>    <br>    train = stock_prices.iloc[:-WINDOW]<br>    test = stock_prices.iloc[-WINDOW:]<br>    <br>    return train, test<br>#-------------------------------------------------------------------------------------------------------------<br>#-------------------------------------------------------------------------------------------------------------<br>def train_validation_split(train, percentage):<br>    &quot;&quot;&quot;<br>    Divides the training set into train and validation set depending on the percentage indicated<br>    &quot;&quot;&quot;<br>    assert type(train) == pd.core.frame.DataFrame, &quot;train musst be a dataframe&quot;<br>    assert type(percentage) == float, &quot;percentage musst be a float&quot;<br>    <br>    train_set = np.array(train.iloc[:int(len(train)*percentage)])<br>    validation_set = np.array(train.iloc[int(len(train)*percentage):])<br>    <br>    <br>    return train_set, validation_set<br>#-------------------------------------------------------------------------------------------------------------<br>#-------------------------------------------------------------------------------------------------------------<br>def plotting(y_val, y_test, pred_test, mae, WINDOW, PREDICTION_SCOPE):<br>    <br>    &quot;&quot;&quot;This function returns a graph where:<br>        - Validation Set<br>        - Test Set<br>        - Future Prediction<br>        - Upper Bound<br>        - Lower Bound<br>    &quot;&quot;&quot;<br>    assert type(WINDOW) == int, &quot;Window musst be an integer&quot;<br>    assert type(PREDICTION_SCOPE) == int, &quot;Preiction scope musst be an integer&quot;<br>    <br>    ploting_pred = [y_test[-1], pred_test]<br>    ploting_test = [y_val[-1]]+list(y_test)</pre><pre>    time = (len(y_val)-1)+(len(ploting_test)-1)+(len(ploting_pred)-1)</pre><pre>    test_time_init = time-(len(ploting_test)-1)-(len(ploting_pred)-1)<br>    test_time_end = time-(len(ploting_pred)-1)+1</pre><pre>    pred_time_init = time-(len(ploting_pred)-1)<br>    pred_time_end = time+1</pre><pre>    x_ticks = list(stock_prices.index[-time:])+[stock_prices.index[-1]+timedelta(PREDICTION_SCOPE+1)]</pre><pre>    values_for_bounds = list(y_val)+list(y_test)+list(pred_test)<br>    upper_band = values_for_bounds+mae<br>    lower_band = values_for_bounds-mae<br>    <br>    print(f&quot;For used windowed data: {WINDOW}&quot;)<br>    print(f&quot;Prediction scope for date {x_ticks[-1]} / {PREDICTION_SCOPE+1} days&quot;)<br>    print(f&quot;The predicted price is {str(round(ploting_pred[-1][0],2))}$&quot;)<br>    print(f&quot;With a spread of MAE is {round(mae,2)}&quot;)<br>    print()<br>    <br>    plt.figure(figsize=(16, 8))</pre><pre>    plt.plot(list(range(test_time_init, test_time_end)),ploting_test, marker=&quot;$m$&quot;, color=&quot;orange&quot;)<br>    plt.plot(list(range(pred_time_init, pred_time_end)),ploting_pred,marker=&quot;$m$&quot;, color=&quot;red&quot;)<br>    plt.plot(y_val, marker=&quot;$m$&quot;)</pre><pre>    plt.plot(upper_band, color=&quot;grey&quot;, alpha=.3)<br>    plt.plot(lower_band, color=&quot;grey&quot;, alpha=.3)</pre><pre>    plt.fill_between(list(range(0, time+1)),upper_band, lower_band, color=&quot;grey&quot;, alpha=.1)</pre><pre>    plt.xticks(list(range(0-1, time)), x_ticks, rotation=45)<br>    plt.text(time-0.5, ploting_pred[-1]+2, str(round(ploting_pred[-1][0],2))+&quot;$&quot;, size=11, color=&#39;red&#39;)<br>    plt.title(f&quot;Target price for date {x_ticks[-1]} / {PREDICTION_SCOPE+1} days, with used past data of {WINDOW} days and a MAE of {round(mae,2)}&quot;, size=15)<br>    plt.legend([&quot;Testing Set (input for Prediction)&quot;, &quot;Prediction&quot;, &quot;Validation&quot;])<br>    plt.show()<br>    <br>    print()<br>    print(&quot;-----------------------------------------------------------------------------&quot;)<br>    print()<br>#-------------------------------------------------------------------------------------------------------------<br>#-------------------------------------------------------------------------------------------------------------<br>def inverse_transformation(X, y, y_hat):<br>    <br>    &quot;&quot;&quot;<br>    This function serves to inverse the rescaled data. <br>    There are two ways in which this can happen:<br>        - There could be the conversion for the validation data to see it on the plotting.<br>        - There could be the conversion for the testing data, to see it plotted.<br>    &quot;&quot;&quot;<br>    assert type(X) == np.ndarray, &quot;X musst be an array&quot;<br>    assert type(y) == np.ndarray, &quot;y musst be an array&quot;<br>    <br>    if X.shape[1]&gt;1:<br>        new_X = []</pre><pre>        for i in range(len(X)):<br>            new_X.append(X[i][0])<br>            <br>        new_X = np.array(new_X)<br>        y = np.expand_dims(y, 1)<br>        <br>        new_X = pd.DataFrame(new_X)<br>        y = pd.DataFrame(y)<br>        y_hat = pd.DataFrame(y_hat)</pre><pre>        real_val = np.array(pd.concat((new_X, y), 1))<br>        pred_val = np.array(pd.concat((new_X, y_hat), 1))<br>        <br>        real_val = pd.DataFrame(scaler.inverse_transform(real_val))<br>        pred_val = pd.DataFrame(scaler.inverse_transform(pred_val))<br>        <br>    else:       <br>        X = X.reshape(X.shape[0], X.shape[1]*X.shape[2])<br>        <br>        new_X = pd.DataFrame(X)<br>        y = pd.DataFrame(y)<br>        y_hat = pd.DataFrame(y_hat)<br>        y_hat = pd.concat((y, y_hat))<br>        y_hat.index = range(len(y_hat))<br>        <br>        real_val = np.array(pd.concat((new_X, y), 1))<br>        pred_val = np.array(pd.concat((new_X, y_hat), 1))<br>        <br>        pred_val = pd.DataFrame(scaler.inverse_transform(pred_val))<br>        real_val = pd.DataFrame(scaler.inverse_transform(real_val))<br>        <br>    return real_val, pred_val<br>#-------------------------------------------------------------------------------------------------------------<br>#-------------------------------------------------------------------------------------------------------------    <br>def window_optimization(plots):<br>    <br>    &quot;&quot;&quot;Returns the key that contains the most optimal window (respect to mae) for t+1&quot;&quot;&quot;<br>    <br>    assert type(plots) == dict, &quot;plots musst be a dictionary&quot;<br>    <br>    rank = []<br>    m = []<br>    for i in plots.keys():<br>        if not rank:<br>            rank.append(plots[i])<br>            m.append(i)<br>        elif plots[i][3]&lt;rank[0][3]:<br>            rank.clear()<br>            m.clear()<br>            rank.append(plots[i])<br>            m.append(i)<br>            <br>    return rank, m</pre><h3>XGBoost</h3><p>XGBoost</p><p>XGBoost, is one of the most highly used supervised ML algorithms nowadays, as it uses a more optimized way to implement a tree-based algorithm, and it is also able to efficiently manage large and complex datasets.</p><p>The methodology followed by this algorithm is the following. XGBoost uses a Greedy algorithm for the building of its tree, meaning it uses a simple intuitive way to optimize the algorithm. The algorithm combines its best model, with previous ones, and so minimizes the error. So, in order to constantly select the models that are actually improving its performance, a target is settled. This target will depend on how much the next model has decreased the error, if there has been no change in the error the target will be set to zero, otherwise, it will be set really high as surpassing its performance can be difficult for the next model.</p><p>For more insights into how this algorithm works, check out this video from <a href="https://www.youtube.com/watch?v=OtD8wVaFm6E&amp;t=649s">StatQuest</a></p><h4>Training the Model</h4><p>For training the model with the best hyperparameters and with the optimal windowing (use of past input data), a time series cross-validation was done on the data. The difference between the conventional cross-validation method is that you must ensure that the algorithm does not randomly take samples of the data to see its performance, since past data is somehow related to future events. So, there was a need to code a <em>user-defined GridsearchCV</em>, this could be done through <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ParameterGrid.html"><em>ParameterGrid</em></a>, where you insert a dictionary of parameters, and this function makes subsets including all combinations of the different parameters. Notice that the more parameters you insert and depending on how you cross-validate (backtest) the data, it is computationally expensive, therefore when implementing this, take into account what are the benefits and the drawbacks of every approach.</p><p>Hyperparamter_optimization</p><h4>In crescendo vs cluster cross-validation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/0*i2pOGSiqrHwnOqgv.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*XhHZvIBIJIqL-boR.png" /></figure><p>Let&#39;s dig into the code:</p><pre>PERCENTAGE = .995<br>WINDOW = 2<br>PREDICTION_SCOPE = 0</pre><p>The algorithm will use the past two trading days (WINDOW=2) in order to predict the next day (PREDICTION_SCOPE=0), using almost the whole data for training, but the last month (except the last <em>WINDOW</em> days), which is used for validation.</p><pre>stock_prices = feature_engineering(stock_prices, SPY)</pre><pre>Output:<br>--&gt;No model yet</pre><p>Load the data with its features.</p><p>After that, it is time to settle the train, validation, and test set. As said before the division will be done according to the selected percentage. Regarding the test set, it will make use of the settled <em>WINDOW</em> to get the X_test values.</p><pre>train, test = train_test_split(stock_prices, WINDOW)<br>train_set, validation_set = train_validation_split(train, PERCENTAGE)</pre><pre>print(f&quot;train_set shape: {train_set.shape}&quot;)<br>print(f&quot;validation_set shape: {validation_set.shape}&quot;)<br>print(f&quot;test shape: {test.shape}&quot;)</pre><pre>Output:<br>--&gt;(5047, 50)<br>--&gt;(26, 50)<br>--&gt;(2, 50)</pre><p>We apply the windowing <a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#udf">UDF</a></p><pre>X_train, y_train, X_val, y_val = windowing(train_set, validation_set, WINDOW, PREDICTION_SCOPE)</pre><pre>#Convert the returned list into arrays<br>X_train = np.array(X_train)<br>y_train = np.array(y_train)<br>X_val = np.array(X_val)<br>y_val = np.array(y_val)</pre><pre>print(f&quot;X_train shape: {X_train.shape}&quot;)<br>print(f&quot;y_train shape: {y_train.shape}&quot;)<br>print(f&quot;X_val shape: {X_val.shape}&quot;)<br>print(f&quot;y_val shape: {y_val.shape}&quot;)</pre><pre>Output:<br>--&gt;X_train shape: (5045, 2, 49)<br>--&gt;y_train shape: (5045,)<br>--&gt;X_val shape: (24, 2, 49)<br>--&gt;y_val shape: (24,)</pre><p>Since the XGBoost algorithm does not allow a three-dimensional input, there is a need to reshape the data into two dimensions. The idea behind the reshape is to join the rows of the windowed days into one big input. In this example, since our <strong>WINDOW=2</strong>, we are going to return the same amount of rows, but instead of only having 49 columns we will multiply this quantity by the WINDOW size.</p><pre>X_train = X_train.reshape(X_train.shape[0], -1)<br>X_val = X_val.reshape(X_val.shape[0], -1)</pre><pre>print(f&quot;X_train shape: {X_train.shape}&quot;)<br>print(f&quot;X_val shape: {X_val.shape}&quot;)</pre><pre>Output:<br>--&gt;X_train shape: (5045, 98)<br>--&gt;X_val shape: (24, 98)</pre><p>Finally, we only have to train the algorithm with the organized data: (The hyperparameters were optimized using the approach located to the left of the training model <a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#hyperparamter_optimization">image</a></p><pre>xgb_model = xgb.XGBRegressor(gamma=1, n_estimators=200)<br>xgb_model.fit(X_train,y_train)</pre><p>Condensed into a function:</p><pre>def xgb_model(X_train, y_train, X_val, y_val, plotting=False):</pre><pre>    &quot;&quot;&quot;<br>    Trains a preoptimized XGBoost model and returns the Mean Absolute Error an a plot if needed<br>    &quot;&quot;&quot;     <br>    xgb_model = xgb.XGBRegressor(gamma=1, n_estimators=200)<br>    xgb_model.fit(X_train,y_train)<br>    <br>    pred_val = xgb_model.predict(X_val)<br>    mae = mean_absolute_error(y_val, pred_val)</pre><pre>    if plotting == True:<br>        <br>        plt.figure(figsize=(15, 6))<br>        <br>        sns.set_theme(style=&quot;white&quot;)<br>        sns.lineplot(range(len(y_val)), y_val, color=&quot;grey&quot;, alpha=.4)<br>        sns.lineplot(range(len(y_val)),pred_val, color=&quot;red&quot;)</pre><pre>        plt.xlabel(&quot;Time&quot;)<br>        plt.ylabel(&quot;AAPL stock price&quot;)<br>        plt.title(f&quot;The MAE for this period is: {round(mae, 3)}&quot;)<br>    <br>    return  mae, xgb_model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/0*mCYK3AuGTb4FRkjO.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/0*9JUd2qSn-wjCuhLm.png" /></figure><p>Since the window size is 2, the feature performance considers twice the features, meaning, if there are 50 features, <strong>f97 == f47</strong> or likewise <strong>f73 == f23.</strong></p><h4>Add the predictions (if needed)</h4><p>Sometimes, it is also interesting, to use the training/validation predictions as a new feature, this will be used to be more accurate while passing to the test set. in this case, there was no clear evidence that this approach improves the performance, maybe more tryouts are needed in order to get nice outcomes. Nonetheless, find attached the code of how a prediction feature could be added to your model:</p><pre>#try:<br>    #y_hat_train = np.expand_dims(xgb_model.predict(X_train), 1)<br>    #array = np.empty((stock_prices.shape[0]-y_hat_train.shape[0], 1))<br>    #array[:] = np.nan<br>    #predictions = np.concatenate((array, y_hat_train))<br>#except NameError:<br>    #print(&quot;No Model&quot;)<br>    <br>#new_stock_prices = feature_engineering(stock_prices, SPY, predictions=predictions)</pre><pre>#train, test = train_test_split(new_stock_prices, WINDOW)</pre><pre>#train_set, validation_set = train_validation_split(train, PERCENTAGE)<br>#X_train, y_train, X_val, y_val = windowing(train_set, validation_set, WINDOW, PREDICTION_SCOPE)</pre><pre>#Reshaping the data<br>#X_train = np.array(X_train)<br>#y_train = np.array(y_train)</pre><pre>#X_val = np.array(X_val)<br>#y_val = np.array(y_val)</pre><pre>#X_train = X_train.reshape(X_train.shape[0], -1)<br>#X_val = X_val.reshape(X_val.shape[0], -1)</pre><pre>#new_mae, new_xgb_model = xgb_model(X_train, y_train, X_val, y_val, plotting=True)</pre><pre>#print(new_mae)</pre><h4>Testing the Model</h4><p>Of course, even if all the obtained results seem to be nice, it is important to see the model performing in a real-life situation. For this, let us obtain the prediction for the next day t+1.</p><p>To get the predictions, the same approach as for the train and validation is required.</p><pre>X_test = np.array(test.iloc[:, :-1])<br>y_test = np.array(test.iloc[:, -1])<br>X_test = X_test.reshape(1, -1)</pre><pre>print(f&quot;X_test shape: {X_test.shape}&quot;)</pre><pre>Output:<br>--&gt;X_test shape: (1, 98)</pre><p>Let&#39;s predict and plot the results:</p><pre>#Apply the xgboost model on the Test Data<br>pred_test_xgb = xgb_model.predict(X_test)<br>plotting(y_val, y_test, pred_test_xgb, mae, WINDOW, PREDICTION_SCOPE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*29WwwMFyIwN-Hq56.png" /></figure><h4>Saving the XGBoost parameters for future usage</h4><pre>#joblib.dump(xgb_model, &quot;XGBoost.pkl&quot;)</pre><h4>Window and Percentage Optimization</h4><pre>plots = {}<br>for window in [1, 2, 3, 4, 5, 6, 7, 10, 20, 25, 30, 35]:<br>    <br>    for percentage in [.92, .95, .97, .98, .99, .995]:</pre><pre>        WINDOW = window<br>        PREDICTION_SCOPE = 0<br>        PERCENTAGE = percentage</pre><pre>        train = stock_prices.iloc[:int(len(stock_prices))-WINDOW]<br>        test = stock_prices.iloc[-WINDOW:]<br>        <br>        train_set, validation_set = train_validation_split(train, PERCENTAGE)</pre><pre>        X_train, y_train, X_val, y_val = windowing(train_set, validation_set, WINDOW, PREDICTION_SCOPE)</pre><pre>        X_train = np.array(X_train)<br>        y_train = np.array(y_train)</pre><pre>        X_val = np.array(X_val)<br>        y_val = np.array(y_val)</pre><pre>        X_test = np.array(test.iloc[:, :-1])<br>        y_test = np.array(test.iloc[:, -1])</pre><pre>        X_train = X_train.reshape(X_train.shape[0], -1)<br>        try:<br>            X_val = X_val.reshape(X_val.shape[0], -1)<br>            X_test = X_test.reshape(1, -1)<br>        except ValueError:<br>            break</pre><pre>        xgb_model = xgb.XGBRegressor(gamma=1)<br>        xgb_model.fit(X_train, y_train)</pre><pre>        pred_val = xgb_model.predict(X_val)</pre><pre>        mae = mean_absolute_error(y_val, pred_val)</pre><pre>        pred_test = xgb_model.predict(X_test)<br>        plotii= [y_test[-1], pred_test]</pre><pre>        plots[str(window)+str(pred_scope)] = [y_val, y_test, pred_test, mae, WINDOW, PREDICTION_SCOPE, PERCENTAGE]<br>        <br>print()<br>print(plots[&quot;20&quot;])</pre><pre>Output:<br>--&gt;[array([179.28999329, 179.38000488, 178.19999695, 177.57000732,<br>       182.00999451, 179.69999695, 174.91999817, 172.        ,<br>       172.16999817, 172.19000244, 175.08000183, 175.52999878,<br>       172.19000244, 173.07000732, 169.80000305, 166.22999573,<br>       164.50999451, 162.41000366, 161.61999512, 159.77999878,<br>       159.69000244, 159.22000122, 170.33000183, 174.77999878]), <br>       array([174.61000061, 174.47009277]), array([171.02782], dtype=float32), 3.8374627431233725, 2, #0, 0.995]</pre><p>Extract the most optimized parameters:</p><pre>print(window_optimization(plots))</pre><p>See the multiple performances:</p><pre>for key in list(plots.keys())[5:9]:<br>    plotting(plots[key][0], plots[key][1], plots[key][2], plots[key][3], plots[key][4], plots[key][5])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*RNM_-3Ss-QMBBXeW.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/0*CNx9rw9rfT20uhuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*6bOFLqRxbjrXgTnn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/0*BXBJDO5Y9fRl3tN5.png" /></figure><h3>Long-Short Term Memory</h3><p>LSTM</p><p>Long Short Term Memory or LSTM is a type of Neural Network, which was developed on the basis provided by the Recurrent Neural Networks, or RNN. The structure of the LSTM layer can be visualized in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*RkFisWmsLWtGCByq.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*BesGS1dkd_5JI1Ed.png" /></figure><p>To get more clarifications on the syntax and the math behind this algorithm, I encourage you to visit my other <a href="https://github.com/Hupperich-Manuel/Deep_Networks">repository</a> where it gets into the fundamentals of a Neural Network or do this course of <a href="https://www.coursera.org/specializations/deep-learning">DeepLearning.AI</a>.</p><p>The algorithms main usage falls into NLPs or Time Series, and the main idea behind it relates to processing the information they receive from the previous neuron and applying the activation function from scratch (as the CNN/RNN do), they actually divide the neuron into three main parts from which to set up the input from the next layer of neurons: Learn, Unlearn and Retain gate.</p><p>While training the Apple series, several combinations of algorithms were used, whether RNNs, CNNs, or DNNs, however when it comes to time series, the LSTM has a significant advantage over its predecessor, the RNNs. For those who might not be familiar with these Neural Networks, RNNs had a scaling effect on the gradient* when the weights (W) were either very low or very high, leading to no change in the loss or an extreme change. In order to fix this, the LSTM was created, which, thanks to the different <em>gates</em> that are used in each node, is able to omit this radical change making the difference more stable (reducing the likelihood of vanishing gradients). If there is an interest to dig further in the update from an RNN to LSTM, visit <a href="https://www.geeksforgeeks.org/understanding-of-lstm-networks/">GeeksforGeeks</a>.</p><p>Nonetheless, the training procedure followed a structured methodology, which implied going from a simpler model (using only Dense layers) to a more complex one (Bidirectional LSTMs or CNNs), in the end, the LSTM returned the most optimal performance.</p><p>Notice that using the LSTM has also some drawbacks, i.e. higher computation costs, slower training, etc</p><p>For the sake of optimization, parameter tunning was needed, this entailed finding the input and hidden layer size, the batch_size, the number of epochs, and the rolling window size for the analysis that minimizes the loss function.</p><p>*<em>gradient</em>: measures the change in weights of a NN (W·x+b) according to the loss function.</p><pre>#Parameters for the LSTM<br>PERCENTAGE = .98 #Split train/val and test set<br>CALLBACK = .031 #Used to stop training the Network when the MAE from the validation set reached a perormance below 3.1%<br>BATCH_SIZE = 20 #Number of samples that will be propagated through the network. I chose almost a trading month<br>EPOCH = 50 #Settled to train the model<br>WINDOW_LSTM = 30 #The window used for the input data<br>PREDICTION_SCOPE = 0 #How many period to predict, being 0=1</pre><p>Once settled the optimal values, the next step is to split the dataset:</p><pre>train_lstm, test_lstm = train_test_split(stock_prices, WINDOW_LSTM)<br>train_split_lstm, validation_split_lstm = train_validation_split(train_lstm, PERCENTAGE)</pre><pre>train_split_lstm = np.array(train_split_lstm)<br>validation_split_lstm = np.array(validation_split_lstm)</pre><p>Rescaling to train the LSTM</p><p>To improve the performance of the network, the data had to be rescaled. This is mainly due to the fact that when the data is in its original format, the loss function might adopt a shape that is far difficult to achieve its minimum, whereas, after rescaling the global minimum is easier achievable (moreover you avoid stagnation in local minimums). For this study, the MinMax Scaler was used. The algorithm rescales the data into a range from 0 to 1. The drawback is that it is sensitive to outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e3ZC9UBAvTbpXgUG.png" /></figure><p>What is important to consider is that the fitting of the scaler has to be done on the training set only since it will allow transforming the validation and the test set compared to the train set, without including it in the rescaling. This is especially helpful in time series as several values do increase in value over time.</p><pre>scaler = MinMaxScaler()<br>scaler.fit(train_split_lstm)</pre><pre>train_scale_lstm = scaler.transform(train_split_lstm)<br>val_scale_lstm = scaler.transform(validation_split_lstm)<br>test_scale_lstm = scaler.transform(test_lstm)</pre><pre>print(train_scale_lstm.shape)<br>print(val_scale_lstm.shape)<br>print(test_scale_lstm.shape)</pre><pre>Output:<br>--&gt;(4938, 50)<br>--&gt;(101, 50)<br>--&gt;(30, 50)</pre><p>Now there is a need to window the data for further procedure. Since NN allows to ingest multidimensional input, there is no need to rescale the data before training the net.</p><pre>model_lstm = lstm_model(X_train_lstm, y_train_lstm, X_val_lstm, y_val_lstm, EPOCH, BATCH_SIZE, CALLBACK, plotting=True)</pre><pre>X_train_lstm = np.array(X_train_lstm)<br>y_train_lstm = np.array(y_train_lstm)</pre><pre>X_val_lstm = np.array(X_val_lstm)<br>y_val_lstm = np.array(y_val_lstm)</pre><pre>X_test_lstm = np.array(test_scale_lstm[:, :-1])<br>y_test_lstm = np.array(test_scale_lstm[:, -1])</pre><pre>print(X_train_lstm.shape)<br>print(X_val_lstm.shape)<br>print(X_test_lstm.shape)</pre><pre>Output:<br>--&gt;(4908, 30, 49)<br>--&gt;(71, 30, 49)<br>--&gt;(30, 49)</pre><p>Now is the moment where our data is prepared to be trained by the algorithm: Some comments:</p><ul><li>The first lines of code are used to clear the memory of the Keras API, being especially useful when training a model several times as you ensure raw hyperparameter tuning, without the influence of a previously trained model.</li><li>There was a need to create a <em>callback</em> class, which stops the iteration over the epochs when the loss function achieves a certain level of performance.</li><li>The optimal approach for this time series was through a neural network of one input layer, two LSTM hidden layers, and an output layer or Dense layer.</li><li>Each hidden layer has 32 neurons, which tends to be defined as related to the number of observations in our dataset.</li><li>For the input layer, it was necessary to define the input shape, which basically considers the window size and the number of features.</li><li>For the sake of optimization, a Stochastic Gradient Descent was used with a momentum of .85. Moreover, a learning rate scheduler was coded, that aimed to return the best performing learning rate for this series. Before settling down the final value, the neural net was trained on a small number of epochs, in order to identify the optimal value (the last stable value of the loss curve, is a reference).</li><li>For the compiler, the Huber loss function was used to not punish the outliers excessively and the metrics, through which the entire analysis is based is the Mean Absolute Error.</li><li>Finally, when fitting the model:</li><li>A batch size of 20 was used, as it represents approximately one trading month. The batch size is the subset of the data that is taken from the training data to run the neural network.</li><li>The number of epochs sums up to 50, as it equals the number of exploratory variables.</li><li>The callback was settled to 3.1%, which indicates that the algorithm will stop running when the loss for the validation set undercuts this predefined value. This means that the data has been trained with a spread of below 3%.</li></ul><pre>tf.keras.backend.clear_session()<br>tf.random.set_seed(51)<br>np.random.seed(51)</pre><pre>class myCallback(tf.keras.callbacks.Callback):<br>    def on_epoch_end(self, epoch, logs={}):<br>        if (logs.get(&quot;val_mae&quot;)&lt;CALLBACK):<br>            print(&quot;\n Accuracy reached %, so cancelling training&quot;)<br>            self.model.stop_training=True</pre><pre>callbacks = myCallback()</pre><pre> model = tf.keras.models.Sequential([<br>    tf.keras.layers.LSTM(32, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True),<br>    tf.keras.layers.LSTM(32),<br>    tf.keras.layers.Dense(1)<br>])</pre><pre>#lr_schedule = tf.keras.callbacks.LearningRateScheduler(<br>    #lambda epoch: 0.228 * 10**(epoch / 20))<br>optimizer = tf.keras.optimizers.SGD(learning_rate=0.228, momentum =.85)<br>model.compile(loss=tf.keras.losses.Huber(), optimizer=optimizer, metrics=&quot;mae&quot;)<br>history = model.fit(X_train, y_train,batch_size=BATCH_SIZE, epochs=EPOCH,callbacks=[callbacks],  validation_data=[X_val, y_val], verbose=1)</pre><pre>Output:<br>Epoch 1/30<br>328/328 [==============================] - 6s 13ms/step - loss: 5.1591e-04 - mae: 0.0161 - val_loss: 0.0046 - val_mae: 0.0819<br>Epoch 2/30<br>328/328 [==============================] - 4s 11ms/step - loss: 1.1767e-04 - mae: 0.0099 - val_loss: 0.0036 - val_mae: 0.0696<br>Epoch 3/30<br>328/328 [==============================] - 4s 11ms/step - loss: 9.9146e-05 - mae: 0.0093 - val_loss: 0.0024 - val_mae: 0.0530<br>Epoch 4/30<br>328/328 [==============================] - 4s 12ms/step - loss: 9.9358e-05 - mae: 0.0092 - val_loss: 0.0019 - val_mae: 0.0452<br>Epoch 5/30<br>328/328 [==============================] - 4s 11ms/step - loss: 7.3892e-05 - mae: 0.0077 - val_loss: 0.0012 - val_mae: 0.0355<br>Epoch 6/30<br>328/328 [==============================] - 4s 11ms/step - loss: 7.3492e-05 - mae: 0.0078 - val_loss: 0.0012 - val_mae: 0.0343<br>Epoch 7/30<br>328/328 [==============================] - 4s 12ms/step - loss: 6.5693e-05 - mae: 0.0073 - val_loss: 0.0025 - val_mae: 0.0554<br>Epoch 8/30<br>328/328 [==============================] - 6s 17ms/step - loss: 6.7699e-05 - mae: 0.0074 - val_loss: 0.0017 - val_mae: 0.0418<br>Epoch 9/30<br>328/328 [==============================] - 7s 21ms/step - loss: 6.3419e-05 - mae: 0.0074 - val_loss: 0.0012 - val_mae: 0.0361<br>Epoch 10/30<br>328/328 [==============================] - 5s 15ms/step - loss: 5.7910e-05 - mae: 0.0068 - val_loss: 0.0012 - val_mae: 0.0347<br>Epoch 11/30<br>328/328 [==============================] - 5s 15ms/step - loss: 5.7053e-05 - mae: 0.0068 - val_loss: 0.0024 - val_mae: 0.0539<br>Epoch 12/30<br>328/328 [==============================] - 5s 14ms/step - loss: 5.3417e-05 - mae: 0.0065 - val_loss: 0.0011 - val_mae: 0.0335<br>Epoch 13/30<br>328/328 [==============================] - 5s 15ms/step - loss: 5.8176e-05 - mae: 0.0069 - val_loss: 0.0014 - val_mae: 0.0377<br>Epoch 14/30<br>328/328 [==============================] - 5s 14ms/step - loss: 5.0530e-05 - mae: 0.0063 - val_loss: 0.0020 - val_mae: 0.0504<br>Epoch 15/30<br>327/328 [============================&gt;.] - ETA: 0s - loss: 5.0619e-05 - mae: 0.0063<br> Accuracy reached %, so cancelling training<br>328/328 [==============================] - 5s 15ms/step - loss: 5.0593e-05 - mae: 0.0063 - val_loss: 8.6126e-04 - val_mae: 0.0298</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*BnHhsKdyohP4yori.png" /></figure><p>Notice that the loss curve is pretty stable after the initial sharp decrease at the very beginning (first epochs), showing that there is no evidence the data is overfitted</p><p>Nonetheless, the loss function seems extraordinarily low, one has to consider that the data were rescaled. In order to defined the real loss on the data, one has to inverse transform the input into its original shape. This is done with the <em>inverse_transformation</em> <a href="https://github.com/Hupperich-Manuel/LSTM-XGBoost/edit/main/README.md#udf">UDF</a>.</p><pre>#Set up predictions for train and validation set<br>y_hat_lstm = model_lstm.predict(X_val_lstm)<br>y_hat_train_lstm = model_lstm.predict(X_train_lstm)</pre><pre>#Validation Transormation<br>mae_lstm = mean_absolute_error(y_hat_lstm, y_hat_lstm)<br>real_val, pred_val = inverse_transformation(X_val_lstm, y_val_lstm, y_hat_lstm)<br>mae_lstm = mean_absolute_error(real_val.iloc[:, 49], pred_val.iloc[:, 49])</pre><p>Validation Set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*_6A2K2ZLQqDwtfzU.png" /></figure><pre>real_train, pred_train = inverse_transformation(X_train_lstm, y_train_lstm, y_hat_train_lstm)</pre><p>Training Set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*B8LlTeRuDy8QnERI.png" /></figure><pre>tf.keras.backend.clear_session()<br>tf.random.set_seed(51)<br>np.random.seed(51)</pre><pre>def lstm_model(X_train, y_train, X_val, y_val, EPOCH,BATCH_SIZE,CALLBACK,  plotting=False):<br>    <br>    class myCallback(tf.keras.callbacks.Callback):<br>        def on_epoch_end(self, epoch, logs={}):<br>            if (logs.get(&quot;val_mae&quot;)&lt;CALLBACK):<br>                print(&quot;\n Accuracy % so cancelling training&quot;)<br>                self.model.stop_training=True</pre><pre>    callbacks = myCallback()<br>    <br>    model = tf.keras.models.Sequential([<br>        tf.keras.layers.LSTM(32, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True),<br>        tf.keras.layers.LSTM(32),<br>        tf.keras.layers.Dense(1)<br>    ])</pre><pre>    lr_schedule = tf.keras.callbacks.LearningRateScheduler(<br>        lambda epoch: 0.228 * 10**(epoch / 20))<br>    optimizer = tf.keras.optimizers.SGD(learning_rate=0.228, momentum =.85)<br>    model.compile(loss=tf.keras.losses.Huber(), optimizer=optimizer, metrics=&quot;mae&quot;)<br>    history = model.fit(X_train, y_train,batch_size=BATCH_SIZE, epochs=EPOCH,callbacks=[callbacks],  validation_data=[X_val, y_val], verbose=1)<br>    <br>    if plotting == True:<br>        plt.figure(figsize=(18, 6))</pre><pre>        lrs = 1e-5 * (10 ** (np.arange(len(history.history[&quot;loss&quot;])) / 20))<br>        plt.semilogx(lrs, history.history[&quot;loss&quot;])<br>        plt.xticks(size=14)<br>        plt.show()<br>    <br>    return model</pre><h4>Testing the Model</h4><p>Let&#39;s obtain the prediction for t+1:</p><pre>X_test_formula = X_test_lstm.reshape(X_test_lstm.shape[0], 1, X_test_lstm.shape[1])<br>print(X_test_formula.shape)</pre><pre>Output:<br>--&gt;(30, 1, 49)# 30 rows (window=30) | 1 Cluster of Rows | 49 features</pre><p>See that the shape is not what we want, since there should only be 1 row, which entails a window of 30 days with 49 features. That is why there is a need to reshape this array.</p><pre>X_test_lstm = X_test_formula.reshape(1, X_test_formula.shape[0], X_test_formula.shape[2])<br>print(X_test_lstm.shape)</pre><pre>Output:<br>--&gt;(30, 1, 49)# 30 rows (window=30) | 1 Cluster of Rows | 49 features</pre><p>Now we can predict:</p><pre>y_hat_test_lstm = model_lstm.predict(X_test_lstm)<br>real_test, pred_test = inverse_transformation(X_test_lstm, y_test_lstm, y_hat_test_lstm)</pre><pre>#For plotting purposes<br>y_val_lstm = np.array(real_val.iloc[-30:, 49])<br>y_test_lstm = np.array(real_test.iloc[:, 49])<br>pred_test = np.array(pred_test.iloc[-1:, 49])<br>mae_lstm = mean_absolute_error(real_val.iloc[:, 49], pred_val.iloc[:, 49])</pre><pre>#UDF<br>plotting(y_val_lstm, y_test_lstm, pred_test, mae_lstm, WINDOW_LSTM, PREDICTION_SCOPE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*DFpR2oIN8Nsx78E5.png" /></figure><h4>Saving the LSTM parameters for transfer learning</h4><pre>#model_lstm.save(&#39;./LSTM&#39;)<br>#lstm_model = tf.keras.models.load_model(&quot;LSTM&quot;) //in case you want to load it</pre><h3>Ensemble Modelling</h3><p>Hybrid_Approach</p><p>In order to get the most out of the two models, a good practice is to combine those two and apply a higher weight on the model which got a lower loss function (mean absolute error). The reason is mainly that sometimes a neural network performs really well on the loss function, but when it comes to a real-life situation, the algorithm only learns the shape of the original data and copies this with one delay (+1 lag). Combining this with a decision tree regressor might mitigate this duplicate effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*4WXmZdXK6Bq_srtE.png" /></figure><p>In our case we saw that the MAE of the LSTM was lower than the one from the XGBoost, therefore we will give a higher weight on the predictions returned from the LSTM model.</p><pre>mae_xgboost = mae<br>print(f&quot;The LSTM prediction is: {pred_test})<br>print(f&quot;The XGBoost prediction is: {pred_test_xgb})</pre><pre>Output:<br>--&gt;The LSTM prediction is: array([158.05817538])<br>--&gt;The XGBoost prediction is: array([157.07529])</pre><p>Let&#39;s apply the above-defined formula:</p><pre>prediction_ensemble = predictions(mae_lstm, mae_xgboost, pred_test_xgb, pred_test)<br>avg_mae = (mae_lstm + mae_xgboost)/2<br>plotting(y_val, y_test, prediction_ensemble, avg_mae, WINDOW, PREDICTION_SCOPE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*HhvfD_stOv1DtqEB.png" /></figure><h3>Alternatives</h3><p>Alternatives</p><p>Driving into the end of this work, you might ask why don’t use simpler models in order to see if there is a way to benchmark the selected algorithms in this study. So, for this reason, several <em>simpler</em> machine learning models were applied to the stock data, and the results might be a bit confusing.</p><pre>#Linear Regression<br>lr = LinearRegression()<br>lr.fit(X_train_reg, y_train_reg)<br>y_hat_lr= lr.predict(X_val_reg)<br>mae_lr = mean_absolute_error(y_val_reg, y_hat_lr)</pre><pre>print(&quot;MSE: {}&quot;.format(np.mean((y_hat_lr-y_val_reg)**2)))<br>print(&quot;MAE: {}&quot;.format(mae_lr))</pre><pre>Output: <br>--&gt;MSE:14.15..<br>--&gt;MAE:2.97...</pre><pre>#Random Forest Regressor<br>rf = RandomForestRegressor()<br>rf.fit(X_train_reg, y_train_reg)<br>y_hat_rf= rf.predict(X_val_reg)<br>mae_rf= mean_absolute_error(y_val_reg, y_hat_rf)</pre><pre>print(&quot;MSE: {}&quot;.format(np.mean((y_hat_rf-y_val_reg)**2)))<br>print(&quot;MAE: {}&quot;.format(mae_rf))</pre><pre>Output: <br>--&gt;MSE:94.60..<br>--&gt;MAE:8.59...</pre><p>Focusing just on the results obtained, you should question why on earth using a more complex algorithm as LSTM or XGBoost it is. Well, the answer can be seen when plotting the predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*OpA-hMemeWgPpici.png" /></figure><pre>pred_test_lr = lr.predict(X_test_reg)<br>plotting(y_val_reg, y_test_reg, pred_test_lr, mae_lr, WINDOW, PREDICTION_SCOPE)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*Qlxs6wrDwgtPCm7E.png" /></figure><p>See that the outperforming algorithm is the Linear Regression, with a very small error rate. Nonetheless, as seen in the graph the predictions seem to replicate the validation values but with a lag of one (remember this happened also in the LSTM for small batch sizes). So, if we wanted to proceed with this one, a good approach would also be to embed the algorithm with a different one. This would be good practice as you do not further rely on a unique methodology.</p><h3>Conclusion</h3><p>Conclusion</p><p>Reaching the end of this work, there are some key points that should be mentioned in the wrap up:</p><p>The first thing is that this work has more about self-development and a way to connect with people who might work on similar projects and want to engage with than to obtain skyrocketing profits. As said at the beginning of this work, the extended version of this code remains hidden in the VSCode of my local machine. Nonetheless, one can build up really interesting stuff on the foundations provided in this work.</p><p>The second thing is that the selection of the embedding algorithms might not be the optimal choice, but as said in point one, the intention was to learn, not to get the highest returns. Learning about the most used tree-based regressor and Neural Networks are two very interesting topics that will help me in future projects, those will have more a focus on computer vision and image recognition.</p><p>Regarding hyperparameter optimization, someone has to face sometimes the limits of its hardware while trying to estimate the best performing parameters for its machine learning algorithm. Nonetheless, I pushed the limits to balance my resources for a good-performing model.</p><p>When it comes to feature engineering, I was able to play around with the data and see if there is more information to extract, and as I said in the study, this is in most of the cases where ML Engineers and Data Scientists probably spend the most of their time. Whether it is because of outlier processing, missing values, encoders, or just model performance optimization, one can spend several weeks/months trying to identify the best possible combination.</p><p>More than ever, when deploying an ML model in real life, the results might differ from the ones obtained while training and testing it. Therefore, it is recomendable to always upgrade the model in case you want to make use of it on a real basis.</p><p>Said this, I wanted to thank those that took their time to help me with this project, guiding me through it or simply pushing me to go the extra mile.</p><p>I hope you enjoyed this case study, and whenever you have some struggles and/or questions, do not hesitate to contact me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ff9e419c020" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Titanic Predictions with LDA]]></title>
            <link>https://medium.com/@mahupperich/titanic-predictions-with-lda-267f3b888ad1?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/267f3b888ad1</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lda]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Sun, 28 Nov 2021 11:59:59 GMT</pubDate>
            <atom:updated>2021-11-28T14:55:03.387Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*4b74w1AuPig9s5fb.png" /></figure><h3>Titanic Classification with LDA</h3><p>Classified a binary outcome, either survived at the titanic or not, according to different features</p><p>The titanic is one of the most iconic and at the same time saddest stories in the history of human beings. There are barely any individuals who are not familiar with its story and how lucky some people were on that liner, just because of certain characteristics that they took with them. Whether they were kids or had a higher purchasing power, there was a pattern to follow when predicting the probability of getting a safe boat, and so leaving unharmed the ship.</p><h3>1. Explanatory Data Analysis</h3><p>The cleaning of the data is by far the most challenging part in most of the machine learning projects since you can extremely improve (or harm) your model according to the individual features and the types of features you train your model with.</p><h3>1.1. Feature Selection</h3><p>For feature selection, we will go through three main aspects.</p><p>The first approach tried to identify the features that would add no value to our classification, as they are not significant. Those features were the ID of the Passenger, the Name of this, the Embarked location, and the Ticket number. You might argue that the Ticket number could be an indicator of the purchasing power of the Passenger, however, there were other variables such as Fare, that actually delivered the Price of the ordered Cabin, giving us enough indices to correctly classify the passengers. Lastly, after having trained our model (but without having accomplished the desired score), there was evidence that the Cabin feature did all but a good job on the model since there were a lot of Null values contained in it (even replacing the values with dummy variables, did not improve the performance).</p><p>The second one focuses on the identification of multicollinearity in the data and eliminates the explanatory variables which are highly correlated as they will double their effect/importance in the model, and therefore do not show the reality of the data. Consequently, we drop those columns too.</p><p>The third approach was more focused on the normalization of the data. Since the data combined Categorical and Continuous Variables, there was a need to rescale the data so that there were no imbalances. To do so, a pipeline was coded, that aimed to pass the data through a Standard Scaling (setting the mean = 0 and standard deviation = 1)and then apply Principal Component Analysis (PCA), since the chosen algorithm could easily overfit the data, but thanks to the prior dimensionality reduction, this issue could be tackled (there could have been other dimensionality reducers as the regularized LDA, which makes use of the shrinkage method, but it worked without this more complex approach).</p><h3>2. Train the Model</h3><p>The way the model was approached went through an initial study of the data, observing its behavior. As it could be seen, some patterns indicated to classifiers such as the Linear Discriminant Analysis (LDA), Logistic Regression, or the Bayes Classifier. Therefore an ensemble modeling seemed to be a straightforward method to quickly identify the performance of multiple algorithms at a time and then eliminate the underperforming ones. Finally, the model that performed the best on the given data ended up being LDA, so a deeper study needed to be done on this specific algorithm.</p><h3>2.1. Linear Discriminant Analysis</h3><p>With the LDA we are going to model the conditional distribution of the independent variable Y (1,0), given the explanatory variables X. The idea behind LDA is to use the logistic regression modeling Pr(Y = k|X = x), but through a less direct approach, which is basically to model the distribution of the features X given Y Pr(X = x|Y = k) and then use the Bayes theorem to flip it to estimate Pr(Y = k|X = x). In other words, instead of predicting what is the probability of surviving or not given certain attributes, the selected algorithm rephrases the estimation saying what is the probability that a person has a specific profile, given that she/he has survived or not, and then since some assumptions are fulfilled, we can flip this assumption and estimate the probability for the test sample. So, in order to get positive takeaways from the model the data has to follow some assumptions:</p><ul><li>The classes are well separated.</li><li>The observations follow a multivariate Gaussian distribution, as it assumes that each feature follows a one-dimensional normal distribution X ∼ N(μ, Σ), with some correlation between the pairs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/0*N_MZKmDGcOUmOTrb.png" /><figcaption><em>Source: Seaborn Library Python 3.0.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/0*rpdu-LocN4KUU4UB.png" /><figcaption><em>Source: Stats Jarque-Bera Test Python 3.0.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*Ve0dUW13dXduTg5S.png" /><figcaption><em>Source: Matplotlib Pyplot Library Python 3.0.</em></figcaption></figure><p>LDA reduces the dimensionality of the data (similar to PCA), with the main difference that it focuses on maximizing the separability between the two classes. In the case of a dataset with more than one explanatory variable, we will handle the maximization through a mean vector and a covariance matrix, aiming not only to maximize the means but to minimize the scatter.</p><p>In order to calculate the probability that a certain observation belongs to the <em>kth</em> class, we denote the density function as follows: fk(X) ≡ Pr(X = x|Y = k). So, applying the Bayes theorem to this formula we end up getting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/0*naJud10Xv-OiP6gF.png" /></figure><p>Where:</p><ul><li>πk is the prior probability that an observation belongs to <em>kth</em> class (which is simple in this case as we only have to compute the fraction of the training observations that belong to the <em>kth</em> class).</li><li>when it comes to the estimation of fk(X) we will have to see if we can approximate the Bayes classifier (for a given observation, which class maximizes pk(X), while at the same time reducing the error rate).</li><li>The density function fk(X), should be large if there is a high probability that the observation <em>X</em> belongs to the <em>kth</em> class, and vice versa.</li></ul><p>Given that we have more than one feature we will denote the multivariate Gaussian density function as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*3ZljydUv_zJ1TG1q.png" /></figure><p>Where:</p><ul><li>μ is a vector of means of the variables</li><li>Σ is the covariance matrix of the variables</li></ul><p>Now we only have to replace this formula in the Bayes formula, which I previously mentioned, and see what is the maximum value of:</p><p>δk(x) = xT Σ−1μk − 1/2μTk Σ−1μk + log πk (LDA depends on the linear combination of the elements of x)</p><p>given an observation classified as a <em>kth</em> class.</p><p>The decision boundaries are the only points where the probability to belong to one class is equal to one belonging to the other. In our case, we will have one decision boundary, as we only have 2 classes.</p><p>When it comes to the evaluation we can see how the error rate was minimized, perfectly defining the True positives and the True Negatives:</p><figure><img alt="Source: Matplotlib Pyplot Library Python 3.0." src="https://cdn-images-1.medium.com/max/355/0*hCU8RzIj1CTgLeU6.png" /><figcaption><em>Source: Matplotlib Pyplot Library Python 3.0.</em></figcaption></figure><p>Looking at the sensitivity (True positive rate) and the specificity (True negative rate), which characterize the performance of our LDA, compared to other classifiers, we observe that it is almost perfect. The AUC is about 97%, which shows that there is a large increase in true positives with a little to no change in false positives, the model fits almost perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*Cy5WW-UWI57Mebi3.png" /><figcaption><em>Source: Matplotlib Pyplot Library Python 3.0.</em></figcaption></figure><h3>3. Code the LDA</h3><p>Now we are entering into the interesting part of this study, the code. We covered the whole code in a function that returns a dataframe with the y_hat predictions and the Passenger_ID. In this section, we will break up the code into its components explaining so each step.</p><p>The first part was the import of the different libraries and the datasets:</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.pipeline import Pipeline<br>from sklearn.metrics import roc_curve, auc<br>from sklearn.metrics import accuracy_score<br>from sklearn.discriminant_analysis import LinearDiscriminantAnalysis<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.decomposition import PCA<br>import warnings<br>warnings.filterwarnings(&quot;ignore&quot;)</pre><pre>df_train = pd.read_csv(&quot;train.csv&quot;, index_col=0)<br>df_test = pd.read_csv(&quot;test.csv&quot;, index_col=0)<br>gender_sub = pd.read_csv(&quot;gender_submission.csv&quot;, index_col=0)</pre><p>Then as said previously, there was a need to drop the unnecessary columns and so weigh only the one who truly would add value to our model:</p><pre>#Feature Selection<br>df_test = pd.merge(df_test, gender_sub, left_index=True, right_index=True) #merges the test set to see if there are missing values<br>df_test[&#39;Sex&#39;] = df_test[&#39;Sex&#39;].replace(&quot;male&quot;, 0).replace(&quot;female&quot;, 1) #binary values for the &#39;Sex&#39; feature<br>df_train[&#39;Sex&#39;] = df_train[&#39;Sex&#39;].replace(&quot;male&quot;, 0).replace(&quot;female&quot;, 1)<br>df_train.drop([&#39;Name&#39;, &#39;Embarked&#39;, &#39;Ticket&#39;, &#39;Cabin&#39;], 1, inplace=True) #dropping the irrelevant variables<br>df_test.drop([&#39;Name&#39;, &#39;Embarked&#39;, &#39;Ticket&#39;,&#39;Cabin&#39;], 1, inplace=True)</pre><p>Since the data could have some Null values in its columns, there was a need to identify if the frequency of the Null Values was significant (basically to make a decision whether to eliminate the rows or not). Moreover, since ‘Pclass’ had three categorical variables, which referred to the ticket class the Passenger traveled with, dummy variables were created for simplification.</p><pre>#Was for training set and the test set<br>for i in df_test.columns:<br>    print(i, sum(df_test[i].isnull())/df_test.shape[0])<br>    <br>age = []</pre><pre>for i in df_train[&#39;Age&#39;]:<br>    if str(i) == &#39;nan&#39;:<br>        age.append(df_train[&#39;Age&#39;].mean())<br>    else:<br>        age.append(i)</pre><pre>df_train[&#39;Age&#39;] = age</pre><pre>df_train = pd.get_dummies(df_train, columns=[&#39;Pclass&#39;])<br>df_test = pd.get_dummies(df_test, columns=[&#39;Pclass&#39;])</pre><pre>age = []</pre><pre>for i in df_test[&#39;Age&#39;]:<br>    if str(i) == &#39;nan&#39;:<br>        age.append(df_test[&#39;Age&#39;].mean())<br>    else:<br>        age.append(i)</pre><pre>df_test[&#39;Age&#39;] = age</pre><pre>df_test.dropna(inplace=True)</pre><p>Finally, the pandas DataFrame was converted into arrays to be trained on the sklearn libraries</p><pre>train_X, train_y = np.array(df_train[df_train.columns[1:]]),  np.array(df_train[df_train.columns[0]])<br>test_X, test_Y =  np.array(df_test[df_test.columns[:-1]]), np.array(df_test[df_test.columns[-1:]])</pre><pre>train_Y = np.expand_dims(train_y, axis=0) #this converts the shape from (15,) into (15,1), which need to be done, otherwise the sklearn libraries will return an error<br>train_Y = train_Y.T</pre><p>Now the data was ready to be trained. To do a Pipeline was coded that Standardized, Reduced, and Trained the Data with the Linear Discriminant Analysis algorithm (with its standard parameters), through cross-validation of 5 and the return of the training score. After that, we fitted the training X features with the training Y (‘Survived’) and evaluated the predictions with the score.</p><pre>pipe = Pipeline([(&#39;scaler&#39;, StandardScaler()),(&#39;pca&#39;,PCA()),(&#39;model&#39;,LinearDiscriminantAnalysis())])</pre><pre>params = {&#39;model__solver&#39;:[&#39;svd&#39;]}</pre><pre>search = GridSearchCV(pipe, param_grid=params, cv = 5, return_train_score=True)</pre><pre>search.fit(train_X, train_Y)<br>y_hat = search.predict(test_X)</pre><pre>score = search.score(test_X, test_Y)<br>#print(&quot;LDA Score: %.02f&quot;%(score))</pre><p>The score was 1.0 which is the highest possible value you can get, getting us to the top 170 performers of the Kaggle competition.</p><h3>4. Conclusion</h3><p>To conclude I would like to thank my partner <a href="https://www.linkedin.com/in/ignacio-gonzalez-granero/">Igancio Gonzalez Granero</a>, for joining me in this competition. To continuously seek new challenges that get us out of our comfort zone, and make us improve in every aspect of our lives are some of the values that define us. Through Data Science competitions, we are enabled to gain a lot of knowledge, which could not be more useful, when achieving the goals that each of us had previously settled.</p><p>Have a nice day!</p><h3>5. References</h3><ul><li><a href="https://stackoverflow.com/">Stack Overflow</a></li><li><a href="https://www.statlearning.com/">Introduction to Statistical Learning</a></li><li><a href="https://www.kaggle.com/c/titanic/data?select=test.csv">Kaggle</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=267f3b888ad1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[APPLICATION OF THE GARCH MODEL ON FINANCIAL TIME SERIES]]></title>
            <link>https://medium.com/@mahupperich/application-of-the-garch-model-on-financial-time-series-83131414ee2f?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/83131414ee2f</guid>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Fri, 22 Oct 2021 07:54:04 GMT</pubDate>
            <atom:updated>2021-10-22T08:24:41.096Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*pSQmccMxVvZKgHaNOt9-7g.png" /></figure><p>Wanting to become a DataScientist in the near future, I early began to implement several models on data in order to seek the truth behind those values. In this case, I will be using the Bollerslev GARCH model to see if stock market volatility can be used for risk hedging in portfolio construction.</p><p>Feel free to share your opinion on this work, and check also my other projects that are on my <a href="https://github.com/Hupperich-Manuel?tab=repositories">GitHub</a> repositories.</p><p>Enjoy the read!!!</p><h3>GARCH_Model_Startegy</h3><p>This code returns the performance of a statistical GARCH model trading strategy</p><h3>1. INTRODUCTION</h3><p>To properly introduce this work, I have to mention one of my life references, Nassim Taleb’s point of view about financial markets, basically, if you think that you can handle it or even predict it, you have lost. The Gaussian bell curve, represented by a normal distribution, does not consider the events in Extremistan1, and this is the main and most mistaken assumption in the financial environment. When it comes to portfolio hedging this misinterpretation takes with it a lot of risk and could end up with high capital losses. There are numerous theories which imply this assumption, a good example could be the Nobel Prize winner William Forsyth Sharpe (1990). Together with other economists, they developed a ratio where he made use of the Gaussian curve to predict the future volatility of a certain series (a risk-return ratio, which worked on the same basis, got included in Markowitz’s Modern Portfolio Theory 1952). Unfortunately, by the time of 2007 any portfolio model based on this ratio failed because of the appearance of an unpredictable event, a Black Swan2. Even by considering this event as “predictable”, as the default risk could have been identified through a rational way (turning the Black Swan into a Grey Swan), no ratio did foresee that.</p><p>Another false assumption is done trying to estimate the Value at Risk through a Monte Carlo generator, where the series are considered to be normally distributed, and it returns the lowest possible price value that the series can reach in a certain time window. But it is a huge mistake to assume the returns are normally distributed, as it implies that the rare events can be estimated. A perfect bell curve does not exist in this environment. As hypocrite as it sounds, this work would not be the optimal solution for Mr. Taleb, since we are going to forecast a future outcome basing ourselves on a past event, same as our Nobel Prize counterparts. But, the complex models studied in this work, consider a conditional volatility and conditional returns, which has shown to make certain sense. Periods with high participation and radical movements in the financial markets are followed by similar periods, and vice versa. This is quite reasonable if you remember high volatility in the markets have been historically relevant, mostly in the way of downturns. Furthermore, we erase the assumption of a normal standard distribution, proving so our lower ignorance. So, as having mitigated these misperceptions we are able to take advantage of the results to even hedge your portfolio, or even taking advantage of the radical market volatility to earn considerable returns in the form of a trading strategy.</p><p>The inspiration I got to get here is basically thanks to my track record. During the past 6 years I have gotten deeply involved with the financial markets. I started in this system focusing merely on trading (speculating) the forex market, it allowed me to study most of the technical analysis strategies which are based on statistical probabilities. As I got further in my bachelor degree, I started to course: Statistics, Forecasting Methods and Econometrics, which gave me a quick view about the fundamentals of the statistical environment. Having red several books from Nassim Nicholas Taleb and other professionals I realized the great profitability that the statistical models could generate inside of a trading strategy and how false statistical/logical assumptions have harmed firms, funds, governments and individuals. Later on, I became aware of the importance of software in order to manage much more information to get more accurate results leading so to higher profits through a trading system. That is why I started to tech myself, with several resources, the so complex coding languages (for this work I needed to learn a whole coding language from zero).</p><p>This work focuses on the usage of the conditional variance models on a financial time series, where it is very common to observe that this variable suffers significant changes over a time frame. The models used, known as ARCH and GARCH (Autoregressive and General Autoregressive Conditional Heteroscedasticity Models), are able to pick up the unstable error term, which results from the conditional mean estimation (ARIMA), leading so to a correct adjustment of the time series.</p><p>With the outcomes acquired from the R code, we will enable to extrapolate the information obtained from the model and implement it into a functioning investment system. This strategy will have its core point in buying or selling mid-term volatility ETNs, through the accomplishment of stated conditions of our studied heteroscedasticity models.</p><p>The main message to be transmitted through the whole work is to show the importance of a volatility hedger in any kind of investment purpose, since it is the principal driver of huge losses of investment funds, traders and individual investors. Additionally, I want to raise awareness about the inability to forecast the financial markets, and that it relies on a logical mind to consider the numerous risks.</p><h3>2. GARCH MODEL AND ITS TAKEAWAYS</h3><h3>2.1 The ARCH and GARCH Models</h3><p><em>2.1.1. How the Heteroscedasticity variance works, and how such models can be implemented in the time series modelling</em></p><p>Let’s start with a brief introduction to understand how the heteroscedasticity variance works, and how can we implement such models on a time series.</p><p>Most of the financial time series, as shares, interest rates, exchange rates have certain characteristics in common, which are not considered by the conventional classical lineal models such as AR or ARMA.</p><p>One of the main watermarks of these series is the conditional heteroscedasticity, that means, a conditional variance which varies through the series. To get rid of these problems the Autoregressive Conditional Heteroscedasticity Models where created.</p><p>The ARCH Model dates back to Engel (1982), where the conditional variance can be explained by the standardized squared errors. Bollerslev (1989) works with the GARCH Model. The GARCH Model from Bollerslev (1986), which explains the conditional variance through the squared errors and the past conditional variances.</p><p>Later on, models such as EGARCH, TGARCH, IGARCH, GJRGARCH were derived from these two.</p><h3>2.2. ARCH(q): Definition and Properties</h3><p><em>2.2.1. Properties of the ARCH(q)</em> The Autoregressive Conditional Heteroscedasticity Models help us to model financial time series, as the returns on a financial asset.</p><p>The returns on a financial asset can be formulated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*M4aXCZg5gd-0lsSQ.png" /></figure><p>The core ARCH formula, which allows us to predict future returns is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*6C1b-XvsThLVzFXL.png" /></figure><p>ε_t is the process of White noise formed by random variables normally distributed with mean=0 and variance =1 and 𝜎_t is the volatility factor, both are stationary processes and independent beyond them.</p><p>The independence of the aforementioned variables allows the returns mean to be zero, that there is no autocorrelation and enables the generation of White Noise. This can be seen in the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*ICtJ67ORiKH9jyjy.png" /></figure><p>So, the conditional mean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*8cgmmnsTzMmQm5uW.png" /></figure><p>As the returns (r_t) are a stationary process, we become a constant marginal σ². But as stated before the differentiation starts here, with the implementation of a condition on this constant marginal variance, turning into conditional variance.</p><p>Marginal Variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*AaFB-6_7H2O9Mqpl.png" /></figure><p>Conditional Variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*lFS8DsbXgt5jt-0M.png" /></figure><p>The volatility represents the conditional variance of the time series at a specific point in time, which varies depending on the series structure.</p><p><em>2.2.2. Model Estimation ARCH(p)</em></p><p>Said this, we can state that the we will start from a return series with a mean equal to 0, and a non-constant conditional variance (σ_(t/(t-p))²), which depends on the past returns. But first, in order to become an unbiased estimator of σ_(t/(t-p))², we square the returns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*jYz2sQC8pUmTbngC.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*2H14FrqnPEwyTTOz.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*lPxPDMec8eilNh38.png" /></figure><ul><li>Where:</li><li>α_0 lowest observed value of the conditional variance and α_i is a necessary condition to prove the existence of an unconditional and conditional variance (variance is always positive). Both are constants</li><li>ε_t is a white noise process mean = 0, variance = 1 and covariance j=0.</li><li>With random and independent variables beyond them and with past periods, poor stationarity of an ARCH(p) has white noise and allows a conditional variance which varies through the series.</li></ul><p>The above-mentioned model explains though that the periods with high volatility will depend on the values of p past periods, so we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*nWSww94SrYZ1L-oS.png" /></figure><p>From this formula if we isolate the marginal variance we obtain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*h_i9yhUEPzfnxQUA.png" /></figure><p>This ends up with the unconditional constant variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*hBj50BoXhhPS1Cx3.png" /></figure><p>with positive values of the constants and below 1.</p><p>Now let’s take an observed value ϵ_t which represents a non-correlated series with a mean equal to 0 and also non-correlated with the past returns. This observed value is the deduction of the squared returns minus the conditional variance. If we isolate and substitute we obtain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*fiw9LuulfGcQO_qQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*2yK27hXuJMenrRcl.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*uV6VzyEMl4oAkUFJ.png" /></figure><p>In an ARCH model, if its standardized errors follow a normal distribution, its corresponding stationary distribution 0&lt;α_i&lt;1 will have fat tails, meaning that its Kurtosis is higher than 0. The ARCH has a Kurtosis above 3, means it is Leptokurtic, that means a higher concentration of data near by the distribution mean (equal to 0), leaving the intermediate areas with respect to the concentrated ones, less filled, generating so fatter tails and a sharper peak with respect to a normal standard distribution. (see Fig.:1).</p><p>To sum up the main takeaways from this model, it is important to consider that the conditional mean of the returns is equal to 0; the marginal variance is constant, rather than the conditional variance which depends on the squared returns of past periods. The process of white noise is independent and has no common distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/0*dYUEzFZqJ4ef0k3j.png" /></figure><p>To sum up the main takeaways from this model, it is important to consider that the conditional mean of the returns is equal to 0; the marginal variance is constant, rather than the conditional variance which depends on the squared returns of past periods. The process of white noise is independent and has no common distribution.</p><h3>2.3. Generalizet ARCH Model: GARCH(p,q)</h3><p><em>2.3.1. Properties of the GARCH(p,q)</em></p><p>The ARCH Model take some inefficiencies with it. In order get rid of them, the General ARCH Model was settled. The GARCH Model basically considers, besides the past returns r_(t-p), it now adds the past volatility values σ_(t-q). Finally, considering the white noise (error term), we get the following formula 〖r_t=σ〗_t ε_t.</p><p>In order to present correctly the dynamic of the variance and the implementation of a fixed delay, the ARCH model required a lot of autoregressive parameters. Bollerslev proposed the GARCH models in order to smooth these restrictions.</p><p>_2.3.2. Model Estimation GARCH (p,q)</p><p>Bollerslev stated that the conditional variance σ_(t/(t-q))² depends on its own previous values and the previous values of the returns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*2z-VXEHownPAviRP.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*p7LuO0FzZSajSuOe.png" /></figure><p>Notice that we basically add another constant β_j≥0 the other constants α_0&gt;0 and α_i≥0 from the ARCH(p) model. Being β_q σ_(t-q)² the previous volatility value, so that the GARCH model considers that the returns distribution at one point will have similar volatility as in r_(t+q). In case of the GARCH (1, 1) which is supposed to be the order for which most of the time series can be fitted, basically considers the return at r_(t-1) and the volatility σ_(t-1).</p><p>GARCH (1, 1):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*7VpZLgth8K0HkXTp.png" /></figure><p>This model GARCH (p, q), is basically the seen ARCH(q) when p=0. Bollerslev establishes several conditions of stationarity in this model, proving r_t to be slightly stationary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*8p4wEfmhDqhM6RyX.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*tGZM8Ye2rUN34xPM.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*EhvNO6qKoagCo8QF.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*Pfs6UK3__Naoh1ca.png" /></figure><p>From the standard GARCH (p, q) other models are obtained. This work will use the alternative model GJRGARCH. The Glosten-Jagannathan-Runkle GARCH model assumes a specific parametric form for this conditional heteroscedasticity, which basically means that it will get the leverage effect of the stylized facts of financial returns (for example: the returns are not normally distributed), which makes the added features more relevant (a correct estimation of the order will be sufficient to beat the standard GARCH) and on the other hand the GJRGARCH model will allow more precision than the standard one. Additionally, the updated model will not work worse than the normal one (Vanilla GARCH), so it is more efficient to make use of it.</p><h3>2.4. Implementation of the ARCH and GARCH Models</h3><p>While trying to estimate a correct ARCH and GARCH model it is required to eliminate most of the atypical values that are in the series, since it will allow us to see if the conditional variance models are required or not. In theory the maximum likelihood methodology is used in order to estimate the right models, which are formed by the product of conditional densities, under several assumptions. In our case we will use the statistical software R, which contains several packages which enables us to get in deep with the time series analysis and realize a consistent and reliable conditional heteroskedasticity model estimation. To do this analysis there was required to run several libraries which entail the necessary packages. As a principal data to work on, the ETF from the S&amp;P500 financial time series got chosen. This ETF replicates exactly the global reference index. The financial data is downloaded from yahoo-finance data source (see Fig.: 2), and the time window goes from February 2018 until now, so that we are able to see how the COVID-19 downturn affects our model. Said this, it is also interesting to see the Autocorrelation and the Partial Autocorrelation from this unedited time series, there we observe that the series is non-stationary in its regular part. Both present the same structure as with nearly any other graph, which show us that we can start working on it, and get the ARIMA-ARCH modulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/0*rs6xAqcze4J9JJFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*jwaBYprpkVHwZnwD.png" /></figure><p>Our modelling starts on the financial returns of the studied index. In order to get a correct model, we are required to take the logarithmic returns of this financial time series, so that we get rid of the inconveniencies from the ACF and PACF of the S&amp;P500 index. This process is called differentiation. Observing the return graph (see Fig.:4) we realize that there is a white noise (ε_t) along the series, and we can slightly observe that periods with low changes in its returns are followed by low changes and so with the large changes (volatility). If we look at the histogram, we can clearly observe how the data is very close to the mean, compared to a normal distribution, leading so to fatter tails and kurtosis. Even if we take a more precise view on the histogram we can see that the distribution in slightly skewed to the right, it is important to consider this view for future calculations (makes sense since it will have more positive than negative returns).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*6529IM0YYwg8tTAz.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*6KPrK5Z4Dnf_LwwZ.png" /></figure><p>Now that we have the required data to work on we have to estimate an ARIMA model in order to eliminate any linear dependence in the series. As we have seen in the S&amp;P500 Returns graph (Fig.: 4) there is no trend, which means that the returns are stationary (To reaffirm these statements a Dickey-Fuller test could be done for a 1% significance). As we have no trend the d in the ARIMA model will be 0. Having a look on the Autocorrelations we can observe that several lags hit the significance line, so that we have some possible order combinations. A special attention should be giving to the first five lags (see Fig.:6): since the first one is equal in both graphs, and the following ones which are either significant at both graphs or at one of them. Depending on a certain significance level, there will be some lags which are far more outstanding than the average ones, for example the seventh lag, which could be a random result and as being so far away from the relevant lags it will have no effect on our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/0*R7iNYtey47Ln3yax.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/0*v8g5XwTQSbUno6O2.png" /></figure><p>But to do it in a quicker and more organized way we will use the auto.arima function which will give us the optimal order for the lowest AIC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*ORtmZ0EK24S59Bc1.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*FJ3H9pOWUpJoPZS1.png" /></figure><p>Having tried the optimal order, which came up to be (4, 0, 1), and seen that the variables where not significant I went on with the following orders ran by the function (along the AIC order from smaller to bigger). The order which lastly showed to be a good choice was the ARIMA (3, 0, 0) (Table:2). Through the coefficient test, we can observe that the variables are all significant for a 1% significant level. As we noticed significance in the second lag (see Fig.:6) it was convenient to consider the orders ARIMA (2, 0, 0) and (1, 0, 0) to evaluate which would be the best choice. According to the AIC formula AIC=2k-2ln⁡(L) where 2k is the number of parameters and 2ln(L) the maximum likelihood of the model, we are required to choose the lowest possible value which means lower number of parameters. Additionally, a negative AIC indicates less information loss than a positive AIC and therefore a better model (Table: 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/0*DsqBf_AF687DcKUy.png" /></figure><p>Now let’s study the residuals of these series, since we have to get sure that there is white noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*4w1f-g_5dYY0t4-d.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/0*h2wmsa7ffwvRj_MN.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*2cgLqbv4o8yLBDJ7.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/0*FWbiuS8QlpXmM2m5.png" /></figure><p>We see that the residuals (Fig.: 7) show us that there is conditional heteroscedasticity. Coming to the ACF and PACF (see Fig.:8), we observe that the lags are not significant (except the random outstanding ones), which means that our (3, 0, 0) seems to be a good adjustment for our time series. Done this, we are required to make some tests in order to see if there is correlation between the squared errors and we have an ARCH effect on the studied financial series. The a Ljung-Box Test and the Arch Test, give us this information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/0*XrjcR6fBCj2fd_pS.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/0*5DcRhdJ337cLFk9X.png" /></figure><p>We see that both p-values are below a 1% significant level, that means that we can reject the Null-Hypothesis of No ARCH effect, and we can continue with our analysis. Having seen this, we are required to build a GARCH model, since the ARIMA by itself is not enough. The Multiplicative Lagrange hypothesis would reaffirm the errors have a significant GARCH effect, but since we have had several affirmations we can go straightforward to the GARCH model estimation and see if there are significant variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/0*Abms-AZFQd5Ax44E.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*LFr693QzrNCVHRnS.png" /></figure><p>Having a look on the Correlograms (Fig.:10), the lags on the ACF are nearly all significant, and that they are decreasing slightly. On the PACF the lags do also decrease one by one. The takeaway of the observation is that the first lags is very significant on ACF and PACF, which maybe leads us to a GARCH (0, 1)/(1, 1) or similar model. To get a good order, we have to do the coefficient test, to test the parameters significance at a 1% level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/0*7phvJW8S0A9z_swB.png" /></figure><p>In our favour the GARCH(1, 1) seemed to have its components very significant. Having tried other orders, it turned out that the aforementioned one would allow us to code in most efficient way, shorten the code lines. Now we are going to plot the model on the time series to see if the model. It is seen that the conditional variance model “gjrGARCH“ works better for most of the series than the “vanilla” GARCH, that is why it is also used in this one (Explained at the end of the GARCH Modelling part). Then as we had seen in the histogram (see Fig.:6) the returns follow a Skewed T-Student Distribution that is why the distribution model is equal to “sstd”. Finally, the ARMA or mean.model (3,0,0).</p><p>Having fixed the model we realize a back test which is going to show us the performance of the model during the time 2018/02/01 to 2020/05/16, for a 1% VaR limit. In this case I used a rolling 120-day window that refits every day. To note that if the window is too short, it could possible fail to converge instances which would disallow converting predictions to a data frame.</p><p>The analysis is going to return the Expected Exceedance and the Actual Exceedance (the amount of time the blue lines hits the orange one), the optimum would be that Actual&lt;Expected. But the important analysis is done with the Unconditional Coverage of Kupiec and Christoffersen, which are going to show us if we have to reject the Null Hypothesis of Correct Exceedances. We will see that we do not have to reject the Hypothesis, meaning that those are Correct exceedances and that the model would work good for the stated period. ARIMA(3, 0, 0)-GARCH(1, 1) turned out to be the correct order to model this time series in this time frame, so the formula acquired would be the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*kWmImbr8EXucJBAB.png" /></figure><p>As I said previously, having low amount of data can lead to the inaccurate results, by the time we analyse a longer time series (see later in this work) we will work with more reliable data results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/0*_jkKymScsPXMYS8V.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/0*FSpe_no2rnk7qy_h.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/0*6yHo_ufoCJXMeuVD.png" /></figure><p>Having seen that this model has done a good performance during the past, we can now plot an accurate forecast, for the next 10 periods. A wider forecast range increases the probability to fail with the predictions. We observe how the Yellow area, which are the sigma bands (σ), follow a similar pattern to the previous periods has a quite similar length to the previous period deviations. According to our model the extreme time frame from March and April, where we had extreme volatility values, will not occur in the next n=10 periods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/0*5eKp-ms-_xPXyeIn.png" /></figure><h3>2.5 Trading Startegy</h3><p>Having reached this part of the work, means that even you are a volatility enthusiast which has been caught by the idea (thank Mr. Engel and Mr. Bollerslev), to maybe arrive to a point where we can certainly predict the financial market or on the other hand you maybe are a professional or individual investor which skipped the more theoretical part to go straight to the practical section. Even if you are one or the other, asking yourself about how to fix this model in a real-life situation, where it is possible to get profitable results, is a very justified question. To get rid of this curiosity, we are about to get deep into the second main part of this study. The practical application consists in a trading strategy, which makes use of the historical volatility bands (see yellow bands from Fig.:12) and compares them with the Chicago Board Options Exchange Market Volatility Index (VIX). If there are certain volatility conditions fulfilled, we are going to enter actively in a financial derivate or another. It is important to mention at this point, that if you are a straight supporter of unpredictability, and completely risk averse, it advisable that you skip this part and go directly to the conclusions, without resentment. The reason is that it completely goes against that point of view, and focuses mostly on the mitigation of the impact which we thank the rare events in Extremistan. Additionally, it is advisable to be open-minded, in a sense that the strategy should always be followed by a rational mind, which is able to question it and considers the negative aspects. If the above-mentioned points are covered, we are ready to plunge into the complex and fascinating world of volatility trading.</p><p><em>2.5.1. Introduction</em></p><p>In recent years there have been launched several Exchange Trade Funds (ETF’s) or Exchange Traded Notes (ETN’s) which allowed volatility trading strategies to be accessible to the normal investor or a fund manager without the necessity to access to the future market. I will present a volatility trading strategy, that is going to allow the individual investor either to include this in its trading system or even for the value investor to hedge its own portfolio. To understand better the volatility market there are three important facts: stock market volatility is predictable (unlike returns), changes in volatility are negatively correlated with market prices and investors are prepared to pay a premium (Volatility Risk Premium) to take volatility risk from them. In order to understand the trading strategy, it is important to understand how the volatility is measured. To do so, I introduce the VIX (Chicago Board Options Exchange Market Volatility Index). The VIX is a non-direct tradeable index which measures the market expectations of the average volatility over the next 30 days. These changes in the VIX are negatively correlated with the changes in the S&amp;P500. It tends to be mean reverting which means that extreme values are followed by less extreme values (similar to the GARCH model). Having now a base point, from where to build up the strategy I will go over this part through four parts: -The Volatility Risk Premium. Which basically explains, how do I get the profits? -The Trading Variables. This part explains how the strategy works, in a theoretical way. -The Risks. As the market is subjective, there will be always risks to be considered. We see the impact and exposures on different risks. -Trading Strategy with the studied GARCH Model. With a main focus on the actual virus crisis. -Long-Term Strategy Performance. We are going to model a long-term GJRGARCH order to improve the accuracy of the trading strategy results. -Finally, the Takeaways. Which is a review of the main important points of the whole analysis.</p><p><em>2.5.2. Volatility Risk Premium</em></p><p>The Volatility Risk Premium (VRP) is one of the risks that an investor can take in order to earn higher returns than the basic risk-free rate. VRP is a premium that an investor pays in some assets (in our case the S&amp;P500) to reduce the exposure to the volatility of the future return on an asset. The VRP can be seen in the Stock index options by those been overpriced (sellers of the options receive the VRP for these higher prices). The VIX is a measure of the volatility in these options. But it is a biased estimator, therefore the VIX is higher than it should be, referencing with the future S&amp;P500 volatility (roughly said the overprediction of the VIX compared to the S&amp;P500 volatility futures is the Volatility Risk Premium).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*jIIIXlVFozdzJVWP.png" /></figure><p>Note that there are periods where the S&amp;P500 future volatility exceeds the VIX, returning a negative volatility risk premium. In that case we will have to reverse positions to earn the premium. An important point in the VIX future market is that if participants predict the VIX to rise, means the VIX is Contango, meanwhile if they predict a fall is known as Backwardation (black swans tend to be backwardation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/199/0*nF7MkixYZQnu7xVm.png" /></figure><p>As I have stated previously the VIX is not tradeable, that is why I make use of ETN (Exchange Trade Notes) in order to implement my strategy. In order to earn profits from the VIX future market we have to consider the roll yield. The roll yield is the difference between the spot VIX and the future price. But we actually do not earn the profit from this differentiation rather than considering that both values will converge while reaching the maturity date. It is important to clearly differentiate these two topics, otherwise this misperception could generate additional risks for the investor. If by the end of the of the maturity the future price increases by 5$ for example the long trader will have won, this difference of 5$ is the Volatility Risk Premium (note that the VIX sourly have increased by some amount, the difference between the new VIX spot and the new future value is the new roll yield). The roll yield tends to be positive in contango (VIX spot &lt; Future Price) and negative in backwardation (VIX spot &gt; Future Price). For my trading strategy I will the future notes ZIV (Inverse VIX future) and VXZ (VIX future, is highly correlated with the Volatility Index) Medium-Term futures (4, 5, 6, 7 months), both futures are negatively correlated meaning a percentage increase in ZIV leads to a decrease in VXZ (see correlation matrix Table:5). Those generally get less profits than the Short-Term Futures, but I had no. sufficient data available in order to do the analysis with the shorter ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/0*kW72z7KnMNTVhtve.png" /></figure><p>_2.5.3. Trading Variables</p><p>My trading strategy mainly focuses on the estimation of the Variance Risk Premium. As the VRP is a measure of excess pricing in the market. I compared two really suitable strategies:</p><ul><li>The first one is through the GARCH model I estimate, which has been shown to work good in the past. I deduct the VIX from the GARCH volatility predictions in order to estimate if the VRP is positive or negative expected. That means that if the difference is negative I go long in ZIV (contango) and if it is positive I go long XVZ (backwarding). As the backwarding situation mostly comes out in a black swan environment (example: COVID-19 stock crash), it will help the investor to reduce the drawdown, hedging so the portfolio.</li><li>The second one compares the historical volatility of n=21 labour days minus the Closed VIX. If the difference is positive we go long in the future exchange note VXZ (backwarding situation where roll yield is negative), and if the difference is negative we go long in ZIV (contango situation where the roll yield is positive). There are other parameters that could be settled down in order to improve this strategy, but that will depend on the preferences of the investor</li></ul><p><em>2.5.4. Risks</em></p><p>As any other investment strategy this one does also cover risks. First, we need to consider the fact that with higher volatility in daily returns the compounded return will decrease. Second it is required that this strategy is adjusted to time synchronization so that short term movements from contango to backwardation do not harm the final return. As we have chosen to go for a kind of strategy which hedges us in times of extreme volatility, we are covered from a massive drawdown or a decrease in returns. When it comes to other strategies like buy and hold, a black swan situation would harm a lot the final outcome, that is why they are not such a good choice. But, our strategy has the negative aspects of failing in entering at the right moment on a long position even on one ETN or the other, resulting so in profit losses and in worst cases, losses. That is the risk of making a higher number of trades. Then we have the data risks. It consists in the possibility that the data used for the analysis, mostly the long-term analysis, is not correct and lead to bad results. Even if we had modelled a significant order for our GARCH, there might be better ones, which return into a better performance in the studied time frame. We should also include the commission risks, since in my analysis I do not include the respective brokers commission, as it is very different from one to another. Some brokers require a higher rate for ETN, and maybe require to prove yourself to be aware of the risks of these financial instruments. Another consideration is the liquidity risks, since these ETN are not so highly traded or even known, there might trade at a low volume. This could lead to some sales and buy gaps which could cost the investor important profits.</p><p><em>2.5.5. Trading Strategy with the studied GARCH Model</em></p><p>Let’s start with the analysis and the code explanation. A Black Swan is a topic defined by the professional Nassim Taleb, as an unpredictable scenario which even has really good outcomes, or extremely bad ones. In economic terms the best Black Swan scenario could be exemplified by an economic crisis which led to high stock index drawdowns. Ironically, we are now facing the consequences of one of those super rare events, a stock crash because of a global pandemic, caused by the COVID-19. Considered to be the worst downturn from the past century (because of the short time frame), it is also one of the highest measured volatility, with daily closes about -12% and temporary stock exchange closes as virus panic led to huge capital flight. So, it would be interesting to test the conditional volatility model on this data, but, since the strategy works mainly with low amount of data, in order to focus on the effect of a Black Swan environment, the results might be not 100% accurate. For that reason, later on this work, I study the long-term performance of the strategy, considering a new GARCH order, and other adjustment that where required in order to get the most reliable results possible. In order to start with the strategy, we will have to convert the GARCH predictions into a data frame. Then we get the closing prices of the VIX index, to make the differentiation with the GARCH predictions or the historical SPY volatility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/0*zy3huU2O9V1HPGGd.png" /></figure><p>Remember the sigma bands of the forecast (Fig.:13, head method), we will make use of these values in order to estimate the volatility estimated by the studied heteroskedasticity model. These values will be annualized and multiplied by 100 in order to be at the same scale as the VIX values.!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/0*vLH0t5M3ZNMqfv2C.png" /></figure><p>Now that we have the correct GARCH predictions, and the VIX values, we are now required to download the financial ETN, which are the ones who are going to be traded. The required notes are, as seen, the mid-term future ETN of ZIV and VXZ (downloaded from yahoo-finance). Done this, we get the closing prices of the abovementioned notes, so that we can see how the strategy had worked historically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/0*2P0Zn1T-hMTM2PnF.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/0*v7G-pCQyyPvFk-Jt.png" /></figure><p>After that we make the differentiation and plot the rule that if the difference is negative we go long on the ZIV ETN as it shows us that our volatility sigma bands are below the VIX value, whereas if the equation returns a positive value it is the signal to go long on the VXZ, as our predicted volatility acquires a higher value than the one of the VIX. Both scenarios can occur on a backwardation or contango market. The results will be returned in the variable garchOut. What we have done here is basically see how accurate or profitable is to use the estimated GARCH model, while buying the correct financial asset if the closing VIX is above or below the GARCH predictions. Said this we have to do the same, subtracting the VIX from the SPY historical volatility. The historical volatility covers the last 21 days which equals a labour month, which is the one I used for the long-term analysis as well. But even if opted to choose n=10 days historical volatility, the results would differ, in a sense of higher fluctuations (we will get deep into this with the result interpretation). With the following functions we get the average of both strategies and we append the three statistics. Then with the stratStrats function we get the comparison of these three, in which we will see several outcome results which will help us in our decision making if these are good strategies or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/0*tmUcDHWADugeESaH.png" /></figure><p>Performance Table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*Q8P2P0wqvcZsdbCo.png" /></figure><p>Downside Risk Table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/0*ayFN3anHN6qvMapi.png" /></figure><p>In this table we can see the comparison of the performance of the estimated GJRGARCH Model and the same strategy using the historical volatility of the past 21 days from the S&amp;P500. When analysing the results, we can observe that the GJRGARCH strategy performed highly better than the historical volume one in terms of annualized returns, this can be easily explained because we considered a historical volatility of 21 days meaning that since the markets felt in such a fast way, the strategy failed to enter into the VXZ ETN, remaining so in the ZIV and consequently suffering the huge drawdown (we see that the blue line is very similar to the ZIV graph in Fig.:17). If we had considered a n=10, the graph would be as follows and so a better performance of the histVol investment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/0*o0dp8aY8jJaUGAaq.png" /></figure><p>The negative consequence of getting lower volatility days is that it is much more exposed to radical price changes. Notice also the extreme bad performance of a Buy Hold Strategy from the ZIV ETN. Going back to the table (see Table:7) we observe that despite our fitted model did a worst performance on most of the graph it was able to catch up the huge downturn so that we could mitigate the losses. By the end of 2018, where high volatility days where the day to day at the stock market because of a high uncertainty caused by the trade war between the US and China, we observe a slight downturn from our studied model, basically because it failed to enter long in the VXZ ETN as it was the start point of our data collection. Finally, through 2019 the strategy had a really bad performance, that was because we were long on the ZIV ETN, and as we see on its graph we can observe a lateral or neutral movement. Since it represents a mid-term volatility future contract it was not expected to have high changes in the next months. The Sharp Ratio turns out to be lower negative for the GJRGARCH, easily understandable because of the negative returns (which were higher than the risk-free ones). This can be explained because of low time window of two years we took in order to focus on the COVID-19 stock crash. In this period, a full exposure on this strategy would not be justified. In the case of being active traders we could have generated positive returns while entering at the right moments. Despite a considerable drawdown, about 39,66%, by the time of February 2020, which is the “target” time frame where we want to focus, to see if the strategy could hedge the downturn we observe, there is no drawdown. Coming to the Calmar Ratio, which is (like the Sharpe Ratio), a measure of risk-return, has a negative outcome too (-23%), this can also be justified with low annual returns of -9% in this period. Lastly, The Ulcer Performance Index, is an indicator which shows how long does the investment recover from its drawdowns. Normally a high drawdown leads to a slow recover to the pre-drawdown valuation. In this case the Index shows us a -0,38. Notice that through the alternative strategy (histVol) we would need more than 3 times to recover from the market drop. As the data covered for these results does not cover a high range of values, many indicators are not really accurate. So, in order to show the real performance of this volatility trading strategy a higher time frame is required. In the next part of this work we are going to see a summed GARCH modelling, which will end up with the same strategy as explained above.</p><p>_2.5.6. Takeaways of the Startegy</p><p>Comparing this strategy with others like buying and holding future notes, it turned to be a much more profitable strategy. Even if it might be a more aggressive strategy, that covers much more trades along time, if we would have had this strategy from 2004–2019 we would have got an annualized return of 16,85% with the GARCH model and 17,95% through the historical volatility method (consider that this period includes the financial crisis of 2007), compared to lower buy hold return. Additionally, the essence of using these methods is to reduce the market volatility exposure not adding another sufferer. The GARCH model allowed us to have a much lower drawdown than any other strategy, which by the time we decide to hedge our portfolio might be the better option. The profitability respect to the risk, does a really good performance, as well; confirming us the hedger-effect this strategy takes with it. So, considering that it is a high-risk investment way, and that we are not certain about the reliability of the used long-term data (which gives us a better impression of the overall performance), my recommendation is for a given portfolio dedicate 5–15% of the total invested value to a volatility risk premium hedging strategy, depending on the type of portfolio and your risk tolerance. The optimum would be to use the quantitative finance tools, to make an asset allocation, which focused on the correlation between its assets, and combines (according to past returns): shares, government debt, ETF and cash. Then through a rational analyst examination estimate the exposure on the volatility hedger. In the case, you are a swing trader*, your risk tolerance will be in accordance to the stop-loss fit. Finally, I recommend to invest always with common sense, being aware of the risks that these markets take with them. Since the strategy might have worked in the past I does not ensure positive returns in future, since there can be the possibility that the data used for the long-term testing might not be 100% certain. Surely there are far better trading strategies than this one, but in the end, everyone is looking for a way to maximize its profitability and so this way should not be walked on.</p><h3>3. LETS LOOK HOW THIS WOULD BE IN R</h3><p>First lets call the whole cluster of libraries needed for our model:</p><pre>library(quantmod) <br>library(dygraphs) <br>library(tseries)<br>library(zoo)<br>library(forecast)<br>library(FinTS)<br>library(rugarch)<br>library(stats)<br>require(TTR) <br>require(PerformanceAnalytics) <br>require(downloader)<br>library(FinTS)<br>library(lmtest)<br>par(mfrow=c(1,1))</pre><p>Done this, we get the S&amp;P500 Logarithmic returns and see how they perform on ACF/PACF and how they behave on the bell curve, seeking randomness and seek if there is stationarity in the time series:</p><pre>ticker &lt;- c(&quot;SPY&quot;)<br>sector &lt;- c(&quot;SP500&quot;)<br>symbols &lt;- getSymbols(&#39;SPY&#39;,from=&#39;2018-02-01&#39;, to=&#39;2020-06-01&#39;)<br>SP500_prices &lt;- do.call(merge, lapply(symbols, function(x) Cl(get(x)))) <br>plot(SP500_prices)<br>acf=acf(SP500_prices$SPY.Close,main=&#39;ACF SPY&#39;,lag.max=20) <br>pacf=pacf(SP500_prices$SPY.Close,main=&#39;PACF SPY&#39;,lag.max=20)<br>spyRetsClose &lt;- na.omit(Return.calculate(Cl(SPY)) ,method=c(&#39;log&#39;))<br>plot(spyRetsClose, main = &quot;Compound Returns&quot;, xlab = &quot;Date&quot;, ylab = &quot;Return in percent&quot;) <br>acf=acf(spyRetsClose,main=&#39;ACF SPY Returns&#39;,lag.max=20) <br>pacf=pacf(spyRetsClose,main=&#39;PACF SPY Returns&#39;,lag.max=20)<br>h &lt;- hist(spyRetsClose, breaks=100) xfit&lt;-seq(min(spyRetsClose), max(spyRetsClose), length=90)<br>yfit&lt;-dnorm(xfit, mean=mean(spyRetsClose, sd=sd(spyRetsClose))) <br>yfit&lt;-yfit*diff(h$mids[1:2]*length(spyRetsClose))<br>lines(xfit, yfit)</pre><p>Now we are able to implement our General Autoregressive Conditional Heteroscedasticity Model on the series (notice how several ordeers where tested in order to get the best possible outcome):</p><pre>fit1 &lt;- auto.arima(spyRetsClose, trace=TRUE, test=&quot;kpss&quot;, ic=&quot;bic&quot;) <br>x&lt;-arima(spyRetsClose, order = c(3, 0, 0),include.mean = FALSE) <br>coeftest(x)<br>arima100 &lt;- arima(spyRetsClose,c(1,0,0),include.mean = FALSE) <br>arima200 &lt;- arima(spyRetsClose,c(2,0,0),include.mean = FALSE) <br>arima300 &lt;- arima(spyRetsClose,c(3,0,0),include.mean = FALSE) <br>MeanModel &lt;-c(&#39;arima100&#39;,&#39;arima200&#39;,&#39;arima300&#39;)<br>AICValue &lt;-c(arima100$aic,arima200$aic,arima300$aic) <br>Table &lt;-data.frame(MeanModel,AICValue) <br>subset(Table,AICValue ==min(Table$AICValue)) <br>head(ModeloMedia_TablaComparacion)</pre><pre>res.ar=x$residuals<br>plot(res.ar, main = &quot;Residuals&quot;, xlab = &quot;Date&quot;, ylab = &quot;Return in percent&quot;) <br>acf(res.ar,main=&#39;ACF Residuals&#39;,lag.max=20)<br>pacf(res.ar,main=&#39;PACF Residuals&#39;,lag.max=20)<br>res.ar2=res.ar^2<br>plot(res.ar2)<br>acf.res2=acf(res.ar2,main=&#39;ACF Squared Residuals&#39;,lag.max=20) <br>pacf.res2=pacf(res.ar2,main=&#39;PACF Squared Residuals&#39;,lag.max=20)<br>Box.test(x$residuals^2,lag=12, type=&quot;Ljung-Box&quot;) <br>ArchTest(residuals(x), lag=5)<br>garch11 &lt;- garch(x$residuals,order=c(1,1),trace=TRUE) <br>coeftest(garch11)<br>gjrGARCH_spec &lt;- ugarchspec(mean.model = list(armaOrder = c(3,0)),variance.model = list(model = &quot;gjrGARCH&quot;,variance.targeting = TRUE),distribution.model = &quot;sstd&quot;)<br>gjrGARCH_fit &lt;- ugarchfit(spec = gjrGARCH_spec, data = spyRetsClose) <br>ctrl = list(tol = 1e-7, delta = 1e-9)<br>gjrGARCH_roll &lt;- ugarchroll(gjrGARCH_spec, spyRetsClose, n.start = 120, refit.every = 1, refit.window = &quot;moving&quot;, solver = &quot;hybrid&quot;, calculate.VaR = TRUE, VaR.alpha = 0.01, keep.coef = TRUE, solver.control = ctrl, fit.control = list(scale = 1))<br>report(gjrGARCH_roll, type = &quot;VaR&quot;, VaR.alpha = 0.01, conf.level = 0.99)<br>plot(gjrGARCH_fit)<br>gjrGARCH_fcst &lt;- ugarchforecast(gjrGARCH_fit, n.ahead = 10)<br>plot(gjrGARCH_fcst)<br>garchroll = as.data.frame(gjrGARCH_roll)</pre><p>The next step is to get the historical values from ZIP and VIX volatility indexes and see how our predicted volatility model performs in predicting high periods of fluctuations. Moreover, the code is going to display a comparison with the benchmark (in this case the S&amp;P500 index), and see how our styrategy performed compared to that index:</p><pre>getSymbols(&#39;^VIX&#39;,from=&#39;2018-02-01&#39;)<br>plot(VIX$VIX.Close)<br>garchPreds = xts(garchroll$Sigma * sqrt(252)*100, order.by=as.Date(rownames(garchroll))) <br>diff = garchPreds - Cl(VIX)<br>ziv=getSymbols(&#39;ZIV&#39;,from=&#39;2018-02-01&#39;)<br>etf_ziv &lt;- do.call(merge, lapply(ziv, function(x) Cl(get(x))))<br>plot(etf_ziv)<br>vxz=getSymbols(&#39;VXZ&#39;,from=&#39;2018-02-01&#39;)<br>etf_vxz &lt;- do.call(merge, lapply(vxz, function(x) Cl(get(x))))<br>plot(etf_vxz)<br>zivRets = na.omit(Return.calculate(Cl(ZIV)) ,method=c(&#39;log&#39;))<br>vxzRets = na.omit(Return.calculate(Cl(VXZ)) ,method=c(&#39;log&#39;))<br>zivSig = diff &lt; 0<br>vxzSig = diff &gt; 0<br>garchOut = lag(zivSig, 2) * zivRets + lag(vxzSig, 2) * vxzRets<br>histSpy = runSD(spyRetsClose, n = 21, sample = FALSE) * sqrt(252) * 100<br>spyDiff = histSpy - Cl(VIX)<br>zivSig = spyDiff &lt; 0<br>vxzSig = spyDiff &gt; 0<br>spyOut = lag(zivSig, 2) * zivRets + lag(vxzSig, 2) * vxzRets<br>avg = (garchOut + spyOut)/2<br>compare = na.omit(cbind(garchOut, spyOut, avg))<br>colnames(compare) = c(&quot;gjrGARCH&quot;, &quot;histVol&quot;, &quot;avg&quot;)<br>stratStats &lt;- function(rets) {<br>  stats &lt;- rbind(table.AnnualizedReturns(rets), maxDrawdown(rets)) <br>  stats[5,] &lt;- stats[1,]/stats[4,] <br>  stats[6,] &lt;- stats[1,]/UlcerIndex(rets) <br>  rownames(stats)[4] &lt;- &quot;Worst Drawdown&quot;<br>  rownames(stats)[5] &lt;- &quot;Calmar Ratio&quot;<br>  rownames(stats)[6] &lt;- &quot;Ulcer Performance Index&quot;<br>  return(stats)<br>}<br>charts.PerformanceSummary(compare,colorset=rich6equal) <br>table.DownsideRisk(compare,Rf=.03/12) stratStats(compare)</pre><p>This is all for the code. As you see those are not that numerous lines, but you need the technical knowledge if you do not want to get unreliable results from your studied model.</p><h3>4. CONCLUSION</h3><p>This work contains a brief explanation of the conditional variance of the time series theory and how can this be turned into a trading strategy. The studied financial asset time series, showed that its variance changes its significance through time, so that analysing it through the conditional mean models would generate many biases. Unlike the aforementioned models, the conditional variance models are able, to pick up the unstable error term, which results from the conditional mean estimation, leading so to a correct adjustment of the time series. After the brief theoretical part, we went through several R functions, which enables us to make a correct ARIMA order (3, 0, 0) followed by the correct GARCH (1, 1) order. The orders were established on a real-life example, an ETF that replicates the S&amp;P500 index (world reference index which contains the 500 most valued entities in the US) by the time of 2018 until now. To reiterate that this modelling can be applied on any other financial time series, as seen in the works of Elzbieta Ferenstein and Miroslaw Gasowski in their analysis on Polish stock-returns, or in Cyprian Ondieki Omari, Peter Nyamuhanga Mwita and Anthony Gichuhi Waititu’s work on the Euro Kenyan shilling. Lastly, we went to the study a volatility trading strategy, where we tried to explain how profitable can an ETN strategy become if we apply the studied heteroscedasticity models. The strategy got implemented on a short-term basis, relying the focus on the COVID-19 effects and a log-term time frame where we wanted to improve the reliability of the results, and see if this could be a good volatility hedger in the so harmful drawdowns. All in all, it is true that predicting the future price movement basing yourself on a conditional historical variance and the past returns calculated through an “optimized” GARCH model, may be a good additional tool to your investment but it does not guarantee you a 100% prediction accuracy. The future behaviour of a time series is unknown and this model, as any other one, is completely useless in predicting a Black Swan. As an amazed investor and student, when it comes to the financial market, I am required to state that, not considering the possibility to cluster an error range, would be a misunderstanding of several probability concepts, and maybe led to lot of profit loses. From a personal perspective, having to manage a portfolio, requires me to consider a volatility hedger to reduce the recovery time from a downturn. But, the core time should be invested in a reliable value investment strategy, picking long-term valuable stocks among others, since it turned out to be more profitable and secure. If you are sceptical, simply ask, Mr. Buffet.</p><h3>5. REFERENCES</h3><p>Cooper, T., (2013), Easy Volatility Investing, Double-Digit Numerics. Iorgulescu, F., (2012), Backtesting value-at-risk: Case study on a Romanian Capital Market, Elsevier, Bucharest. Davis, C., (2015), The Skewed Generalized T Distribution Tree Package Vignette, cran.r-project. Ghalanos, A. (2020). Univariate GARCH Models, Version 1.4–2, cran.r-project. Daróczi, G., (2013), Introduction to R for Quantitative Finance, (Modified in Shishir Shakya economic modeling). Bollerslev, T. (1986), Generalized Autoregressive Conditional Heteroskedasticity, Journal of Econometrics, 31:307–327. Engle, R. F., (1982), Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation, Econometrica, 50(4): 987–1007. Taleb, N. N. (2010), The Black Swan: The Impact of the Highly Improbable, 2nd Ed, Random House, New York. Kipnis, I., (2018), GARCH and a rudimentary application to Vol Trading, QuantStrat TradeR. Sopipan, N., (2017), Trading Gold Future with ARIMA-GARCH model, Thai Journal of Mathematics (2018). Hilpisch, Y., (2018), Python for Finance Mastering Data-Driven Finance, 2nd Ed, O’Reilly. Lynch, P., (2000), One up on Wall Street, Simon and Schuster. Carl, P., and Peterson, B. G., (2020), Performance Analytics Charts and Tables Overview, cran.r-project. Fuller, W. A., (1976), Introduction to Statistical Time Series, New York: John Willey &amp; Sons Inc. Er, S., and Fidan, N., (2013), Modeling Istanbul Stock Exchange-100 Daily Stock Returns: A Nonparametric GARCH Approach, Journal of Business, Economics &amp; Finance, Volume: 2, Issue: 1. Omari, C. O., Mwita, P. N. and Waititu, A. G., (2017), Modeling USD/KES Exchange Rate Volatility Using GARCH Models, Journal of Business, Economics &amp; Finance, Volume: 8, Issue:1. Graham, B., (2003), The Intelligent Investor, Reprint, HarperBusiness; Ed: 01 (Originally Work Published 1949), United States.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83131414ee2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GETTING MORE WOMEN TO CODE: HERE’S HOW]]></title>
            <link>https://medium.com/@mahupperich/getting-more-women-to-code-heres-how-9100c7002eb?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/9100c7002eb</guid>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 17:42:04 GMT</pubDate>
            <atom:updated>2021-09-13T17:42:04.556Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*AQ_WKuD4jpphmltgwkePYg.png" /><figcaption>Computer operators with an Eniac — the world’s first programmable general-purpose computer.Credit…Corbis/Getty Images</figcaption></figure><p>It is “Blog time”! This time, covering a very important theme:</p><p>How is it possible that even living in the so “advanced” 21st century, more than half of the population represent the minority in a so important topic as coding is? Is there a clear explanation for this gap? Could it be predicted? or is there no explanation at all?.</p><p>This post tries to answer all these questions enabling you to track how history polarized the affinity to coding. Moreover, it continues to seek the accomplishment of bringing people closer to coding, especially women.</p><p>I would really appreciate your feedback and feel free to share your opinion on this.</p><p><a href="https://www.linkedin.com/feed/hashtag/?keywords=womenwhocode&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#womenwhocode</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=reshmasaujani&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#ReshmaSaujani</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=codingisforeveryone&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#codingisforeveryone</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=blogpost&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#blogpost</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=newinprogramming&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#newinprogramming</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=technology&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#technology</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=personaldevelopment&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#personaldevelopment</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=motivation&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6843213896602423297">#motivation</a></p><h3>GETTING MORE WOMEN TO CODE: HERE’S HOW</h3><p>“<em>The success of every woman should be the inspiration to another. We should raise each other up. Make sure you’re very courageous: be strong, be extremely kind, and above all be humble.</em>” — <strong>Serena Williams</strong></p><p>Nowadays, coding skills continue to rise, not only as a matter of self-improvement, but they are even growing in demand for the most exciting jobs. Although, this seems very attractive the willingness to act in response has been very unilateral.</p><p>According to Stack Overflows research for 2020, only 8% of the total number of developers engaged in the survey were women.</p><p>Indeed, the main reason for this gap has its roots in the lack of interest in the technology industry, where it seems that men have the market under their mandate.</p><p>Whether there might not be a ubiquitous justification for this indifference, many studies have identified patterns in the evolution of technological interest through the past century. I will try to show you how crucial women’s roles were at the very beginning of the tech industry, adding significant improvements to this sector. Moreover, how numerous movements try to beat this contemporary differentiation through a variety of ways. <strong>HERE WE GO…</strong></p><h3>NUMBERS SPEAK MORE THAN WORDS</h3><p>“<em>Girls are taught to avoid risks, failure, to be perfect, etc. but not to be brave</em>” — <strong>Reshma Saujani</strong></p><p>In the past, only a tiny percentage of the global population were actual members in this industry. As it took a leading role in our society, more people got involved in it, leading to men outnumbering women. Numerous researches and surveys have shown the reality of a gender gap in the programming environment. Through the past years, tons of data have been analyzed, seeking the impetus of this polarity through history. To get an insight into it, I picked up some of them that allow us to get a broad understanding of what happened there.</p><p>Firstly, let me ask you some questions. For instance, Did you know that the first people that engaged with coding were women? Or that there was a higher presence of women than men in the “tech” industry back in the ’30s.</p><p>By the time of the first computers, the male counterparts were not really interested in this new machinery. Consequently, the gender gap was not so present in terms of an equal amount of workers but in terms of wages.</p><p>However, what is very surprising and sad is that according to the researcher Allan Fisher the abovementioned labor force “equilibrium” rapidly shifted into a stronger presence of males in the industry, meaning that the gender gap increased from the end of the 20th Century until now.</p><p>Yet uncommon, there is an answer to this question. In 2002, Mr. Fisher and the social scientist Jane Margolis published the book: <a href="https://www.amazon.es/Unlocking-Clubhouse-Computing-Jane-Margolis/dp/0262133989">“<em>Unlocking the Clubhouse: Women in Computing</em>”</a>, which basically covered the findings of a survey done back in 1999. The study revealed the answers of 100 undergraduates, male and female, in Carnegie Mellon’s computer science department.</p><p>They found its female students covered barely 10% of the overall number in this degree and revealed very curious facts that could explain this gender differentiation increase.</p><p>As computers started to became the key component of the tech boom by the end of the 20th Century, they got commercialized by the leading entities and quickly turned into a new member of most private households. What is relevant is that those newbies were largely only settled in the sons’ room. Consequently, the boys had much more exposure to this new technology than girls. By the time they entered the undergraduate degree, they were twice as prepared as their peer.</p><p>Moreover, the young boys followed their fathers to gather experience, engaging with construction and electric kits, whereas girls were steered towards dolls and toy kitchens.</p><p>“<em>Computes are for boys</em>” Was the message taught in schools by the time. Undoubtedly erroneous.</p><p>Nowadays, 20 years later, the actual numbers are the following. According to a recent report from <a href="https://media.kasperskydaily.com/wp-content/uploads/sites/92/2021/02/15054851/Kaspersky-Women-in-tech-Report_UPD_Regional-Stories.pdf">Kaspersky</a>, things are apparently moving in the right direction. Whereas the number of male-majority teams continues to overcome the female ones, 56% of asked women agreed that the gender inequality has decreased in the past 3 years. The reason for this huger rate, compared to 2018, comes through an increase in the number of women in an IT role, the feel of a better career progression, and their skills were more considered (70% argued that).</p><p>We see, although there is a long way to go, things are changing fast. Additionally, there is a bunch of work to be done at the very core levels of education. Founder of Girls Who Code, Ms. Saujani, talked about this in her <a href="https://www.ted.com/talks/reshma_saujani_teach_girls_bravery_not_perfection?language=es#t-605675">TED talk</a>, which I totally recommend. She stated that it is not a matter of ability, rather than the difference in approaching a challenge. In order to succeed in coding, it is crucial to spread a common mindset across society, specially the younger generations, that being imperfect gets you closer to personal and professional progress. Every economy can profit from risk-taking women who pursue their dreams.</p><p>Luckily, there are courageous idols that have shown how everyone can make a difference, stay ahead…</p><h3>WOMEN THAT INSPIRE</h3><p>“<em>You can’t be what you can’t see</em>” — <strong>Marian Wright Edelman</strong></p><p>There is consensus that if you want to succeed in your passion, you need to have mentors or idols to look up to, especially at the beginning of the journey. In the past, successful humans got extremely influenced by antecessors that took a similar path or got known because of specific skills. Learning from them allows you to extrapolate their knowledge to your situation, closing you to your goal.</p><p>In this way, investigations have proved that bringing closer successful women to the upcoming generation positively impacts the latter ones, as it gets rid of the misbelief that IT has no space for female newcomers, however a male domain.</p><p>Just to mention few of those success stories. In the 19th Century, it was a woman who scratched down the worlds first algorithm. Her name was Ada Lovelace. The mathematician was in charge of creating applications for the WORLD’S FIRST GENERAL COMPUTER.</p><p>Another inspiration in the field of programming was the eminence, Grace Brewster Murray Hopper, also known as the mother of the modern computer programming world.</p><p>By the time of World War II, many women got engaged in coding. For instance, a famous group of women created the ENIAC, whose technology was primarily held to calculate missile trajectories or other variables that could favour the outcome of the war.</p><p>After being rejected to serve her country in World War II, <strong>Ms. Hopper began her computing career by the age of 38</strong>. Persistently seeking to help her nation in this disastrous war, she finally was allowed to contribute her to her bit to the US Navy. The Ph.D. mathematician worked on IBMs and Harvard University’s Harvard Mark I, the electromechanical computer which served as a fundament for future technologies, thanks to the innovative technique, called linkers, she used in its computation. By the time she retired in 1967, she was a computer scientist and United States Navy rear admiral. This perseverance and passion for technology helped many younger generations to follow her path in computer programming. To the point that a convention was founded, aimed to get women closer to tech, said convention is named <a href="https://ghc.anitab.org/"><strong>THE GRACE HOPPER CELEBRATION</strong></a>.</p><p>Nowadays, we continuously find more and more examples of women who decided to become active members of this community. Only to mention a few of those:</p><p>- <strong><em>Avril Haines</em></strong>, <em>U.S. Director of National Intelligence</em>. Becoming the first woman in history to lead the US Intelligent Community.</p><p>- <strong><em>Kimberly Bryant</em></strong>, <em>founder, and CEO of </em><a href="https://www.blackgirlscode.com/what-we-do.html"><em>Black Girls Code</em></a>. The lack of diversity she found in her numerous engineering positions made her decide to found the Black Girls CODE. This foundation pursues to grow the number of women of color in tech by exposing young girls to computer science and tech.</p><p>- <strong><em>Reshma Saujani</em></strong>, <em>Founder and CEO of </em><a href="https://girlswhocode.com/"><em>Girls Who Code</em></a>. Graduate of Illinois, Harvard, and Yale, she began her career as an activist. After emerging on the political scene, being the first Indian American woman to run for the U.S. Congress, she also noticed a severe gender gap in computing-related fields. Therefore, she founded the non-profit organization Girls Who Code which focuses, similar to Ms. Bryant, to bring the next generations of young women closer to the coding environment. Moreover, she became numerously awarded for her podcast: “<em>Brave, Not Perfect</em>” which continues to be the voice of young women in tech.</p><h3>THE FUTURE OF WOMEN IN CODING</h3><p>“<em>COMPUTERS ARE FOR EVERYONE</em>” is the message that has to prevail from now on.</p><p>The misbelief that coding has something to do with the biological difference between the two genders is complete nonsense. Therefore we are required to create equal opportunities to succeed. Works as seen from Allen Fisher and Jane Margolis are a good start, as they revealed relevant facts that are explanatory, but they are not enough. We have to raise awareness on this issue, and once social consciousness is acquired, the snowball will roll alone.</p><p>According to the educational professor Linda Sax, in 2012 the percentage of female members in computer science rose at rates not seen for 35 years.</p><p>In recent years we have been harvesting the fruits of the numerous efforts, that chase gap dissipation. For example, in 2017, during a Tech Crunch hackathon Competition, more than 700 coders had 24 hours to develop new products. <a href="https://techcrunch.com/2017/05/14/revive-wins-the-disrupt-ny-2017-hackathon-grand-prize/">Sowmya Patapati, Akshaya Dinesh, and Amulya Balakrishnan</a> won the competition by developing a virtual reality app that tests children for signs of A.D.H.D. <strong>BREATHTAKING!!!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*3y2XmNvg8UUvRLgbOnCfbw.png" /><figcaption><a href="https://techcrunch.com/2017/05/14/revive-wins-the-disrupt-ny-2017-hackathon-grand-prize/">https://techcrunch.com/2017/05/14/revive-wins-the-disrupt-ny-2017-hackathon-grand-prize/</a></figcaption></figure><p>More than an impressive price and a blizzard of camera shots, it clearly impacts the role of women in this community. This might get younger generations out there that may want to follow their idols, creating value for our society.</p><p>To conclude this post, I would like to underline the necessity to vanish the gender gap we have in the computing group, eliminating the anachronistic belief that new technologies are reserved for only half of the population. But more importantly, thank all those individuals and communities out there that are doing an incredible job, bringing the world of coding to the younger female generation. Working all together, men and women, will leverage our society to the next level, accelerating current projects and coming with extraordinary new inventions.</p><p>Thank you for your time, hope you enjoyed the post.</p><p>Hope to see you in the next one</p><p>Have a nice day, and do not forget to code…</p><p>Yours sincerely,</p><p>Manuel Hupperich</p><h3>REFERENCES</h3><p><a href="https://usa.kaspersky.com/blog/influential-women-in-tech/24384/">https://usa.kaspersky.com/blog/influential-women-in-tech/24384/</a></p><p><a href="https://es.wikipedia.org/wiki/Ada_Lovelace">https://es.wikipedia.org/wiki/Ada_Lovelace</a></p><p><a href="https://www.create-learn.us/blog/famous-women-in-computer-science/">https://www.create-learn.us/blog/famous-women-in-computer-science/</a></p><p><a href="https://insights.stackoverflow.com/survey/2020#developer-profile-gender-all-respondents2">https://insights.stackoverflow.com/survey/2020#developer-profile-gender-all-respondents2</a></p><p><a href="https://media.kasperskydaily.com/wp-content/uploads/sites/92/2021/02/15054851/Kaspersky-Women-in-tech-Report_UPD_Regional-Stories.pdf">https://media.kasperskydaily.com/wp-content/uploads/sites/92/2021/02/15054851/Kaspersky-Women-in-tech-Report_UPD_Regional-Stories.pdf</a></p><p><a href="https://www.nytimes.com/2019/02/13/magazine/women-coding-computer-programming.html">https://www.nytimes.com/2019/02/13/magazine/women-coding-computer-programming.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9100c7002eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 SECRETS NO ONE HAS TOLD YOU ABOUT CODING]]></title>
            <link>https://medium.com/@mahupperich/5-secrets-no-one-has-told-you-about-coding-793d4ad43a89?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/793d4ad43a89</guid>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 17:35:33 GMT</pubDate>
            <atom:updated>2021-09-13T17:35:33.671Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*84_i32mK--twIID7U16iqg.jpeg" /></figure><p>Same as in algebra, where a complex formula gets simplified to a more understandable format through a variety of techniques, “LinkedIn” or your “Android” device can also be simplified into lines of logical script that aims to accomplish a specific task.</p><p>For those beyond you who are distant to programming languages, but are known to feel comfortable in the discomfort, let me give you a brief insight into how tech works.</p><p>Is it possible to gain valuable gains not only professionally but also personally, entering into the fascinating world of coding?</p><p>And of course, for those who share my enthusiasm for coding and its takeaways, feel free to share your first experience with this hidden beauty in order to get people, from not directly related fields, into programming.</p><p>Hope you enjoy it!</p><p>Have a nice weekend.</p><p><a href="https://www.linkedin.com/feed/hashtag/?keywords=programming">#programming</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=coding">#coding</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=newbies">#newbies</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=tech">#tech</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=skillset">#skillset</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=fascinate">#fascinate</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=_0010_blogpost">#0010_blogpost</a></p><h3>5 SECRETS NO ONE HAS TOLD YOU ABOUT CODING</h3><p>Nowadays, we live in a society where people get continuously more engaged with the newest technological innovations. In other words, technology has turned into our loyal partner on our day-to-day issues.</p><p>However, there is a ubiquitous excitement about the endless opportunities technological advantages gives us most people refuse to understand what is below the peak of the iceberg. So, tell me, why is there a widespread “distancing” on that?</p><p>Some will argue: it is too humdrum, too challenging, it requires advanced statistical and mathematical background, or even it is too weird.</p><p>Fortunately, it requires far less than brave heart in order to dig deep into the fascinating world of coding. Therefore, I am going to tell you 5 secrets that bring light to this unknown subject.</p><h3>1. EVERYONE CAN LEARN TO CODE (but find your purpose)</h3><p>It is true, no computer science degree nor Einstein’s IQ are basic needs to start your first code, but find your purpose.</p><p>A purpose is basically your aim. Your ambition can take several shapes: a decision, understanding, something you seek, a design, etc. Having a clear vision of what you want to achieve contributes to a much straightforward workflow and a less mundane routine.</p><p>In almost every aspect of life, setting objectives will help you to overgo the numerous drawbacks the path takes with it. Same in coding, the willingness to reach that apex is tremendously helpful to avoid doubt, lack of motivation, or stagnation.</p><p>Moreover, one goal leads to another since programming languages are scalable. Unlike spoken languages, having managed one language well, learning another will not require much more effort as it shares the structures, which is extremely helpful when approaching different projects.</p><p>CODING IS THE TOOL TO ACHIEVE YOUR PURPOSE.</p><h3>2. UTILITY RAISES EXPONENTIALLY</h3><p>It would be needless to point out what role technology plays in our daily life. Nonetheless, it might not be out of place to mention the inflection era we live in today. To do so, I will accompany you through history and see if we find a similar pattern to this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KK3uWXlVZY1_PAIBfEQEGg.png" /></figure><p>A blog post in 2016 from <a href="https://www.prendacodeclub.com/blog/coding-2016-like-reading-1816/">Prenda</a> showed a graph (see figure above) that indicates how the literate population shifted from a smooth percentage change at the beginning of the 20th Century because of the monotonous work dynamic that came from the industrial revolution to a much more aggressive slope in the second half due to a more demanding work environment and because of minority groups starting to enter higher education.</p><p>Extrapolating this shift to our current situation, we live in an era where technology drives progress in our society and covers nearly all aspects of life. However, very few actually know how all of this works. Extracting the basic member statistics from the most known online coding communities (GitHub, Stack Overflow, Reddit, etc.), there is a rough estimation that about 10 to 30 million people are able to write code*, from which 10 to 20% are women.</p><p>*Those numbers, consider those members who code more that just a “Hello World”.</p><p>Roughly said, only a bunch of people know what drives the machine. Fortunately, we haven’t missed the boat yet, but we have to ginger up.</p><p>Opportunities raise on a daily basis. For instance, imagine you could offer an analysis that efficiently identifies whether the patient has an illness or not, thanks to a technique that studies the physiological data of the scope. Hence getting a bilateral objection on the final decision makes it more reliable and much more transparent.</p><p>Similarly, imagine you could develop a client cluster seeking to adapt your business product according to each individual, avoiding so, irrelevant spam that harms your customer relationship and cost-inefficiency. Additionally, you can enhance the interaction with your client with a beautiful interface that makes communication much more straightforward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*aBKR7mxf1DYfM6WiLh-cbQ.png" /></figure><p>Those things are currently done to improve the healthcare and the retail sector, among others. You see, there is much more behind than random words on a dark screen.</p><p>From my personal experience, Business Administration relies on tons of data used for decision-making. I found that by combing the technical knowledge in business/finance with coding, one could scale up the analysis, achieving efficiency and more reliable results. The takeaways are better customer satisfaction, investment opportunities, risks hedgers, among others. Summing up, finding the utility is positively correlated with the persuasiveness of learning programming languages.</p><h3>3. ON YOUR MARKS, SET, CODE!</h3><p>“Online learning is rapidly becoming one of the most cost-effective ways to educate the world’s rapidly expanding workforce.” — Jack Messman, former CEO at Novell, Cambridge Technology Partners, Unionacific Resources, Somerset House Corp.</p><p>There might be curious minds that see the potential and the beauty of a functioning algorithm, but they fail to start.</p><p>For those of you who have access to a variety of books, I encourage you to make a quick research on which are in line with your focus of interest and give them a glance.</p><p>For those who not, luckily, the World Wide Web allows everyone to gain knowledge of nearly everything. One can find millions of videos and courses, that aim to explain the foundations of how to use the newest programming resources for your personal or professional purposes, spending little to no money.</p><p>For a deeper understanding, the educational programs that seek to persuade the usage of code has risen in the last years. Not only big companies or universities, but also individual blogs as in <a href="https://dev.to/">DEV</a>, offer open source content in order to get you started in the coding community and helping to improve your skills.</p><p>On a national scale, things are changing too. The European Union founded in 2013, <a href="https://codeweek.eu/">CodeWeek</a>. Chasing to help more young people to master the basics of coding and computational thinking, the event has skyrocket in the past years. In 2019, the attendance was of about 4.2 million people (2.7 million in 2017) of over 80 countries, from which the average attendance age was 11 years old and 49% of the total were women or girls. Furthermore, many high schools in Germany, Denmark and the UK have started to include computer programming as a mandatory curriculum, as governments try to prepare already digital native pupils for the future labour market.</p><p>You see, it is not a matter of resources that slows our progress. There are numerous ways to start engaging with the newest programming languages, therefore it just a matter of determination.</p><h3>4. HELPS TO GET YOUR DREAM JOB</h3><p>“Whether you want to uncover the secrets of the universe, or you just want to pursue a career in the 21st century, basic computer programming is an essential skill to learn.” — Stephen Hawking, Theoretical Physicist, Cosmologist, Author</p><p>The future of work depends on several factors, demographic expansion, economic growth, etc. However, there is high certainty that digitalization is and will continue to be the one principal driver of nearly every position.</p><p>According to a study of <a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/984671/DCMS_and_Ipsos_MORI_Understanding_the_AI_Labour_Market_2020_Full_Report.pdf">Ipsos MORI</a> in 2020, 62% of the asked firms had faced problems with tech skills gaps. Those skills were: understanding of AI concepts and algorithms (55%), programming skills and languages (52%), software and systems engineering (52%), among others.</p><p>Occupations that require coding skills grow 12% faster than the market average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*_AhivZjCZDfV_Oae9NfZHA.png" /></figure><p>Since there is a low supply of reliable, skilled workers and the demand is growing as said, salaries rose at a constant rate during the recent years.</p><p>An analysis of a recruitment site in the UK revealed in 2019 that computing skills had the highest average salaries, compared to positions that require language, creative or business skills. In 2020, the mean advertised salary was 64000 euros for an AI and data science-related posting. Yet, being a specific position, it clearly shows the potential.</p><p>Notably, more and more young graduates publish their non-conformity with the required years of experience in the labour market. Nonetheless, having developed the necessary coding skills reduces the probability of not finding a job because of the lack of experience (Not that this is the case of a Data Science Career, there might be differences compared to other sectors).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*quM3wASpDCIHhz-x2F7mKw.png" /></figure><p>Leaving the statistics about the advantages of coding behind, let’s get an idea about the profound effect, focusing on the personal gains.</p><h3>5. PERSONAL DEVELOPMENT THROUGH CODING</h3><p>“I have not failed. I’ve just found 10,000 ways that won’t work.” ― Thomas A. Edison</p><p>For those of you who have never written a line of code, it is pretty similar to those summer holidays where you take a never-ending walk through the beach hoping to find that right spot. Therefore, in coding, one consistently goes through drawbacks to accomplish the stated goals.</p><p>Besides the importance of focusing on the bigger picture, negligent works pay off in the form of uncountable errors. Therefore, a tenacious mindset helps coders to reach the next level.</p><p>But determination is not the only takeaway, planning is also a skill you must develop. Consistently writing algorithms requires you to follow a similar structure; however, be open to continuous changes in this environment. Consequently, one adopts a skeptical attitude towards the written lines, as past failure forces you to question the things you do constructively.</p><p>From a personal perspective, coding pushes your imagination, spicing your work with your personal watermark.</p><p>Moreover, even though you might think it is a vastly individualistic work, nothing is further from reality. Online communities in the coding environment grow year by year. Joiners interact together in order to help others to improve their algorithm or to take the first steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*DImq5Bfc4eYAmeZw0v260g.png" /></figure><h3>WRAPPING UP</h3><p>To conclude, although coding is not the panacea to achieving your dreams, it is a valuable skill that truly eases your way to success.</p><p>Whereas it gets you to your dream job or opens you the door for a higher salary, it scales you up on a personal level.</p><p>This fast-changing environment will always force you to get out of your comfort zone, be always up-to-date, and continuously improving your skillset, particularly at the very beginning. Nonetheless, the remuneration of this investment can help you in nearly every aspect of life, and this (from my personal point of view), is somewhat not everything can teach you, ergo it is worth to give a try.</p><p>Thank you for taking your time to read my post, feel free to comment or share it.</p><p>Hope you are all well and staying healthy.</p><p>See you in the next post,</p><p>Best regards,</p><p>Manuel Hupperich</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793d4ad43a89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[START A BLOG AT 23: HERE’S WHY]]></title>
            <link>https://medium.com/@mahupperich/start-a-blog-at-23-heres-why-2c85ef2fe4d6?source=rss-31491333ba59------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c85ef2fe4d6</guid>
            <dc:creator><![CDATA[Manuel Hupperich]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 17:25:49 GMT</pubDate>
            <atom:updated>2021-09-13T17:25:49.431Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ZVw2VnrdRj_oiljplsbpbA.jpeg" /></figure><p>Think about this paradox, nearly everyone interacts with algorithms on a daily basis, but isn’t even aware of that…¿?</p><p>To get rid of that, I am happy to present you the first of numerous blog posts I will continuously publish on LinkedIn. The blog is aimed to motivate newbies to start their journey in the programming environment increasing the number of individuals able to write algorithms.<br> <br>I took the motivation to start this new project as I got accepted into <a href="https://www.linkedin.com/school/3156146/?legacySchoolId=3156146">IE School of Human Sciences and Technology (HST)</a>, hoping to hear and connect to experts from this field, and learn how to spice up the code I have written till yet.<br> <br>Feel free to share your opinion,</p><p>Enjoy it!</p><p><a href="https://www.linkedin.com/feed/hashtag/?keywords=technology&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6831893143923781632">#technology</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=motivation&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6831893143923781632">#motivation</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=programming&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6831893143923781632">#programming</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=newinblogging&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6831893143923781632">#newinblogging</a> <a href="https://www.linkedin.com/feed/hashtag/?keywords=github&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6831893143923781632">#github</a></p><h3>START A BLOG AT 23: HERE’S WHY</h3><p>Do you speak Python? Are you a native of Ruby? Can you decipher the structure of an HTML website?</p><p>If you answered yes to any of these questions, you might be familiar with the advantages that those take with them. If not, I might be the one who triggers your curiosity.</p><p>Hi, my name is Manuel Hupperich, and I code.</p><h3>WHERE WE’RE COMING — WHERE WE’RE GOING</h3><p>Starting to code was by far not a childhood dream. I was more focused on being a pro soccer player or discovering new spots on my island rather than learning how to implement Gradient Descent on my Python script. Nevertheless, curiosity and ambition were always close partners on my way.</p><p>Notably, it was not before my time at college that I got engaged with this new topic. Coursing numerous quantitative courses and engaging with the mentors of my BBAM, gave me insight into the potential of technology. That gave me the push to start my journey and see if that quirky new topic kept its promise of being an al-rounder.</p><p>After a quick research I realized how through simple visualizations and automatization algorithms one can create astonishing things. Unfortunately, these were not taught in my Bachelor’s subjects. As a consequence, I began self-education in the newest programming languages.</p><p>“Poor me”, was my first thought.</p><h3>SEEKING COMFORT IN DISCOMFORT</h3><p>Those with whom I shared this new project were all but enthusiastic. Arguing with phrases like: “Only software engineers do code…”, “It is useless…”, “I am scared of that strange words on a dark ground…”, “Coding is so mundane…”, “There is a paucity of coding information”, etc.</p><p>Convinced of my purpose and thankful for the few people who decided to support my idea, I agreed to disagree with the above-mentioned statements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*VJLcPcShAozFfJVC1fQRdA.png" /></figure><p>Firstly, I read and tried to replicate the code plotted on several books, such as the ones from <a href="https://www.oreilly.com/">O’Reilly</a>. Those gave me an overview but were too unadjusted for newbies, I would recur back to this source, once I interiorized the foundations. That is when the IoT came in.</p><p>The internet of things has an incredible variety of sources from which to learn. Having self-taught the guitar (chords, music theory, techniques, etc.) made me feel pretty confident in the e-learning environment.</p><p>I swallowed hours and hours of informative videos, read blogs, papers, studies, and wrote my first few lines of code. Persistence, prescience, and logical thinking were only few of the skills I leveraged thanks to my new project.</p><p>After one and a half years, I was able to consistently write algorithms that accomplished my stated purposes. From the thousand lines of codes, there are two I am particularly proud of and are worth to exalt. On the one hand, an exam that asks ones about the name of the accounting statement given it the relative number, returning the correct answer if you failed and adding some basic statistics for performance visualization. On the other hand, a more technical one which implied a trading strategy based on the <a href="https://github.com/Hupperich-Manuel/GARCH_Model_Startegy">General Autoregressive Conditional Heteroskedasticity Model (GARCH Model)</a>, developed for my BBAM Thesis. For the later one I needed to deeply understand that statistical time series model. However, in terms of code, rudimentary stuff was going on there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*Vp3H3tRvzi9waDgQshQ5Dw.png" /><figcaption><a href="https://github.com/Hupperich-Manuel">https://github.com/Hupperich-Manuel</a></figcaption></figure><p>Although free e-learning was extremely helpful to interiorize the fundamentals, I noticed that it was not the panacea for great coding. Therefore, I started to look up for more specific education, which had a special focus on specific tasks, by the time I discovered the bright variety of online courses.</p><p>The pandemic period, where the world was in standby modus, was the best moment to work out extensively the principles of programming. Luckily, I am not keen on procrastinating, and so dedicated multiple hours in getting those projects and courses done. I acquired about 12 different coding certificates. Those were far more straightforward than the open-source info I used previously. Briefly, they covered the very basics of data manipulation through web-scrapping to convolutional networks for image recognition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gff0S5ZQ43PL2CXPDJbP2A.png" /><figcaption><a href="https://github.com/Hupperich-Manuel">https://github.com/Hupperich-Manuel</a></figcaption></figure><p>Evidently, there is still a long way to go. The efficiency of the algorithms needs to be polished, learn new techniques, accomplish new purposes, etc. Nonetheless, even if there are much better codes out there, I have accomplished something that more than half of the world population obviates, to see the power and beauty of an algorithm, but even more important, I took the initiative.</p><p>DO YOU WANT TO KNOW WHAT IS BEHIND OF ALL THAT AND UNDERSTAND WHAT CODING IS REALLY ABOUT?</p><h3>SO, WHY ON EARTH DO YOU BLOG?</h3><p>Easy, because it is a tremendous way to create a community that shares their experiences, whether they succeed or struggled on some topic, and maybe rise curiosity of newcomers.</p><p>This specific blog aims to bring closer the whole coding stuff to those who couldn’t see the forest because of the gorgeous interface.</p><p>Same as the statements I had to deal with at the very beginning, there surely are plenty of individuals who faced the same. Those statements are deep-rooted in our minds.</p><p>I am still not an expert in this field but, I am on my way to becoming one. LinkedIn is the best way to engage with experts from this sector that share their insights and from whom one can learn a lot. Therefore, I am open to any constructive critic, that not only helps me to become a better programmer but also to write better posts.</p><p>My goal is to get rid of the clichés, increase the number of people that engage with coding (hopefully reducing the gender gap our community faces), accentuate the prodigious possibilities it offers, and helping you to take your first steps.</p><p>To conclude, I encourage you to share your point of view in the comments, your experience with coding, or your fears. I would be pleased to hear from you, and you will see that working together will ban the negativity we link to coding.</p><p>Hope you are doing all well,</p><p>Have a nice day, and see you in the next post.</p><p>Kind regards,</p><p>Manuel Hupperich</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c85ef2fe4d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>