<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roman Moser on Medium]]></title>
        <description><![CDATA[Stories by Roman Moser on Medium]]></description>
        <link>https://medium.com/@romanm111987?source=rss-2b754791b7ab------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hR26l-nHHYzpOOJRphPuew.jpeg</url>
            <title>Stories by Roman Moser on Medium</title>
            <link>https://medium.com/@romanm111987?source=rss-2b754791b7ab------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:13:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@romanm111987/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fraud detection with cost-sensitive machine learning]]></title>
            <link>https://medium.com/data-science/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9?source=rss-2b754791b7ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/24b8760d35d9</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[credit-cards]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[fraud]]></category>
            <dc:creator><![CDATA[Roman Moser]]></dc:creator>
            <pubDate>Fri, 29 Mar 2019 16:58:11 GMT</pubDate>
            <atom:updated>2019-03-29T20:30:46.174Z</atom:updated>
            <content:encoded><![CDATA[<h4>The concept of example-dependent cost-sensitive classification algorithms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gq6rTfSQCgL_h5Jz7aaM_w.jpeg" /></figure><p>In traditional two-class classification problems we aim to minimize misclassifications and measure the model performance with metrics like Accuracy, F-score or the AUC-ROC Curve. In certain problems, however, it is for the best to allow more misclassifications at the benefit of lower total costs. If costs associated with misclassifications vary among samples, we should apply an example-dependent cost-sensitive learning approach. But let’s start from the beginning…</p><p>In this article, I will be explaining the concept of example dependent cost-sensitive machine learning by training and testing a variety of models on a credit card fraud data set. Please note that I chose the models for this task for the purpose of illustrating the concept rather than achieving optimal prediction results. Snippets of the code are presented in this article and the full code is available on my <a href="https://github.com/roman807/Fraud-Detection">GitHub</a>.</p><h3>What is Cost-Sensitive learning?</h3><p>Whereas traditional classification models assume that all misclassification errors carry the same cost, cost-sensitive models consider costs that vary by type of classification and across samples.</p><p>Let’s take a look at the case of credit card transactions. Transactions that are not authorized by the true holder are considered fraudulent (usually a very small portion of all transactions). A credit card fraud detection system should automatically identify and block such fraudulent transactions and at the same time avoid blocking legitimate transactions.</p><p>What are the costs associated with each type of classification? Let’s assume the following scenario. If a fraudulent transaction is not recognized by the system, the money is lost and the card holder needs to be reimbursed for the whole transaction amount. If the system labels a transaction as fraudulent, the transaction is blocked. In that case administrative costs occur because the card holder needs to be contacted and the card needs to be replaced (if the transaction was correctly labeled fraudulent) or reactivated (if the transaction was actually legitimate). Let’s also make the simplified assumption that the administrative cost are always identical. If the system correctly labels a transaction as legitimate, the transaction is automatically approved and no costs occur. This results in the following costs associated with each prediction scenario:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D6WxX1QSweIdkd_TuP6sqw.png" /></figure><p>Note that “Positives” are transactions predicted as fraudulent and “Negatives” are transactions predicted as legitimate. “True” and “False” refer to correct and incorrect predictions, respectively.</p><p>Because the transaction cost depends on the sample, the cost of a False Negative can be negligibly low (e.g. for a transaction of $0.10), in which case the administrative costs of a positive prediction would outweigh the reimbursement costs, or very high (e.g. for a transaction of $10,000).</p><blockquote>The idea behind cost-sensitive learning is to take these example dependent costs into account and make predictions that aim to minimize the overall costs instead of minimizing misclassifications.</blockquote><h3>Cost sensitive training vs cost dependent classification</h3><p>Let’s consider two different approaches. The first one is to train a model with a loss function that minimizes the actual costs ($) instead of misclassification errors. In this case we need to provide the loss function with the costs associated with each of the four cases (False Positives, False Negatives, True Positives and True Negatives) so that the model can learn to make optimal predictions accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_FeRhIidN3SPnmKVCFJ5OQ.png" /></figure><p>The second approach is to train a regular model, but classify each sample when making predictions according to the lowest expected costs. In this case the costs on the training set are not needed. However, this approach only works for models that predict a probability which can then be used to calculate the expected costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZFpJILuUJWpy5kZqE5EwA.png" /></figure><blockquote>In the following, I will refer to models that use a cost-sensitive loss function as <strong>“Cost-sensitive models” </strong>and to models that minimize the expected costs when making predictions as <strong>“Cost classification models”</strong></blockquote><h3>Implementing and evaluating models</h3><p>For this case study, I used a credit card fraud data set (available on <a href="https://www.kaggle.com/sion1116/credit-card-fraud-detection/data">Kaggle</a>) with 284,000 samples and 30 features. The target variable indicates whether a transaction is legitimate (0) or fraudulent (1). The data is highly imbalanced with only 0.17% fraudulent transactions. I trained and evaluated the following five models.</p><ol><li><strong>Regular</strong> <strong>Logistic Regression </strong>(from scikit-learn)</li><li><strong>Regular Artificial Neural Network</strong> (built in Keras)</li><li><strong>Cost-sensitive Artificial Neural Network</strong> (Keras)</li><li><strong>Cost classification Logistic Regression</strong></li><li><strong>Cost classification Artificial Neural Network</strong></li></ol><p>In practice, artificial neural networks (“ANNs”) might not be the first choice for fraud detection. Tree based models such as Random Forests and Gradient Boosting Machines have the advantage of interpretability and often perform better. For the purpose of this illustration I used ANNs because of the relatively straightforward implementation of a cost-sensitive loss function. Also, as I will be showing, a simple ANN delivers quite strong results.</p><p>To evaluate the results, I used two different metrics. The first one is the traditional F1-score which weighs precision and recall but does not consider the example dependent cost of misclassifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*sJPoKUgIWh7TPrStD4LIQg.png" /></figure><p>To evaluate a model’s performance in terms of costs, I first calculated the sum of all costs resulting from the predictions based on whether the model predicted a False Positive, False Negative, True Positive or True Negative and the costs associated with each case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dbw16jQ1NoB7fgfR2poa_A.png" /></figure><p>I then calculated the sum of the costs that would occur if all cases were predicted negative (“cost_max”), and define the cost savings as the fraction by which the actual predictions reduce the costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*tFVdP-N4xerZrEgVY6hqxA.png" /></figure><p>To evaluate the models I used 5-fold cross-validation and split the data into five different training (80%) and test sets (20%). The results presented in the subsequent section refer to the average result on the five test sets.</p><h4>Logistic Regression</h4><p>As base model serves a regular Logistic Regression model from the scikit-learn library. The plot below visualizes the distribution between predicted probabilities and transaction amounts. Without cost-sensitive classification there is no visible association between fraud probability and transaction amount.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*Rozu7it9MaBP6cC0Wv1QhA.png" /></figure><p>The logistic regression performs reasonably well with an average test set F1-score of 0.73 and cost savings of 0.48.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*O2WFoJeVl7QLwPcRQdLFuQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*nDETV0Ajn6-4QVOtjzZzMg.png" /></figure><h4>Artificial Neural Network</h4><p>Next, I built an ANN in Keras with three fully connected layers (50, 25 and 15 neurons) and two dropout layers. I ran the model for two epochs and used a batch size of 50. Using the Sequential model API from Keras, the implementation in Python looks like this:</p><pre>from keras.models import Sequential<br>from keras.layers import Dense, Dropout</pre><pre>def ann(indput_dim, dropout=0.2):<br>    model = Sequential([<br>    Dense(units=50, input_dim=indput_dim, activation=&#39;relu&#39;),<br>    Dropout(dropout),<br>    Dense(units=25, activation=&#39;relu&#39;),<br>    Dropout(dropout),<br>    Dense(15, activation=&#39;relu&#39;),<br>    Dense(1, activation=&#39;sigmoid&#39;)])<br>    return model</pre><pre>clf = ann(indput_dim=X_train.shape[1], dropout=0.2)<br>clf.compile(optimizer=&#39;adam&#39;, loss=&#39;binary_crossentropy&#39;)<br>clf.fit(X_train, y_train, batch_size=50, epochs=2, verbose=1)<br>clf.predict(X_test, verbose=1)</pre><p>Below is the distribution of predicted fraud probabilities with the ANN. Similar to the logistic regression model there is no visible relationship between fraud probabilities and transaction amount.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*TOiXJygBIVeT69fQqtn9Qg.png" /></figure><p>The ANN outperformed the logistic regression model in terms of both, F1-score and cost savings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*gQCz9k5L8icuQ-mhqP7lqQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*L6Y0SkB5q3IGfWJ7cn8e5A.png" /></figure><h4>Cost-sensitive Artificial Neural Network</h4><p>Now it gets a bit more interesting. The cost-sensitive ANN is identical to the regular ANN with the difference of a cost-sensitive loss function. Both of the previous models used the logarithmic loss (“binary cross entropy”) as loss function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cx3E_hmPw3yV_52V52Forw.png" /></figure><p>This loss function punishes false negatives and false positives equally. Let’s now take a look at a cost-sensitive loss function. Here, all four possible outcomes (False Positives, False Negatives, True Positives and True Negatives) are being considered and each of the outcomes carries a specified cost. The cost-sensitive loss function looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UaflXwFcYTYwILoA_JRGUg.png" /></figure><p>Remember from the first section that True Positives and False Positives are being considered as equally expensive (fixed administrative cost for blocking a transaction). The cost for True negatives is $0 (no action) and the cost for False Negatives is the transaction amount (assume we have to reimburse the whole transaction). Note that of these four costs, only the cost for false negatives is example-dependent. This has the effect that with a higher transaction amount, the punishment for a not identified fraudulent transaction increases relative to the administrative cost of a positive prediction. The loss function should therefore train a model that is likelier to reject suspicious transaction when the transaction amount is higher. The transaction amounts range anywhere from $0 to $25,691 with a mean of $88 and I assumed a fixed administrative cost of $3.</p><p>In Python we define the costs for False Positives, False Negatives, True Positives and True Negatives accordingly. Since the costs for False Negatives are example-dependent they are represented in a vector of length equal to number of samples.</p><pre>cost_FP = 3<br>cost_FN = data[&#39;Amount&#39;]<br>cost_TP = 3<br>cost_TN = 0</pre><p>Implementing an example dependent loss function in Keras is tricky because Keras does not allow arguments other than y_true and y_pred to be passed to the loss function. Constant variables can be passed to the loss function by wrapping the loss function into another function. However, the costs for False Negatives are example-dependent. I therefore used the trick of adding the costs of False Negatives as digits after the comma to y_true and extracting them inside the custom loss function while rounding y_true to the original integer value. The implementation of the functions to transform y_true and the custom loss function in Keras look like this:</p><pre>import keras.backend as K</pre><pre>def create_y_input(y_train, c_FN):<br>    y_str = pd.Series(y_train).reset_index(drop=True).\<br>            apply(lambda x: str(int(x)))<br>    c_FN_str = pd.Series(c_FN).reset_index(drop=True).\<br>            apply(lambda x: &#39;0&#39;*(5-len(str(int(x)))) + str(int(x))<br>    return y_str + &#39;.&#39; + c_FN_str</pre><pre>def custom_loss(c_FP, c_TP, c_TN):<br>    def loss_function(y_input, y_pred):<br>        y_true = K.round(y_input)<br>        c_FN = (y_input - y_true) * 1e5<br>        cost = y_true * K.log(y_pred) * c_FN + <br>               y_true * K.log(1 - y_pred) * c_TP) +<br>               (1 - y_true) * K.log(1 - y_pred) * c_FP +<br>               (1 - y_true) * K.log(y_pred) * c_TN)<br>        return - K.mean(cost, axis=-1)<br>    return loss_function</pre><p>I then called the defined functions to create the y_input vector, train the cost-sensitive ANN and make predictions:</p><pre>y_input = create_y_input(y_train, cost_FN_train).apply(float)<br>clf = ann(indput_dim=X_train.shape[1], dropout=0.2)<br>clf.compile(optimizer=&#39;adam&#39;, loss=custom_loss(cost_FP, cost_TP,<br>            cost_TN))<br>clf.fit(X_train, y_input, batch_size=50, epochs=2, verbose=1)<br>clf.predict(X_test, verbose=1)</pre><p>In the distribution plot below we can see the effect of cost-sensitive learning. With an increasing transaction amount, the general distribution of predictions expands to the right (higher fraud probabilities). Note that in this case, due to the nature of the problem and definition of the loss function, “predicted fraud probability” means “should we identify the transaction as fraudulent?” rather than “is the transaction fraudulent”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*T4vbPoBuNpQXCZUfs6WgGg.png" /></figure><p>The evaluation shows the expected effect of cost-sensitive learning. The cost savings increased by 5% and the F1-score decreased by a similar margin. The consequence of cost-sensitive classification is a higher number of misclassifications at the benefit of lower total misclassification costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*vps2QWIX_ivsN8sO_NUpkg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*XOJDsInSqKglbOEZaDHxtg.png" /></figure><h4>Cost classification models</h4><p>As opposed to a cost-sensitive model that trains with a customized loss function, cost classification models calculate the expected costs based on predicted probabilities. The expected costs for a predicting a legitimate and a fraudulent transaction are calculated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhjK1yV75wywopWvxTrq9g.png" /></figure><p>The classifier then chooses whichever prediction is expected to result in lower costs.</p><p>I therefore used the probability prediction results from the regular logistic regression and ANN and reclassified the predictions based on the expected costs. The plot below visualizes the effect of the cost-dependent classification for the example of the logistic regression model. Note that the distribution of predicted probabilities did not change from the one produced by the regular logistic regression model. However, with cost dependent classification, the model tends to identify transactions with a small fraud probability as fraudulent as the transaction amount increases. On the right side of the plot we see that transactions with a very small amount are predicted as legitimate even as the fraud probabilities approach 1. This is due to the assumption that True Positives carry administrative costs of $3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*wmteJPJHaqQ9es1vjrE9tg.png" /></figure><p>Classifying the predictions based on expected costs leads to even better results in terms of cost savings (and significantly worse results in terms of F1-score). While implementing a cost-sensitive loss function for the ANN reduced the costs by 5%, the cost classification ANN was able to reduce costs by 10%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*fJDOEHZ4p62P85zNVbK21A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*PWgGNtkSeT5PWMbQmWbFkg.png" /></figure><h3>Conclusion</h3><p>This article illustrates two fundamentally different approaches for example based cost-sensitive classification on credit card fraud prediction. While cost-sensitive training models require a custom loss function, cost classification models only require the probabilities for each class and the costs associated with each outcome to classify a transaction. In my sample case cost classification models achieved slightly better cost savings at the expense of a high number of misclassifications. Additionally, a cost-classification model is easier to implement as it does not require a custom loss function for training. However, the cost classification method is only applicable for models that predict probabilities, which a logistic regression and ANN conveniently do. Tree based models, adopted more widely for fraud detection, however, generally separate predictions directly into classes, making the cost classification approach infeasible. A cost-sensitive approach for tree based models is, while conceptually similar to the one presented in this article, more complicated in implementation. If you are interested in this topic I would suggest to take a look at the paper referenced below.</p><p>Thanks for reading this article. Please feel free to comment or ask questions via the comment section below or to connect with me on <a href="https://www.linkedin.com/in/roman807/">LinkedIn</a>.</p><p>The code created for this illustration can be accessed on my <a href="https://github.com/roman807/Fraud-Detection">GitHub</a></p><p>The credit card fraud data set is available on <a href="https://www.kaggle.com/sion1116/credit-card-fraud-detection/data">Kaggle</a></p><p>If you are interested in learning more about cost-sensitive learning with tree based models I suggest <a href="https://arxiv.org/pdf/1505.04637.pdf">this paper</a> from A. C. Bahnsen, D. Aouada and B. Ottersten along with the <a href="https://github.com/albahnsen/CostSensitiveClassification">costcla GitHub repository</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24b8760d35d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9">Fraud detection with cost-sensitive machine learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting stock market crashes]]></title>
            <link>https://medium.com/data-science/predicting-stock-market-crashes-with-statistical-machine-learning-techniques-and-neural-networks-bb66bc3e3ccd?source=rss-2b754791b7ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb66bc3e3ccd</guid>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Roman Moser]]></dc:creator>
            <pubDate>Mon, 14 Jan 2019 14:44:47 GMT</pubDate>
            <atom:updated>2021-08-24T21:46:17.435Z</atom:updated>
            <content:encoded><![CDATA[<h4>An attempt with statistical machine learning techniques and neural networks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0AMNtyZhP1w-mPGms6dHg.jpeg" /></figure><p>With this blog post I am introducing the design of a machine learning algorithm that aims to forecast crashes in stock markets solely based on past price information. I start with a quick background on the problem and elaborate on my approach and findings. All the code and data are available on <a href="https://github.com/roman807/Predicting-Stock-Market-Crashes">GitHub</a>.</p><p>A stock market crash is a sharp and quick drop in total value of a market with prices typically declining more than 10% within a few days. Famous examples of major stock market crashes are the Black Monday in 1987 and the real estate bubble in 2008. A crash is usually attributable to the burst of a price bubble and is due to a massive sell-off that occurs when a majority of market participants try to sell their assets at the same time.</p><p>The occurrence of price bubbles implies that markets are not efficient. In inefficient markets prices do not always reflect fundamental asset values but are inflated or deflated based on traders’ expectations. These expectations are reinforced by traders’ subsequent actions which further inflate (or deflate) prices. This leads to positive (or negative) price bubbles which eventually burst. This phenomenon was described by George Soros as reflexivity and is the basic assumption for forecasting methods used in technical analysis.</p><p>Today, there is not much debate over the existence of bubbles in financial markets. However, understanding these inefficiencies and predicting when price bubbles will burst is a highly difficult task. Imagine you could identify a bubble that is about to build up and predict when the market will crash. You would not only be able to make a profit while prices are increasing but also to sell at the right moment to avoid losses.</p><p>Some mathematicians and physicists attempted to tackle this problem by investigating the mathematics behind price structures. One such physicist is Professor Didier Sornette who successfully predicted multiple financial crashes [1]. Sornette uses log-periodic power laws (LPPLs) to describe how price bubbles build up and burst. In essence, the LPPL fits the price movements leading up to a crash to a faster than exponentially increasing function with a log-periodic component (reflecting price volatility with increasing magnitude and frequency).</p><p>And this is where the idea for this project is coming from. If the recurring price structures found by researchers exist, should it not be possible for a machine learning algorithm to learn these patterns and predict crashes? Such an algorithm would not need to be aware of the underlying mathematical laws but would instead be trained on data with pre-identified crashes, and identify and learn these patterns on its own.</p><h4>Data and Crashes</h4><p>The first step was to collect financial data and identify crashes. I was looking for daily price information from low correlated major stock markets. Low cross-correlation is important for valid cross validation and testing of the model. The matrix below shows the cross-correlation of daily returns from 11 major stock markets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*RDXiff7swLd_rUGSCnJWEA.png" /><figcaption><strong>Correlation matrix of daily price returns for 11 major stock market indices</strong></figcaption></figure><p>To avoid having any two data sets with a cross-correlation greater than 0.5 in my collection, I proceeded with only data from the S&amp;P 500 (USA), Nikkei, (Japan), HSI (Hong Kong), SSE (Shanghai), BSESN (India), SMI (Switzerland) and BVSP (Brazil).</p><p>To identify crashes in each data set, I first calculated price drawdowns. A drawdown is a persistent decrease in price over consecutive days from the last price maximum to the next price minimum. The example below shows three such drawdowns in the S&amp;P 500 over the period from end of July to mid August 2018.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*wN8wk2QLxc8D5LHEh6294w.png" /><figcaption>Example of three drawdowns. The first one shown lasted from July 25th to July 30th 2018 and has a total loss of approximately (2846–2803)/2846 = 1.5%</figcaption></figure><p>I considered two different methodologies to identify crashes. The first one follows a suggestion by Emilie Jacobsson [2] who defines crashes in each market as drawdowns in the 99.5% quantile . With this methodology I found drawdown thresholds that classify a crash ranging from around 10% for less volatile markets like the S&amp;P 500 to more than 20% for volatile markets such as the Brazilian one. The second methodology follows the suggestion from Johansen and Sornette [3] who identify crashes as outliers, that is drawdowns that lie far from the fitted Weibul distribution when the logarithm of the rank of drawdowns in a data set is plotted vs the drawdown magnitude.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*TH2TseGluRx4DBpK0blj4w.png" /><figcaption><strong>Distribution of drawdowns by rank as an example for the Shanghai index since 1996.</strong></figcaption></figure><p>I tested my algorithms with both crash identification methodologies and concluded that the first methodology (Jacobsson) is advantageous for two reasons. First, Sornette does not clearly state how much deviation from the Weibul distribution classifies a drawdown as a crash, thus human judgement is necessary. Second, his methodology leads to the identification of fewer crashes which leads to heavily imbalanced data sets. This makes it harder to collect a sufficient amount of data for a machine learning algorithm to train on.</p><p>With the collection of the seven data sets mentioned above I accumulated a total of 59,738 rows of daily stock prices and identified a total of 76 crashes.</p><h4>Problem statement and feature selection</h4><p>I formulated a classification problem with the goal of predicting, for each point in time (e.g. each trading day), whether or not a crash will occur within the next 1, 3, or 6 months.</p><p>If past price patterns are indicative of future price events, the relevant information to make a prediction on a certain day, is contained in the daily price changes of all days prior to that day. Thus, to predict a crash on day <em>t</em>, the daily price changes from each day prior to <em>t</em> could be used as a feature. However, because models presented with too many features do get slower and less accurate (“curse of dimensionality”), it makes sense to extract a few features that capture the essence of past price movements at any point in time. I therefore defined 8 different time windows that measure mean price changes over the past year (252 trading days) for each day. I used increasing window sizes from 5 days (leading up to day <em>t</em>) to 126 days (for <em>t-₁₂₆</em> to <em>t-₂₅₂</em>) to get a higher resolution of price changes in more recent times. Because price volatility is not captured when averaging price changes over multiple days, I added 8 features for the mean price volatilities over the same time windows. For each data set I normalized the mean price changes and volatilities.</p><p>To evaluate the feature selection, I performed a logistic regression and analyzed the regression coefficients. The logistic regression coefficients correspond to the change in log odds of the associated feature meaning the logarithm of how the odds (ratio of the probability of a crash vs no crash) change with a change in that feature when all other features are being held constant. For the plot below I transformed the log odds to odds. Odds greater than 1 indicate that the crash probability increases with an increase in the corresponding feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*AJMZnUfSXNgV62rvez4NAw.png" /><figcaption>Logistic regression coefficients indicating the influence of the features on the predictive variable</figcaption></figure><p>The coefficient analysis shows that the volatility over the past several days is the strongest indicator of an upcoming crash. A recent price increase, however, does not seem to indicate a crash. This is surprising at first glance because a bubble is typically characterized through an exponential increase in price. However, many of the found crashes did not occur immediately after a peak in price, instead, prices decreased over some time leading up to a crash. A high price increase over the past 6 to 12 months increases the likelihood of a predicted crash, indicating that a general price increase over the long term makes a crash more likely and that price movements over longer time periods contain valuable information for crash forecasting.</p><h4>Training, validation, and test set</h4><p>I selected the S&amp;P 500 data set for testing and the remaining 6 data sets for training and validation. I chose the S&amp;P 500 for testing because it is the largest data set (daily price information since 1950) and contains the largest amount of crashes (20). For training I performed 6-fold cross-validation. This meant running each model six times and using five data sets for training and the remaining one for validation.</p><h4>Scoring</h4><p>To evaluate the performance of each model I used the F-beta score. The F-beta score is the weighted harmonic mean of precision and recall. The beta parameter determines how precision and recall are weighted. A beta larger than one prioritizes recall and a beta smaller than one prioritizes precision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*mf9P63RFMaTrRhfdhajvzQ.png" /></figure><p>I chose a beta of 2 which puts more emphasis on recall meaning that an undetected crash gets a stronger punishment than a predicted crash that did not occur. This makes sense under a risk averse approach assuming that not predicting a crash that occurs has more severe consequences (loss of money) than expecting a crash that does not occur (missing out on potential profits).</p><h4>Regression models, Support vector machines and Decision trees</h4><p>I started with linear and logistic regression models. Regression models find the optimal coefficients for a function by minimizing the difference of the predicted and the actual target variable over all training samples. While the linear regression estimates a continuous target variable, the logistic regression estimates probabilities and is therefore generally better suited for classification problems. However, when I compared the prediction results of both models, logistic regression outperformed linear regression only in some cases. While this came as a surprise it is important to note that even though the logistic regression might provide a better fit to estimate probabilities of a crash, the sub-optimal fit of a linear regression is not necessarily a disadvantage in practice if the chosen threshold effectively separates the binary predictions. This threshold was optimized to maximize the F-beta score on the training set.</p><p>Next, I tested Support Vector Machines (SVMs). SVMs use a kernel function to project the input features into a multidimensional space and determine a hyperplane to separate positive from negative samples. Important parameters to consider are the penalty parameter C (measure for how much misclassifications should be avoided), the kernel function (polynomial or radial basis function), the kernel coefficient gamma (determines the dimension of the kernel function) and the class weight (determines how to balance positive vs negative predictions). The best SVM models achieved scores similar to those of the regression models. This makes regression models preferable as they train much faster. Decision trees were not able to perform at the same level with any of the other tested models.</p><h4>Recurrent Neural Networks</h4><p>The next step was to implement recurrent neural networks (RNNs). As opposed to traditional machine learning algorithms and traditional artificial neural networks, recurrent neural networks are able to consider the order in which they receive a sequence of input data and thus to allow information to persist. This seems like a crucial characteristic of an algorithm that deals with time series data such as daily stock returns. This is achieved through loops that connect cells so that at time-step <em>t</em> the input is not only the feature <em>xₜ</em> but also the output from the previous time step <em>hₜ-₁</em>. The figure below illustrates this concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vODZnovJafo8L_qZ-Tk1xQ.png" /><figcaption><strong>Recurrent Neural Network</strong></figcaption></figure><p>However, a major issue with regular RNNs is that they have problems learning long term dependencies. If there are too many steps between <em>xₜ-ₙ</em> and <em>hₜ</em>, <em>hₜ</em> might not be able to learn anything from <em>xₜ-ₙ</em>. To help with that, Long Short Term Memory networks (LSTMs) have been introduced. Basically, LSTMs do not only pass the output from the previous cell hₜ-₁, but also the “cell state” <em>cₜ-₁</em> into the next cell. The cell state, gets an update at each step based on the input (<em>xₜ</em> and <em>hₜ-₁</em>) and in return updates the output <em>hₜ</em>. In each LSTM cell, four neural network layers are responsible for the interactions between the inputs <em>xₜ</em>, <em>hₜ-₁</em>, <em>cₜ-₁</em> and the outputs <em>hₜ and cₜ</em>. For a detailed description of the LSTM cell architecture please refer to <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">colah’s blog</a> [4].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5W8FrASMi93Z81NlAui4w.png" /><figcaption><strong>Recurrent Neural Network with Long Short Term Memory (LSTM)</strong></figcaption></figure><p>RNNs with LSTM have the capability to detect relationships and patterns that simple regression models would not be able to find. Therefore, if an RNN LSTM would be able to learn complex price structures that precede crashes, should such a model not be able to outperform the previous tested models?</p><p>To answer that question I implemented two different RNNs with LSTM with the Python library Keras and went through rigorous hyper-parameter tuning. The first decision was the length of the input sequence for each layer. The input sequence for each time step <em>t</em> consists of daily price changes from a sequence of days leading up to <em>t</em>. This number has to be chosen with care since longer input sequences require more memory and slow down the computation. In theory the RNN LSTM should be able to find long term dependencies, however, with an LSTM implementation in Keras, the cell states are only passed from one sequence to the next one if the parameter <em>stateful</em> is set to true. In practice this implementation is cumbersome. To avoid that the network recognizes long term dependencies ranging over different data sets and epochs during training, I implemented manual resetting of the state whenever the training data switches data sets. This algorithm did not deliver strong results so I instead set <em>stateful</em> to <em>false</em> but increased the sequence length from 5 to 10 time steps and supplied the network with additional sequences of average price changes and mean volatilities from time windows prior to 10 days up to 252 trading days back (similar to the features selected for the previous tested models). Finally, I tuned hyper-parameters and tried different loss functions, number of layers, number of neurons per layer and dropout vs no dropout. The best performing RNN LSTM has a sequential layer followed by two LSTM layers with 50 neurons each, uses an adam optimizer, a binary cross entropy loss function and a sigmoid activation function for the last layer.</p><h4>Evaluation</h4><p>While hyper parameter tuning and increasing length of sequences and adding long term features lead to faster training (optimal result on the validation set after around 10 epochs), none of the RNN LSTM models were able to outperform the previous tested models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*_laJHVsYMFuP8FfWXmEB8g.png" /><figcaption>Recall vs precision for all models</figcaption></figure><p>The plot above shows the precision and recall performance of the different models. Different colors indicate different models and different shapes indicate a different predictive variable (crash in 1, 3 or 6 months). The bar plot below visualizes the F-beta scores of all models for a 1, 3 and 6 month crash prediction. <em>Random </em>stands for the expected performance of model with no predictive power that predicts a crash as often as the tested models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*UVvzh57rZ1tqNxnYa2FJ4g.png" /><figcaption>F-Beta score for all models</figcaption></figure><p>The best results show a F-beta score of 41, 37 and 29 for the prediction of a crash within 6, 3 and 1 month respectively. The precision ranged from 12–16% and the recall from 45–71%. This means that while around 50% of the crashes are detected, around 85% of the crash signals are “false alarms”.</p><h4>Conclusion</h4><p>First the bad news. The RNN LSTM is seemingly not able to learn complex price patterns that would enable it to outperform the simpler regression models. This suggests that there aren’t any complex price patterns that occur prior to all (or almost all) crashes but do not occur otherwise. This does not mean Sornette’s hypothesis of crashes that are preceded by certain price patterns fit by the log-periodic power law is not valid. It means however, that if such patterns exist, (1) these patterns also occur in cases when they are not followed by a crash, (2) there are many crashes which are not preceded by these patterns or (3) there is not enough data for a RNN to learn these patterns. While more data would definitely provide more clarity, part of the problem might be a combination of (1) and (2). Sornette fits log-periodic power laws to certain crashes identified as outliers but does not do so for all crashes with a drawdown of similar magnitude. To provide an algorithm that finds the crashes described by Sornette, the training data would need to be specifically labeled with only the crashes that fit these patterns. This might lead to an improvement of identification for those crashes but would not help with (2) since the crashes that are of a different type would not be expected to be detected. However, if provided with sufficient data and a large enough list of identified crashes, it might certainly be worth rerunning the RNN LSTM models.</p><p>The good news is that simple price patterns, defined through long term changes in price and changes in volatility, seem to be occurring regularly before crashes. The best models were able to learn these patterns and forecast a crash significantly better than a comparable random model. For example, for a crash prediction in 3 months the best regression model achieved a precision of 0.15 and a recall of 0.59 on the test set while a comparable random model with no predictive power would be expected to achieve 0.04 precision and 0.16 recall. The results look similar for 1 month and 6 month crash prediction, with the F-beta score being best for 6 month prediction and worst for 1 month prediction. Whether these results are good enough to optimize an investment strategy is debatable. However, risk averse investors might certainly allocate their portfolio positions more conservatively if the discussed regression crash indicator continuously warns of an upcoming crash.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*E8uyxgq1tQHBPSFt3UJalQ.png" /><figcaption><strong>Prediction for a crash in 3 months by the logistic regression model for the S&amp;P 500 from 1958 to 1976</strong></figcaption></figure><p>A look at test data price index charts and crash predictor indicator at the time of crashes showed that while some crashes have been detected remarkably well others occurred with no or almost no warning from the crash predictor. The figure above shows an example of a not predicted crash (in 1962) and three consecutive pretty well predicted crashes (in 1974). That some crashes are being detected better than others is in line with the assumption that certain typical price patterns exist that do precede some but not all crashes. The different algorithms mostly struggled with the same crashes, which is why I did not attempt to combine different models.</p><p>By taking a weighted average of the binary crash predictions over the past 21 days, (more recent predictions weighted stronger), the logistic regression model predicts the likelihood of a crash in the S&amp;P 500 as of November 5, 2018 within 6 months with 98.5%, within 3 months with 97% and within one month with 23%. After reading this study I leave it up to you what to do with this information.</p><p><strong><em>Note from Towards Data Science’s editors:</em></strong><em> While we allow independent authors to publish articles in accordance with our </em><a href="https://towardsdatascience.com/questions-96667b06af5"><em>rules and guidelines</em></a><em>, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our </em><a href="https://towardsdatascience.com/readers-terms-b5d780a700a4"><em>Reader Terms</em></a><em> for details.</em></p><h4>References</h4><p>[1] Why Stock Markets Crash, Didier Sornette. Book available <a href="https://www.amazon.com/Why-Stock-Markets-Crash-Financial/dp/0691175950">here</a>.</p><p>[2] How to predict crashes in financial markets with the Log-Periodic Power Law, Emilie Jacobsson. Find the paper <a href="https://www2.math.su.se/matstat/reports/serieb/2009/rep7/report.pdf">here</a>.</p><p>[3] Large Stock Market Price Drawdowns Are Outliers, (2001) Anders Johansen and Didier Sornette. Find the paper <a href="https://pdfs.semanticscholar.org/e0e9/edfc8eba3f2d1862063fcd09b60f221021f4.pdf">here</a>.</p><p>[4] Understanding LSTM Networks, colah’s blog. Link <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb66bc3e3ccd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predicting-stock-market-crashes-with-statistical-machine-learning-techniques-and-neural-networks-bb66bc3e3ccd">Predicting stock market crashes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting stock market crashes with statistical machine learning techniques and neural networks]]></title>
            <link>https://medium.com/@romanm111987/predicting-stock-market-crashes-with-statistical-machine-learning-techniques-and-neural-networks-b756d9b48497?source=rss-2b754791b7ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/b756d9b48497</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[bubble]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <dc:creator><![CDATA[Roman Moser]]></dc:creator>
            <pubDate>Fri, 09 Nov 2018 03:29:00 GMT</pubDate>
            <atom:updated>2018-11-09T03:29:00.791Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0AMNtyZhP1w-mPGms6dHg.jpeg" /></figure><p>With this blog post I am introducing the design of a machine learning algorithm that aims to forecast crashes in stock markets solely based on past price information. I start with a quick background on the problem and elaborate on my approach and findings. All the code and data are available on <a href="https://github.com/roman807/Predicting-Stock-Market-Crashes">GitHub</a>.</p><p>A stock market crash is a sharp and quick drop in total value of a market with prices typically declining more than 10% within a few days. Famous examples of major stock market crashes are the Black Monday in 1987 and the real estate bubble in 2008. A crash is usually attributable to the burst of a price bubble and is due to a massive sell-off that occurs when a majority of market participants try to sell their assets at the same time.</p><p>The occurrence of price bubbles implies that markets are not efficient. In inefficient markets prices do not always reflect fundamental asset values but are inflated or deflated based on traders’ expectations. These expectations are reinforced by traders’ subsequent actions which further inflate (or deflate) prices. This leads to positive (or negative) price bubbles which eventually burst. This phenomenon was described by George Soros as reflexivity and is the basic assumption for forecasting methods used in technical analysis.</p><p>Today, there is not much debate over the existence of bubbles in financial markets. However, understanding these inefficiencies and predicting when price bubbles will burst is a highly difficult task. Imagine you could identify a bubble that is about to build up and predict when the market will crash. You would not only be able to make a profit while prices are increasing but also to sell at the right moment to avoid losses.</p><p>Some mathematicians and physicists attempted to tackle this problem by investigating the mathematics behind price structures. One such physicist is Professor Didier Sornette who successfully predicted multiple financial crashes [1]. Sornette uses log-periodic power laws (LPPLs) to describe how price bubbles build up and burst. In essence, the LPPL fits the price movements leading up to a crash to a faster than exponentially increasing function with a log-periodic component (reflecting price volatility with increasing magnitude and frequency).</p><p>And this is where the idea for this project is coming from. If the recurring price structures found by researchers exist, should it not be possible for a machine learning algorithm to learn these patterns and predict crashes? Such an algorithm would not need to be aware of the underlying mathematical laws but would instead be trained on data with pre-identified crashes, and identify and learn these patterns on its own.</p><h4>Data and Crashes</h4><p>The first step was to collect financial data and identify crashes. I was looking for daily price information from low correlated major stock markets. Low cross-correlation is important for valid cross validation and testing of the model. The matrix below shows the cross-correlation of daily returns from 11 major stock markets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*RDXiff7swLd_rUGSCnJWEA.png" /><figcaption><strong>Correlation matrix of daily price returns for 11 major stock market indices</strong></figcaption></figure><p>To avoid having any two data sets with a cross-correlation greater than 0.5 in my collection, I proceeded with only data from the S&amp;P 500 (USA), Nikkei, (Japan), HSI (Hong Kong), SSE (Shanghai), BSESN (India), SMI (Switzerland) and BVSP (Brazil).</p><p>To identify crashes in each data set, I first calculated price drawdowns. A drawdown is a persistent decrease in price over consecutive days from the last price maximum to the next price minimum. The example below shows three such drawdowns in the S&amp;P 500 over the period from end of July to mid August 2018.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*wN8wk2QLxc8D5LHEh6294w.png" /><figcaption>Example of three drawdowns. The first one shown lasted from July 25th to July 30th 2018 and has a total loss of approximately (2846–2803)/2846 = 1.5%</figcaption></figure><p>I considered two different methodologies to identify crashes. The first one follows a suggestion by Emilie Jacobsson [2] who defines crashes in each market as drawdowns in the 99.5% quantile . With this methodology I found drawdown thresholds that classify a crash ranging from around 10% for less volatile markets like the S&amp;P 500 to more than 20% for volatile markets such as the Brazilian one. The second methodology follows the suggestion from Johansen and Sornette [3] who identify crashes as outliers, that is drawdowns that lie far from the fitted Weibul distribution when the logarithm of the rank of drawdowns in a data set is plotted vs the drawdown magnitude.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*TH2TseGluRx4DBpK0blj4w.png" /><figcaption><strong>Distribution of drawdowns by rank as an example for the Shanghai index since 1996.</strong></figcaption></figure><p>I tested my algorithms with both crash identification methodologies and concluded that the first methodology (Jacobsson) is advantageous for two reasons. First, Sornette does not clearly state how much deviation from the Weibul distribution classifies a drawdown as a crash, thus human judgement is necessary. Second, his methodology leads to the identification of fewer crashes which leads to heavily imbalanced data sets. This makes it harder to collect a sufficient amount of data for a machine learning algorithm to train on.</p><p>With the collection of the seven data sets mentioned above I accumulated a total of 59,738 rows of daily stock prices and identified a total of 76 crashes.</p><h4>Problem statement and feature selection</h4><p>I formulated a classification problem with the goal of predicting, for each point in time (e.g. each trading day), whether or not a crash will occur within the next 1, 3, or 6 months.</p><p>If past price patterns are indicative of future price events, the relevant information to make a prediction on a certain day, is contained in the daily price changes of all days prior to that day. Thus, to predict a crash on day <em>t</em>, the daily price changes from each day prior to <em>t</em> could be used as a feature. However, because models presented with too many features do get slower and less accurate (“curse of dimensionality”), it makes sense to extract a few features that capture the essence of past price movements at any point in time. I therefore defined 8 different time windows that measure mean price changes over the past year (252 trading days) for each day. I used increasing window sizes from 5 days (leading up to day <em>t</em>) to 126 days (for <em>t-₁₂₆</em> to <em>t-₂₅₂</em>) to get a higher resolution of price changes in more recent times. Because price volatility is not captured when averaging price changes over multiple days, I added 8 features for the mean price volatilities over the same time windows. For each data set I normalized the mean price changes and volatilities.</p><p>To evaluate the feature selection, I performed a logistic regression and analyzed the regression coefficients. The logistic regression coefficients correspond to the change in log odds of the associated feature meaning the logarithm of how the odds (ratio of the probability of a crash vs no crash) change with a change in that feature when all other features are being held constant. For the plot below I transformed the log odds to odds. Odds greater than 1 indicate that the crash probability increases with an increase in the corresponding feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*AJMZnUfSXNgV62rvez4NAw.png" /><figcaption>Logistic regression coefficients indicating the influence of the features on the predictive variable</figcaption></figure><p>The coefficient analysis shows that the volatility over the past several days is the strongest indicator of an upcoming crash. A recent price increase, however, does not seem to indicate a crash. This is surprising at first glance because a bubble is typically characterized through an exponential increase in price. However, many of the found crashes did not occur immediately after a peak in price, instead, prices decreased over some time leading up to a crash. A high price increase over the past 6 to 12 months increases the likelihood of a predicted crash, indicating that a general price increase over the long term makes a crash more likely and that price movements over longer time periods contain valuable information for crash forecasting.</p><h4>Training, validation, and test set</h4><p>I selected the S&amp;P 500 data set for testing and the remaining 6 data sets for training and validation. I chose the S&amp;P 500 for testing because it is the largest data set (daily price information since 1950) and contains the largest amount of crashes (20). For training I performed 6-fold cross-validation. This meant running each model six times and using five data sets for training and the remaining one for validation.</p><h4>Scoring</h4><p>To evaluate the performance of each model I used the F-beta score. The F-beta score is the weighted harmonic mean of precision and recall. The beta parameter determines how precision and recall are weighted. A beta larger than one prioritizes recall and a beta smaller than one prioritizes precision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*mf9P63RFMaTrRhfdhajvzQ.png" /></figure><p>I chose a beta of 2 which puts more emphasis on recall meaning that an undetected crash gets a stronger punishment than a predicted crash that did not occur. This makes sense under a risk averse approach assuming that not predicting a crash that occurs has more severe consequences (loss of money) than expecting a crash that does not occur (missing out on potential profits).</p><h4>Regression models, Support vector machines and Decision trees</h4><p>I started with linear and logistic regression models. Regression models find the optimal coefficients for a function by minimizing the difference of the predicted and the actual target variable over all training samples. While the linear regression estimates a continuous target variable, the logistic regression estimates probabilities and is therefore generally better suited for classification problems. However, when I compared the prediction results of both models, logistic regression outperformed linear regression only in some cases. While this came as a surprise it is important to note that even though the logistic regression might provide a better fit to estimate probabilities of a crash, the sub-optimal fit of a linear regression is not necessarily a disadvantage in practice if the chosen threshold effectively separates the binary predictions. This threshold was optimized to maximize the F-beta score on the training set.</p><p>Next, I tested Support Vector Machines (SVMs). SVMs use a kernel function to project the input features into a multidimensional space and determine a hyperplane to separate positive from negative samples. Important parameters to consider are the penalty parameter C (measure for how much misclassifications should be avoided), the kernel function (polynomial or radial basis function), the kernel coefficient gamma (determines the dimension of the kernel function) and the class weight (determines how to balance positive vs negative predictions). The best SVM models achieved scores similar to those of the regression models. This makes regression models preferable as they train much faster. Decision trees were not able to perform at the same level with any of the other tested models.</p><h4>Recurrent Neural Networks</h4><p>The next step was to implement recurrent neural networks (RNNs). As opposed to traditional machine learning algorithms and traditional artificial neural networks, recurrent neural networks are able to consider the order in which they receive a sequence of input data and thus to allow information to persist. This seems like a crucial characteristic of an algorithm that deals with time series data such as daily stock returns. This is achieved through loops that connect cells so that at time-step <em>t</em> the input is not only the feature <em>xₜ</em> but also the output from the previous time step <em>hₜ-₁</em>. The figure below illustrates this concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vODZnovJafo8L_qZ-Tk1xQ.png" /><figcaption><strong>Recurrent Neural Network</strong></figcaption></figure><p>However, a major issue with regular RNNs is that they have problems learning long term dependencies. If there are too many steps between <em>xₜ-ₙ</em> and <em>hₜ</em>, <em>hₜ</em> might not be able to learn anything from <em>xₜ-ₙ</em>. To help with that, Long Short Term Memory networks (LSTMs) have been introduced. Basically, LSTMs do not only pass the output from the previous cell hₜ-₁, but also the “cell state” <em>cₜ-₁</em> into the next cell. The cell state, gets an update at each step based on the input (<em>xₜ</em> and <em>hₜ-₁</em>) and in return updates the output <em>hₜ</em>. In each LSTM cell, four neural network layers are responsible for the interactions between the inputs <em>xₜ</em>, <em>hₜ-₁</em>, <em>cₜ-₁</em> and the outputs <em>hₜ and cₜ</em>. For a detailed description of the LSTM cell architecture please refer to <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">colah’s blog</a> [4].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5W8FrASMi93Z81NlAui4w.png" /><figcaption><strong>Recurrent Neural Network with Long Short Term Memory (LSTM)</strong></figcaption></figure><p>RNNs with LSTM have the capability to detect relationships and patterns that simple regression models would not be able to find. Therefore, if an RNN LSTM would be able to learn complex price structures that precede crashes, should such a model not be able to outperform the previous tested models?</p><p>To answer that question I implemented two different RNNs with LSTM with the Python library Keras and went through rigorous hyper-parameter tuning. The first decision was the length of the input sequence for each layer. The input sequence for each time step <em>t</em> consists of daily price changes from a sequence of days leading up to <em>t</em>. This number has to be chosen with care since longer input sequences require more memory and slow down the computation. In theory the RNN LSTM should be able to find long term dependencies, however, with an LSTM implementation in Keras, the cell states are only passed from one sequence to the next one if the parameter <em>stateful</em> is set to true. In practice this implementation is cumbersome. To avoid that the network recognizes long term dependencies ranging over different data sets and epochs during training, I implemented manual resetting of the state whenever the training data switches data sets. This algorithm did not deliver strong results so I instead set <em>stateful</em> to <em>false</em> but increased the sequence length from 5 to 10 time steps and supplied the network with additional sequences of average price changes and mean volatilities from time windows prior to 10 days up to 252 trading days back (similar to the features selected for the previous tested models). Finally, I tuned hyper-parameters and tried different loss functions, number of layers, number of neurons per layer and dropout vs no dropout. The best performing RNN LSTM has a sequential layer followed by two LSTM layers with 50 neurons each, uses an adam optimizer, a binary cross entropy loss function and a sigmoid activation function for the last layer.</p><h4>Evaluation</h4><p>While hyper parameter tuning and increasing length of sequences and adding long term features lead to faster training (optimal result on the validation set after around 10 epochs), none of the RNN LSTM models were able to outperform the previous tested models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*_laJHVsYMFuP8FfWXmEB8g.png" /><figcaption>Recall vs precision for all models</figcaption></figure><p>The plot above shows the precision and recall performance of the different models. Different colors indicate different models and different shapes indicate a different predictive variable (crash in 1, 3 or 6 months). The bar plot below visualizes the F-beta scores of all models for a 1, 3 and 6 month crash prediction. <em>Random </em>stands for the expected performance of model with no predictive power that predicts a crash as often as the tested models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*UVvzh57rZ1tqNxnYa2FJ4g.png" /><figcaption>F-Beta score for all models</figcaption></figure><p>The best results show a F-beta score of 41, 37 and 29 for the prediction of a crash within 6, 3 and 1 month respectively. The precision ranged from 12–16% and the recall from 45–71%. This means that while around 50% of the crashes are detected, around 85% of the crash signals are “false alarms”.</p><h4>Conclusion</h4><p>First the bad news. The RNN LSTM is seemingly not able to learn complex price patterns that would enable it to outperform the simpler regression models. This suggests that there aren’t any complex price patterns that occur prior to all (or almost all) crashes but do not occur otherwise. This does not mean Sornette’s hypothesis of crashes that are preceded by certain price patterns fit by the log-periodic power law is not valid. It means however, that if such patterns exist, (1) these patterns also occur in cases when they are not followed by a crash, (2) there are many crashes which are not preceded by these patterns or (3) there is not enough data for a RNN to learn these patterns. While more data would definitely provide more clarity, part of the problem might be a combination of (1) and (2). Sornette fits log-periodic power laws to certain crashes identified as outliers but does not do so for all crashes with a drawdown of similar magnitude. To provide an algorithm that finds the crashes described by Sornette, the training data would need to be specifically labeled with only the crashes that fit these patterns. This might lead to an improvement of identification for those crashes but would not help with (2) since the crashes that are of a different type would not be expected to be detected. However, if provided with sufficient data and a large enough list of identified crashes, it might certainly be worth rerunning the RNN LSTM models.</p><p>The good news is that simple price patterns, defined through long term changes in price and changes in volatility, seem to be occurring regularly before crashes. The best models were able to learn these patterns and forecast a crash significantly better than a comparable random model. For example, for a crash prediction in 3 months the best regression model achieved a precision of 0.15 and a recall of 0.59 on the test set while a comparable random model with no predictive power would be expected to achieve 0.04 precision and 0.16 recall. The results look similar for 1 month and 6 month crash prediction, with the F-beta score being best for 6 month prediction and worst for 1 month prediction. Whether these results are good enough to optimize an investment strategy is debatable. However, risk averse investors might certainly allocate their portfolio positions more conservatively if the discussed regression crash indicator continuously warns of an upcoming crash.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*E8uyxgq1tQHBPSFt3UJalQ.png" /><figcaption><strong>Prediction for a crash in 3 months by the logistic regression model for the S&amp;P 500 from 1958 to 1976</strong></figcaption></figure><p>A look at test data price index charts and crash predictor indicator at the time of crashes showed that while some crashes have been detected remarkably well others occurred with no or almost no warning from the crash predictor. The figure above shows an example of a not predicted crash (in 1962) and three consecutive pretty well predicted crashes (in 1974). That some crashes are being detected better than others is in line with the assumption that certain typical price patterns exist that do precede some but not all crashes. The different algorithms mostly struggled with the same crashes, which is why I did not attempt to combine different models.</p><p>By taking a weighted average of the binary crash predictions over the past 21 days, (more recent predictions weighted stronger), the logistic regression model predicts the likelihood of a crash in the S&amp;P 500 as of November 5, 2018 within 6 months with 98.5%, within 3 months with 97% and within one month with 23%. After reading this study I leave it up to you what to do with this information.</p><h4>References</h4><p>[1] Why Stock Markets Crash, Didier Sornette. Book available <a href="https://www.amazon.com/Why-Stock-Markets-Crash-Financial/dp/0691175950">here</a>.</p><p>[2] How to predict crashes in financial markets with the Log-Periodic Power Law, Emilie Jacobsson. Find the paper <a href="https://www2.math.su.se/matstat/reports/serieb/2009/rep7/report.pdf">here</a>.</p><p>[3] Large Stock Market Price Drawdowns Are Outliers, (2001) Anders Johansen and Didier Sornette. Find the paper <a href="https://pdfs.semanticscholar.org/e0e9/edfc8eba3f2d1862063fcd09b60f221021f4.pdf">here</a>.</p><p>[4] Understanding LSTM Networks, colah’s blog. Link <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b756d9b48497" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>