<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ASHISH DHIMAN on Medium]]></title>
        <description><![CDATA[Stories by ASHISH DHIMAN on Medium]]></description>
        <link>https://medium.com/@ashish1610dhiman?source=rss-5eda505a9933------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Kw20Ad2vN1MbQw4okUU6vQ.png</url>
            <title>Stories by ASHISH DHIMAN on Medium</title>
            <link>https://medium.com/@ashish1610dhiman?source=rss-5eda505a9933------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:55:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashish1610dhiman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Time Series forecasting in Python with Hidden Markov Models]]></title>
            <link>https://medium.datadriveninvestor.com/time-series-forecasting-in-python-with-hidden-markov-models-dac146592237?source=rss-5eda505a9933------2</link>
            <guid isPermaLink="false">https://medium.com/p/dac146592237</guid>
            <category><![CDATA[hidden-markov-models]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[ASHISH DHIMAN]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 05:13:39 GMT</pubDate>
            <atom:updated>2023-06-10T08:49:58.058Z</atom:updated>
            <content:encoded><![CDATA[<h3>Out of Time forecasting with Hidden Markov Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3t-iKhmmoqViZ7ix0OdfA.png" /><figcaption>Schematic of Forecasting with HMM model</figcaption></figure><h3>Summary</h3><p>Time Series Forecasting is a typical problem with applications across a broad spectrum of domains, including meteorology, finance, etc. Many techniques are available to make forecasts, like Exponential Smoothing, ARIMA or RNNs/LSTMs. The hidden Markov Model (or HMM) is another technique that can be used to make forecasts from sequential data. While HMM models are generally used in understanding the intrinsic structure of the process yielding the sequential data, they can also yield powerful forecasts for specific contexts.</p><p><strong>This article presents an example of how to use HMM not only to model the underlying process but further use it to make out of time forecasts</strong>. An example problem based on forecasting sales for slow-moving SKUs is also presented, and the same structure can be easily transposed to many other problems.</p><h3>Hidden Markov Model(HMM): Overview</h3><p>Hidden Markov Models (HMM) have been around since the <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model">1960s</a> and have increasingly become relevant with the availability of enough computing resources. Some researchers also think of HMM as a natural precursor of more state-of-the-art techniques like RNNs and LSTMs.</p><blockquote>The core idea of an HMM model is that there is an underlying process with latent states which can be modelled with a Markov chain. While these latent states are not directly visible to us, there exist measurements that are observable and are directly influenced by these latent states.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ilhcDckbZ828qspRTgz8XQ.png" /><figcaption>Hidden states of an HMM (<a href="https://en.wikipedia.org/wiki/Hidden_Markov_model#/media/File:Hmm_temporal_bayesian_net.svg">source</a>)</figcaption></figure><p>This notion of sequential latent states (x) and corresponding observations(y’s) is compelling since the x and y don’t need to be in the same physical space. Another advantage of HMMs is that the observed states can be either discreet or continuous. For example, a company’s stock price (Y) can be modelled by an underlying process with two distinct hidden states, Bearish or Bullish (X). Note I have used two states as an example here, and there might be any number of states in the process.</p><blockquote>With Markov models, the past doesn’t matter if the current status is known.</blockquote><p>HMM thus allows us to model the underlying system and capture non-linear dynamics. Because it is an extension of the Markov model, the underlying assumption is that the next observation in the sequence is only dependent on the preceding one. This assumption makes HMMs highly relevant for problems where we want much more weightage on the current status. With HMM, we can also find the transition probability of one hidden state to another (or regime switch), which adds another layer of intuition to the model.</p><p>For more details on the HMM model refer <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model">wikipedia</a>!</p><p><a href="https://www.youtube.com/watch?v=v5ijNXvlC5A&amp;t=4319s">https://www.youtube.com/watch?v=v5ijNXvlC5A&amp;t=4319s</a></p><p>The core focus of the article is to make out of time forecasts with HMM models, which is detailed in the section below.</p><h3>Out of Time Forecasting</h3><p>While HMM models are generally used to gain an intrinsic understanding about the underlying data generator process, they can also be extended as an out of time forecasting tool. Usually HMM model is used to sample new data, but we can also implement sort of a search space algorithm to generate out of sample forecasts.</p><p>Forecasting with an HMM model is a little nuanced because the model does not entail a formula in which we can plug t+1 to arrive at the forecast. Instead, forecasting with HMM can be considered a search through a space of observation variables to find the most probable one.</p><p>In other words, we find the forecast Y(t+1) as the outcome, which maximises the Posterior probability of observing it given the previous sequence history and transition probabilities. The mathematics of this approach are given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ws3k-fL-h9MLgrQKjW5sg.png" /><figcaption>We want to find Y(t+1) such that maximum posterior probability is maximised</figcaption></figure><p>Using the Bayes rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lB5oVQysCfsmaXdMkT8K1g.png" /><figcaption>Simplification of Posterior probability</figcaption></figure><p>The next step is then to create a set of candidate Y(t+1) values, score them with the fitted HMM model and pick the one that maximises this score. The Y(t+1) values are then added to the sequence and the process is repeated to forecast Y(t+2). This process can thus be generalised as the following steps:</p><ol><li>Find the set of probable outcomes or candidates</li><li>Out of above set find the outcome that maximises posterior probability</li><li>Add the maximum probability outcome to sequence</li><li>Repeat</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mF8FXhB-JIdvnBHx5Jj1-w.png" /><figcaption>Out of Time Forecasting Process with HMM</figcaption></figure><h3>HMM Forecasting Example: Sales Forecasting</h3><p>To illustrate the above concepts, this section introduces the example problem of 7-day sales forecast for specific items/SKUs basis their history of past sales.</p><h4>Data</h4><p>The data used as part of the project is taken from a hackathon organised for Georgia Tech MS Analytics students by a major North American retailer. The said data can be with the help of DVC <a href="https://github.com/ashish1610dhiman/bestbuy/blob/master/data.dvc">here</a>. The data includes Daily Sales history for a number of SKUs. The goal here is to employ that history to fit an HMM model and use it to forecast sales for 1 week ahead.</p><p>*For a quick starter on dvc refer this <a href="https://towardsdatascience.com/introduction-to-dvc-data-version-control-tool-for-machine-learning-projects-7cb49c229fe0">great TDS article</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*-dCnt25SgudUkMDin8Ik9w.png" /><figcaption>Sales data for a specific SKU</figcaption></figure><h4>Fitting HMM</h4><p>To fit the HMM model we will require a sequence of Y observations. In this case we create the Y observations with the example of lagged daily sales and corresponding difference of lagged sales with sales today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*oAqelEiXfMOYjYzfJGJadw.png" /><figcaption>Vector of lagged sales and corresponding deltas for fitting HMM</figcaption></figure><p>By using the lagged variables to train the HMM, we ensure there is no data leak from train period to out of time period.</p><pre>&quot;&quot;&quot;<br># Created by ashish1610dhiman at 23/12/22<br>Contact at ashish1610dhiman@gmail.com<br>&quot;&quot;&quot;<br><br>import pandas as pd<br>from hmmlearn.hmm import GaussianHMM<br>import itertools<br>import numpy as np<br><br>class sku_predict():<br>    #initialise<br>    def __init__(self,train_test, sku_id):<br>        self.sku_id = sku_id<br>        train_test_sku = train_test[train_test.Encoded_SKU_ID == sku_id]<br>        train_test_sku.index = train_test_sku[&quot;SALES_DATE&quot;]<br>        train_test_sku = train_test_sku.sort_index()<br>        self.train_test_sku = train_test_sku<br>        self.sales_data = None<br>        self.n_lags = None<br>        self.X = None<br>        self.model = None<br>    def get_features(self, n_lags = 3):<br>        self.n_lags = n_lags<br>        sales_data = self.train_test_sku[[&quot;DAILY_UNITS&quot;]]<br>        for lag in range(1,n_lags+1):<br>            du_lag = f&quot;DAILY_UNITS_lag{lag}&quot;<br>            sales_data[du_lag] = sales_data[&quot;DAILY_UNITS&quot;].shift(lag)<br>            sales_data[f&quot;change_lag{lag}&quot;] = (sales_data[&quot;DAILY_UNITS&quot;] - sales_data[du_lag])<br>        print (f&quot;Created {n_lags} lag features&quot;)<br>        self.sales_data = sales_data #featurised data<br>        return sales_data<br>    def split_train_test(self,start_dt):<br>        train1 = self.sales_data[:pd.to_datetime(start_dt)+pd.DateOffset(-1)]<br>        valid1 = self.sales_data[pd.to_datetime(start_dt):]<br>        return (train1,valid1)<br>    def fit_hmm(self,train, start_dt, n_components1 = 2):<br>        hmm = GaussianHMM(n_components=n_components1)<br>        # fit hmm to pct_change and DAILY_UNITS_lag1<br>        lag_price_cols = [f&quot;DAILY_UNITS_lag{lag}&quot; for lag in range(1,self.n_lags+1)]<br>        lag_change_cols = [f&quot;change_lag{lag}&quot; for lag in range(1,self.n_lags+1)]<br>        print (&quot;Training on :&quot;,lag_price_cols,lag_change_cols)<br>        X = train[lag_price_cols + lag_change_cols]<br>        X = X[start_dt:]<br>        self.X = X<br>        hmm.fit(X.dropna())<br>        self.model = hmm</pre><h4>Out of Time Forecasting</h4><p><strong>Step 1: Generating Candidate Set</strong></p><p>As described in the section above, the next step for of time forecasting with an HMM model is to carry out a search in the space of possible observed variables. This is done by creating a grid of possible values for the lagged sales and difference in sales. We then take a cross product of these grids to arrive at the candidate set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5vT9CzPL5cyYd_iZwU1OA.png" /><figcaption>Candidate space to search through</figcaption></figure><pre>def _compute_all_possible_outcomes(self,n_steps_pct,n_steps_price_lag):<br>        # pct_change_range = np.linspace(-0.61, 0.61, n_steps_pct)<br>        # change_range = np.linspace(self.X[&quot;change_lag1&quot;].min(),self.X[&quot;change_lag1&quot;].max(), n_steps_pct)<br>        #TODO<br>        DAILY_UNITS_range = np.unique(self.X.dropna()[[col for col in self.X.columns if &quot;UNITS&quot; in col]])<br>        change_range = np.unique(self.X.dropna()[[col for col in self.X.columns if &quot;change&quot; in col]])<br>        # DAILY_UNITS_range = np.linspace(self.X[&quot;DAILY_UNITS_lag1&quot;].min(),\<br>        #                                      self.X[&quot;DAILY_UNITS_lag1&quot;].quantile(0.75), n_steps_price_lag)<br>        change_list = [DAILY_UNITS_range]*self.n_lags + [change_range]*self.n_lags<br>        all_outcomes = np.array(list(itertools.product(*change_list)))<br>        return (all_outcomes)</pre><p>Note we can choose to create the individual search grids set with any arbitrary level of granularity. For the example above we have search grid simply as the unique values of variable that variable, given the variable has very low variability.</p><p><strong>Step 2: Maximising Posterior Probability</strong></p><p>Each of these outcomes are then scored with the fitted HMM to arrive at the posterior probability of the outcome being next in the sequence. The outcome which maximises this probability is then chosen as the forecasted outcome. This forecasted outcome is then added to the sequence, and the process is repeated for the following outcome (dynamic forecasting).</p><pre>def _get_most_probable_outcome(self,prev_data,n_steps_pct,n_steps_price_lag):<br>        hmm = self.model<br>        all_outcomes = self._compute_all_possible_outcomes(n_steps_pct,n_steps_price_lag)<br>        outcome_score = list(map(lambda x: hmm.score(np.row_stack((prev_data, x))), \<br>                                 all_outcomes))<br>        most_probable_outcome = all_outcomes[np.argmax(outcome_score)]<br>        return (most_probable_outcome)</pre><p><strong>Step 3: Prediction</strong></p><p>After finding out the most probable outcome, the forecast can then be calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0Jyl3V0I5hHSFr09IIb4g.png" /><figcaption>Calculation for forecast | w1,w2 are weights and can be tuned accordingly</figcaption></figure><p>After making the forecast for t+1 time period, we add it to the sequence and repeat the above steps.</p><pre>def predict(self,valid1, pred_latency, n_steps_pct = 100,n_steps_price_lag = 100, w1=0.55, w2 = 0.45):<br>        prev_data_init = self.X[-pred_latency:]<br>        predict_df = pd.DataFrame()<br>        print (&quot;Starting Prediction&quot;)<br>        for i, dt in enumerate(valid1.index):<br>            if i == 0:<br>                prev_data = prev_data_init<br>            else:<br>                # print(prev_data.shape)<br>                self.model.fit(prev_data[-pred_latency:])<br>            print (f&quot;-----&gt; Predicting for day:{i}&quot;)<br>            most_probable_outcome = self._get_most_probable_outcome(prev_data, 100, 100)<br>            temp = pd.DataFrame(most_probable_outcome).T<br>            temp.index = [dt]<br>            temp.columns = prev_data_init.columns<br>            prev_data = pd.concat([prev_data, temp])<br>            predict_df = pd.concat([predict_df, temp])<br>            # print(most_probable_outcome)<br>        predict_df[&quot;predicted1&quot;] = predict_df[&quot;DAILY_UNITS_lag1&quot;] + (predict_df[&quot;change_lag1&quot;])<br>        predict_df[&quot;predicted2&quot;] = predict_df[&quot;DAILY_UNITS_lag2&quot;] + (predict_df[&quot;change_lag2&quot;])<br>        predict_df[&quot;predicted&quot;] = (w1*predict_df[&quot;predicted1&quot;] + w2*predict_df[&quot;predicted2&quot;])<br>        return predict_df</pre><h4>Results</h4><p>In the example scenario, the HMM model provided significant value for SKUs which exhibit a significant shift in sales in recent months or a regime change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Re421nUrqCWDHVQdZsEDDg.png" /><figcaption>Some SKUs show a regime change</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XhRxRinymCmqy8OqTgUflA.png" /><figcaption>Forecast of Sales with a naive model (left) vs HMM forecast (right)</figcaption></figure><p>As we can clearly see in the above graph, HMM was able to forecast sales pickup in the last week, even when there were no sales in the previous month. Other conventional models like Holt Winters and ARIMA (SARIMA) and STL decomposition were found to poorly forecast such scenarios.</p><p>The HMM model was also in the case of SKUs which exhibit high volatility in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*bq3DVZbd-oMOK-ZIsAOZuA.png" /><figcaption>Sample forecast from the HMM model</figcaption></figure><h3>Conclusion</h3><p><strong>HMM is a powerful model for out of time forecasting for a sequence, especially in scenarios where a regime change behaviour might be apparent. </strong>The accuracy of the forecast can be further improved by either increasing the number of observed variables or using more elaborate search space of observation variables. At the same time, care should be taken while fitting HMM since the solution of HMM is a local optimum and it is very easy to overfit the model.</p><p><em>Feel free to connect me on </em><a href="https://www.linkedin.com/in/ashish1610dhiman/"><em>LinkedIn</em></a><em> if you would like to find out more about my project or experiences. I would also be glad to discuss other ideas to improve this article :)</em></p><p><a href="https://ashish1610dhiman.github.io/ashish_portfolio/">https://ashish1610dhiman.github.io/ashish_portfolio/</a></p><h4>Code Repo</h4><p>The implementation and results for this example can be accessed on <a href="https://github.com/ashish1610dhiman/bestbuy/blob/master/src/ad_hmm.py">Github</a>.</p><h4>References</h4><p>[1] <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model">https://en.wikipedia.org/wiki/Hidden_Markov_model</a></p><p>[2] <a href="https://timeseriesreasoning.com/contents/hidden-markov-models/">https://timeseriesreasoning.com/contents/hidden-markov-models/</a></p><p>[3] <a href="https://hmmlearn.readthedocs.io/en/latest/">https://hmmlearn.readthedocs.io/en/latest/</a></p><p>[4] <a href="https://rubikscode.net/2021/09/06/stock-price-prediction-using-hidden-markov-model/">https://rubikscode.net/2021/09/06/stock-price-prediction-using-hidden-markov-model/</a></p><p>[5] <a href="https://users.cs.duke.edu/~bdhingra/papers/stock_hmm.pdf">https://users.cs.duke.edu/~bdhingra/papers/stock_hmm.pdf</a></p><p>[6] <a href="https://www.youtube.com/watch?v=v5ijNXvlC5A">https://www.youtube.com/watch?v=v5ijNXvlC5A</a></p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dac146592237" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/time-series-forecasting-in-python-with-hidden-markov-models-dac146592237">Time Series forecasting in Python with Hidden Markov Models</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Yield Curve with Gaussian Process in PyTorch]]></title>
            <link>https://medium.datadriveninvestor.com/forecasting-yield-curve-with-gaussian-process-in-pytorch-28f6570b47a0?source=rss-5eda505a9933------2</link>
            <guid isPermaLink="false">https://medium.com/p/28f6570b47a0</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[gaussian-process]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[yield-curve]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[ASHISH DHIMAN]]></dc:creator>
            <pubDate>Wed, 07 Jun 2023 05:38:58 GMT</pubDate>
            <atom:updated>2023-06-07T14:43:12.950Z</atom:updated>
            <content:encoded><![CDATA[<h3>Summary</h3><p>The US Treasury yield curve is important as it serves as a vital indicator of the market’s expectations about future economic conditions and interest rates. Forecasting the yield curve provides valuable insights for investors, policymakers, and analysts, aiding in making informed decisions regarding investment strategies, monetary policy, and risk management. In this article I explore the use of Gaussian Process regression, a Bayesian Machine learning framework to forecast the yield curve and the corresponding prediction interval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*oTlFHYuCtYe3rIL02jntsA.png" /><figcaption>Forecasting US Treasury Yield rate with GP</figcaption></figure><h3>Gaussian Process Regression: Overview</h3><p>A Gaussian process is a powerful mathematical framework used for modeling and analysing complex systems and datasets. It provides a flexible and non-parametric approach for estimating and predicting the behavior of unknown functions. At its core, a Gaussian process (GP) is an extension of a Gaussian random vector, and it can be interpreted as a stochastic process that operates on continuous functions. GP assumes that the function values (f) at a set of M inputs are distributed as jointly gaussian.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*-WjHZFHvpshMHjDe9EH1kg.png" /><figcaption>GP assumption</figcaption></figure><p>The essence of this approach lies in the calculation of the conditional distribution, which, within a Bayesian framework, corresponds to deriving the posterior distribution from the prior distribution .In other words a GP defines a prior over a space of functions and then finds a conditional distribution based on seen data. This is achievable by substituting the conventional covariance matrix with a kernel function (K), thereby harnessing the advantages of kernel methods.</p><p>Let’s assume that we have observed function values, y, at N points X, corrupted by Gaussian noise (ε), and our objective is to estimate the function value, f*, at a new, unseen point, x*. Because we assumed joint gaussian distribution over the points N and x*, hence we can write:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6H80WP8r2GPuQbqwzTSpA.png" /></figure><p>This allows us to take advantage of multivariate normal distribution to find the conditional distribution of f* given y and X (or training data) in a closed form. Thus essentially allowing us to find the posterior predictive distribution over the test point x*.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pW0fi0z4j-Hc6rWd1AcbQg.png" /><figcaption>Conditional distribution of x*, for proof check <a href="https://github.com/ashish1610dhiman/math_proofs/blob/main/hw1_iuq_ad.pdf">here</a>!</figcaption></figure><h4>Kernel function</h4><p>The kernel function, denoted as K(xi, xj), plays a fundamental role in Gaussian processes analysis as it defines the covariance between two points. The kernel function is required to be positive definite, often referred to as a Mercer kernel, which is a symmetric function such that K(θ) : X × X → R+, where θ are the hyperparameters. Some common kernels used in Gaussian processes include the squared exponential (or Gaussian), Matérn, rational quadratic, periodic, and linear kernels.</p><p>The choice of kernel function of GP dictates the characteristics of the function f(x) it generates. By selecting an appropriate kernel, we can regulate properties such as regularity, periodicity, and monotonicity in the generated samples. Furthermore, employing operations on kernels enables us to uncover intricate patterns and structures within the data. Kernels can be combined through addition, multiplication, and convolution, resulting in another valid covariance function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZOA3cRyfmKgh3jAxKh4kEg.png" /><figcaption>functions samples from a GP with different kernels; images from Probabilistic ML (Kevin P Murphy)</figcaption></figure><p>The kernel hyperparameters can have a large effect on the predictions from the model, and they are chosen so as to maximise the marginal log likelihood. Once we have an expression for the logarithm of the marginal likelihood and its derivative, we can employ a gradient-based optimizer to estimate the kernel parameters. However, due to the non-convex nature of the objective, local minima can pose a challenge, as demonstrated below. To overcome this, multiple restarts might be necessary.</p><h3>Forecasting Yield Curve</h3><p>The US Treasury yield curve is important because it provides crucial insights into the expectations, sentiments, and perceptions of market participants regarding future economic conditions and interest rate movements. To demonstrate the application of Gaussian process methods in finance, the process of fitting the U.S. yield curve is given below.</p><p>Our objective is to utilize Gaussian Process for generating 1-day ahead forecasts of yield rates for 2-year maturity and 10-year maturity bonds throughout the test period spanning from 2017 to 2018.</p><h4>Data Collection</h4><p>Obtaining the US treasury yield curve data is both free and convenient, as it can be readily downloaded from the official website of the <a href="https://home.treasury.gov/resource-center/data-chart-center/interest-rates/TextView?type=daily_treasury_yield_curve&amp;field_tdr_date_value=2023">U.S. Department of the Treasury</a>. By leveraging the US Treasury website, we can access and download the daily yield information for various durations of US bonds.</p><p>To scrape the data, I have utilised the Python libraries <strong>requests</strong> and <strong>Beautiful Soup</strong>, which allow us to retrieve and extract the desired information from the U.S. Department of the Treasury website.</p><pre>import requests<br>import pandas as pd<br>from bs4 import BeautifulSoup<br>import numpy as np<br><br>res = requests.get(&quot;https://home.treasury.gov/resource-center/data-chart-center/interest-rates/TextView?type=daily_treasury_yield_curve&amp;field_tdr_date_value=2023&quot;)<br>html = BeautifulSoup(res.text, &quot;lxml&quot;)<br><br>link = html.find(&quot;div&quot;, {&quot;class&quot;: &quot;csv-feed views-data-export-feed&quot;}).find(&quot;a&quot;).get(&quot;href&quot;)<br>print (link)<br>#https://home.treasury.gov/resource-center/data-chart-center/interest-rates/daily-treasury-rates.csv/2023/all?type=daily_treasury_yield_curve&amp;field_tdr_date_value=2023&amp;page&amp;_format=csv</pre><pre>df_list = []<br>START = 2010<br>END = 2018<br><br> <br>for year in range(START,END):<br>    print(f&quot;Scraping for {year}&quot;)<br>    response = requests.get(link.replace(&quot;2023&quot;, str(year))).text.split(&quot;\n&quot;)<br> <br>    list_table = [[*row.split(&quot;,&quot;)] for row in response[1:]]<br> <br>    df_year_i = pd.DataFrame(list_table, columns=response[0].split(&quot;,&quot;))<br>    df_year_i.columns = df_year_i.columns.str.strip(&#39;&quot;&#39;)<br>    df_list.append(df_year_i)<br><br>df_all = pd.concat(df_list)<br>df_all.index = pd.DatetimeIndex(df_all[&quot;Date&quot;])<br><br>#cast to numeric<br>cast_cols = df_all.columns.drop(&#39;Date&#39;)<br>df_all[cast_cols] = df_all[cast_cols].apply(pd.to_numeric, errors=&#39;coerce&#39;)<br><br>print (f&quot;Scraped data shape={df_all.shape}&quot;)<br><br>df_all[[&quot;1 Mo&quot;,&quot;3 Mo&quot;,&quot;2 Yr&quot;, &quot;10 Yr&quot;]].plot()<br>#Scraped data shape=(2003, 12)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*2Segk7grUTK6RUpBezJCQw.png" /><figcaption>Time series of daily yields for different duration US bonds from 2010 to 2018</figcaption></figure><h4>EDA</h4><p>After evaluating the correlation between yield rates we find that that bonds of similar maturity exhibit high correlation. This behaviour is expected and helps us identify the potential exogenous variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8ObT00IUsgo2Wc0iYLcCQ.png" /><figcaption>Correlation matrix for yield rates</figcaption></figure><p>The yield rates also exhibit high auto-corelation and memory. One approach to quantify the memory in time-series data is to calculate the Hurst exponent, which provides a measure of long-term autocorrelation. A Hurst component of &gt;0.4 may hint at long term positive auto correlation.</p><pre>def get_hurst_exponent(time_series, max_lag=20):<br>    &quot;&quot;&quot;Returns the Hurst Exponent of the time series&quot;&quot;&quot;<br>    lags = range(2, max_lag)<br>    # variances of the lagged differences<br>    tau = [np.std(np.subtract(time_series[lag:], time_series[:-lag])) for lag in lags]<br>    # calculate the slope of the log plot -&gt; the Hurst Exponent<br>    reg = np.polyfit(np.log(lags), np.log(tau), 1)<br>    return reg[0]<br><br>print(pd.DataFrame(df_clean.apply(lambda x: get_hurst_exponent(x.values, 40))).T)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Prkc33_HPq4gWShzwQN5A.png" /><figcaption>Hurst exponent of yield rates for different maturities</figcaption></figure><p>As we can see from above there is indication of memory in some yield rates. In such cases where we deal with time-series data that demonstrate memory effects, employing a straightforward forecast often produces excellent results when evaluated using standard accuracy measures like mean squared error. Here we use the persistence model as the benchmark model, as per which the current value is taken as a forecast for tomorrow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/72/1*UHdLkxsyV54OWrIwgBBq3w.png" /><figcaption>Persistence model prediction</figcaption></figure><h4>Gaussian Process model</h4><p>In this study, we have undertaken the training and validation of a multi task Gaussian Process (GP) model utilising a cross-validation approach. The use of multitask GP enables the simultaneous learning of similarities among multiple output functions. This approach is valuable in our case since the different yields share common inputs, and also these functions exhibit similarity.</p><pre>class MultitaskGPModel(gpytorch.models.ExactGP):<br>    def __init__(self, train_x, train_y, kernel_func, likelihood):<br>        super(MultitaskGPModel, self).__init__(train_x, train_y, likelihood)<br>        self.mean_module = gpytorch.means.MultitaskMean(<br>            gpytorch.means.ConstantMean(), num_tasks=train_y.shape[-1]<br>        )<br>        self.covar_module = gpytorch.kernels.MultitaskKernel(<br>            kernel_func, num_tasks=train_y.shape[-1], rank=1<br>        )<br><br>    def forward(self, x):<br>        mean_x = self.mean_module(x)<br>        covar_x = self.covar_module(x)<br>        return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)<br>    <br>def train_gp(train_x, train_y, kernel_func, N_ITER = 50,verbose=True):<br>    likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=train_y.shape[-1])<br>    if type(train_x) != torch.Tensor:<br>        train_x_tensor = torch.tensor(train_x.values).float()<br>        train_y_tensor = torch.tensor(train_y.values).float()<br>    else:<br>        train_x_tensor = train_x<br>        train_y_tensor = train_y<br>    model = MultitaskGPModel(train_x_tensor, train_y_tensor,kernel_func, likelihood)<br>#     print (&quot;Initialised GP&quot;)<br>    # Find optimal model hyperparameters<br>    model.train()<br>    likelihood.train()<br>    # Use the adam optimizer<br>    optimizer = torch.optim.Adam(model.parameters(), lr=0.1)  # Includes GaussianLikelihood parameters<br>    # &quot;Loss&quot; for GPs - the marginal log likelihood<br>    mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)<br>    print (&quot;Training GP&quot;)<br>    for i in range(N_ITER):<br>        optimizer.zero_grad()<br>        output = model(train_x_tensor)<br>        loss = -mll(output, train_y_tensor)<br>        loss.backward()<br>        if verbose and ((i%10==0) or (i==N_ITER-1)):<br>            print(&#39;Iter %d/%d - Loss: %.3f&#39; % (i + 1, N_ITER, loss.item()))<br>        optimizer.step()<br>    return model,likelihood<br><br>def predict_gp(test_x,likelihood,model):<br>    # Set into eval mode<br>    model.eval()<br>    likelihood.eval()<br>    test_x_tensor = torch.tensor(test_x.values).float()<br>    # Make predictions<br>    with torch.no_grad(), gpytorch.settings.fast_pred_var():<br>        predictions = likelihood(model(test_x_tensor))<br>        mean = predictions.mean<br>        # lower, upper = predictions.confidence_region()   <br>    return mean,predictions </pre><p>Also note that because we are dealing with time series data, implementing cross-validation becomes more challenging. Randomly assigning samples to either the test set or the train set is not appropriate since it would involve utilizing future values to forecast past values, which is logically incorrect. In simpler terms, we need to avoid incorporating future information when training our model. Given the temporal dependency between observations, it is crucial to preserve this relationship during testing to ensure accurate evaluation. The schematic below explains my CV approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjdwb98HEIFDoe4Cvr_jQw.png" /><figcaption>Time Series CV schematic</figcaption></figure><p>Because the functions sampled from GP are highly dependent on the choice of kernels, I have tested two kernel structures. In future version, I would also like to try deep learning of the kernels.</p><ol><li>RBF or Squared Exponential (SE) kernel</li><li>RBF * Linear kernel</li></ol><pre>kernel_repo = {<br>    #SE<br>    &quot;kernel1&quot;:gpytorch.kernels.RBFKernel(),\<br>    #SE*Lin<br>    &quot;kernel2&quot;:gpytorch.kernels.RBFKernel()*gpytorch.kernels.LinearKernel()<br>}</pre><p>Each kernel is fitted for 20 trials to determine the optimal set of kernel hyperparameters that yield the best fit basis the mean CV MSE. Subsequently, the best-performing model was selected by comparing the performance of the two kernels.</p><pre>def get_kernel_cv_result_raw(kernel_j, N_TRIALS=10, N_TRAINING_ITER=50, verbose=False):<br>    result_j = pd.DataFrame()<br>    model_dict_j = {}<br>    for trial in range(N_TRIALS):<br>        for i, (train_index, test_index) in enumerate(tscv.split(X)):<br>            print(f&quot;Sample {trial}, Fold {i}:&quot;)<br>            train_xi,train_yi = X.iloc[train_index],y.iloc[train_index]<br>            test_xi,test_yi = X.iloc[test_index],y.iloc[test_index]<br>#             if verbose:<br>#                 print(f&quot;Training from {train_xi.index.min()} to {train_xi.index.max()}&quot;)<br>            model_i,lik_i = train_gp(train_xi,train_yi,kernel_j,N_TRAINING_ITER,verbose)<br>            model_dict_j[trial,i,&quot;model&quot;]=model_i<br>            if verbose:<br>                print(f&quot;Testing from {test_xi.index.min()} to {test_xi.index.max()}\n&quot;)<br>            mean_i,pred_i = predict_gp(test_xi,lik_i,model_i)<br>            model_dict_j[trial,i,&quot;prediction&quot;]=pred_i<br>            results_i = {<br>              &quot;trial&quot;:trial,<br>              &quot;fold&quot;:i,<br>              &quot;test_year&quot;:test_xi.index.year.max()}<br>            y_pred_i = pd.DataFrame(mean_i,index=test_yi.index,columns = test_yi.columns)<br>            for test_col in test_yi.columns:<br>                results_i[test_col] = {<br>                  &quot;RMSE&quot;: mean_squared_error(test_yi[test_col],y_pred_i[test_col])**0.5,\<br>                  &quot;MAPE&quot;: mean_absolute_percentage_error(test_yi[test_col],y_pred_i[test_col])    <br>                }<br>            result_j = pd.concat([result_j,pd.DataFrame.from_dict(results_i,orient=&quot;columns&quot;)])<br>    return (result_j,model_dict_j)<br><br>all_results = {}<br>all_results_df = pd.DataFrame()<br>for kernel_name,kernel_func_j in kernel_repo.items():<br>    print (f&quot;Using {kernel_name}&quot;)<br>    kernel_result,kernel_models = get_kernel_cv_result_raw(kernel_func_j,10,200,True)<br>    kernel_results_trials = kernel_result.loc[&quot;RMSE&quot;].reset_index().drop(columns=[&quot;fold&quot;,&quot;test_year&quot;]).groupby([&quot;trial&quot;,&quot;index&quot;]).mean()<br>    kernel_results_trials[&quot;err_avg&quot;] = kernel_results_trials.mean(axis=1)<br>    kernel_results_trials[&quot;rnk_&quot;] = kernel_results_trials[&quot;err_avg&quot;].rank()<br>    all_results[kernel_name] = (kernel_results_trials,kernel_models)<br>    result_i = kernel_results_trials.copy()<br>    result_i = result_i.loc[result_i.rnk_==1]<br>    result_i[&quot;kernel&quot;] = kernel_name<br>    all_results_df = pd.concat([all_results_df,result_i])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*HmksVuh4tLEJkbefyxoN-w.png" /><figcaption>mean CV RMSE for best trials | kernel2: RBF*Lin kernel outperforms kernel1: RBF</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDvctx-0GKPAis22wN32dQ.png" /><figcaption>Comparison of forecast for persistence vs GP for 2016</figcaption></figure><p>In my experiments the RBF*Linear kernel outperformed the vanilla RBF kernel, although the difference is minimal. This chosen kernel is then trained on the full Train + Validation set i.e. the end of 2016. The chosen model is also trained with 20 runs to mitigate the local optimums.</p><h4>Results</h4><p>In this simple example, we observe that the two methods yield equivalent results, i.e. the MSE of GP and persistence model are very similar. Nevertheless, the GP-ARX method stands out by providing the capability to estimate confidence intervals, which can additional utility in practical applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*DENCBwrcIUI7ZoYs3aV-PQ.png" /><figcaption>Yield rate MSE of persistence vs GP for 2017</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*oTlFHYuCtYe3rIL02jntsA.png" /><figcaption>Yield rate forecast with GP for 2017</figcaption></figure><h3>Conclusion</h3><p>Gaussian Processes (GPs) can be employed as a semi-parametric alternative to widely used parametric approaches like the Nelson-Siegel model. However, in my experiments the GPs do not outperform traditional econometric methods when it comes to yield rate forecasting, as both approaches yield comparable results. Nonetheless GPs present a powerful non parametric approaches for a variety of use-cases, and you can refer the resources below for learning more about GP.</p><p>Note however the selection of the kernel function and features plays a crucial role in the analysis. In this particular example, I have examined only two basic kernels, but exploring more sophisticated kernel options could potentially lead to improved outcomes. For the next iteration of this study, I plan to incorporate a deep learning approach to learn the kernel function, allowing for greater flexibility and potential improved performance.</p><p><em>Feel free to connect me on </em><a href="https://www.linkedin.com/in/ashish1610dhiman/"><em>LinkedIn</em></a><em> if you would like to find out more about my project or experiences. I would also be glad to discuss other ideas to improve this analysis :)</em></p><p><a href="https://ashish1610dhiman.github.io/ashish_portfolio/">Ashish Dhiman | Home</a></p><h4>Code</h4><p>The implementation and complete results for this analysis can be found on <a href="https://github.com/ashish1610dhiman/UQ_deep_learning/blob/main/README.md">Github</a>.</p><h3><strong>References</strong></h3><ol><li><a href="https://arxiv.org/pdf/1703.01536.pdf">A Statistical Machine Learning Approach to Yield Curve Forecasting (Sambasivan et al 2017)</a></li><li><a href="http://thierry-roncalli.com/download/Bayesian_Optimization.pdf">Financial Applications of Gaussian Process and Bayesian Optimisation (Gonzalvez et al 2019)</a></li><li><a href="https://probml.github.io/pml-book/book1.html">Probabilistic Machine Learning, Kernel methods (Kevin P Murphy)</a></li></ol><p>4. <a href="https://proceedings.neurips.cc/paper_files/paper/2007/file/66368270ffd51418ec58bd793f2d9b1b-Paper.pdf">Multi-task Gaussian Process Prediction (Bonilla et al 2007)</a></p><p>5. M<a href="https://docs.gpytorch.ai/en/stable/examples/03_Multitask_Exact_GPs/Multitask_GP_Regression.html">ulti Task Regression GPytorch</a></p><p>6. <a href="https://towardsdatascience.com/introduction-to-the-hurst-exponent-with-code-in-python-4da0414ca52e">Introduction to the Hurst exponent</a></p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28f6570b47a0" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/forecasting-yield-curve-with-gaussian-process-in-pytorch-28f6570b47a0">Forecasting Yield Curve with Gaussian Process in PyTorch</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Clustering of stock behaviour during Covid]]></title>
            <link>https://medium.datadriveninvestor.com/time-series-clustering-of-stock-behaviour-during-covid-9bd25b8c7a5?source=rss-5eda505a9933------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bd25b8c7a5</guid>
            <category><![CDATA[dynamic-time-warping]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[sp-500-index]]></category>
            <category><![CDATA[time-series-clustering]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[ASHISH DHIMAN]]></dc:creator>
            <pubDate>Mon, 02 Jan 2023 03:27:09 GMT</pubDate>
            <atom:updated>2023-01-20T22:59:23.867Z</atom:updated>
            <content:encoded><![CDATA[<h3>Time Series Clustering for grouping stock prices during Covid with Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w98rH1XWHnPP2HYW5W8XGQ.jpeg" /><figcaption>Final clustering output</figcaption></figure><h3>Summary</h3><p>The onset of the Covid pandemic brought a profound shock to the financial markets in early 2020. Major indices and stocks took a resounding hit, with <a href="https://www.cnbc.com/2021/03/16/one-year-ago-stocks-dropped-12percent-in-a-single-day-what-investors-have-learned-since-then.html">SP500 showing a decline of about 34% from its February high to its March 23 bottom</a>. Following the initial shock, though, many stocks exhibited a strong recovery on the back of interest rate cuts by the Fed and other government policies.</p><p>This analysis aims to uncover groups among the S&amp;P 500 stocks in their drop and rebound trajectory shown during Covid and identify the drivers behind them. While sector information provides an intrinsic notion of clustering, it does not capture patterns spanning across sectors. Hence the study leverages Time Series Clustering to temporally cluster the stock prices into ’n’ buckets: found through the elbow plot. Finally, the buckets are analysed to identify the drivers.</p><h4>Pipeline</h4><ul><li>Extract S&amp;P 500 composition from Wikipedia</li><li>Pull stock price data with Yahoo Finance</li><li>Preprocess the data</li><li>Run the Clustering with Dynamic Time Wrapping distance on Kaggle</li><li>Analyse the results</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1W0Lo4KrOsEK57hs4KKqZw.png" /><figcaption>Pipeline for the Time Series Clustering</figcaption></figure><h3>Data Collection</h3><p>Yahoo Finance library provides a convenient method to pull daily stock price data for known tickers. So the first step is to get the list of stocks comprising the <a href="https://en.wikipedia.org/wiki/List_of_S%26P_500_companies">S&amp;P500 index from Wikipedia</a>.</p><pre>import pandas as pd<br><br>table=pd.read_html(&#39;https://en.wikipedia.org/wiki/List_of_S%26P_500_companies&#39;)<br>df_ticker = table[0]<br>df_ticker.to_csv(&quot;../../data/sp500_ticker_sector.csv&quot;,index=False)</pre><p>After accessing the list of stocks in S&amp;P500, the stock price data can be pulled as:</p><pre>import yfinance as yf<br>import datetime<br><br>start = datetime.datetime(2015, 1, 1)<br>end = datetime.datetime(2022, 11, 30)<br><br>tickers = list(df_ticker[&quot;Symbol&quot;])<br>print(tickers[:5])<br>#[&#39;MMM&#39;, &#39;AOS&#39;, &#39;ABT&#39;, &#39;ABBV&#39;, &#39;ACN&#39;]<br><br>stock_prices = yf.download(tickers, start=start, end=end)<br>stock_prices.to_csv(&quot;../../data/sp500_stock_prices.csv&quot;, index =None)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uz_b8-g3VT9ysiB0IpJWyg.png" /><figcaption>head for the pulled data of stock prices</figcaption></figure><p><em>Note the S&amp;P500 composition is consciously chosen as of Nov 22, 2022.</em></p><h3>Time Series Clustering: Overview</h3><p><a href="https://tslearn.readthedocs.io/en/stable/user_guide/clustering.html">Time Series Clustering</a> is an unsupervised Machine Learning technique used to group (or cluster) time series together basis some notion of <strong>similarity</strong>. As one might expect, the choice of this similarity measure is critical for the clustering process. While the classical clustering problems generally rely on euclidean or Manhattan distance for the distance (or inverse similarity), these measures do not generalise well in the case of temporal data. On the other hand, the DTW distance metric has shown promising results for time series clustering problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*cGw893ywleUK_ZZce_kpJg.png" /><figcaption>Sample time series clustering (<a href="https://www.semanticscholar.org/paper/1-Characteristic-based-Clustering-for-Time-Series/0a5e8d1390f3ceb851f4a37a7ec8edb95e05f698">source</a>)</figcaption></figure><h4><a href="https://tslearn.readthedocs.io/en/stable/user_guide/dtw.html#dtw">Dynamic Time Warping (DTW)</a></h4><blockquote>In <a href="https://en.wikipedia.org/wiki/Time_series_analysis">time series analysis</a>, <strong>dynamic time warping</strong> (<strong>DTW</strong>) is an algorithm for measuring similarity between two temporal sequences, which may vary in speed, phase or length.</blockquote><blockquote>It aims at aligning two sequences by warping the time axis iteratively until an optimal match (according to a suitable metrics) between the two sequences is found.</blockquote><p>The warping is accomplished by mapping (one to many and many to one) each index in one time series to the other basis a set of defined <a href="https://www.psb.ugent.be/cbd/papers/gentxwarper/DTWalgorithm.htm">rules</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kt_Vf3XHSIDqX7fi0haYHw.png" /><figcaption><a href="https://www.psb.ugent.be/cbd/papers/gentxwarper/DTWAlgorithm.ppt">Source</a></figcaption></figure><h3>Time Series Clustering: Modelling</h3><h4><strong>Data Pre-processing</strong></h4><p><a href="https://tslearn.readthedocs.io/en/stable/gen_modules/clustering/tslearn.clustering.TimeSeriesKMeans.html#tslearn.clustering.TimeSeriesKMeans">Time Series K-means</a> with DTW distance function to group stocks basis their Adjusted Close prices. The said prices differ vastly in their magnitudes, while for this analysis, we are only concerned with the shape of the rebound trajectory of prices. The stock price data has been scaled to zero mean and unit variance to overcome this. Also, the analysis period has been restricted from 2018 to Nov 2022, along with removing stocks with NAN values.</p><pre>from tslearn.preprocessing import TimeSeriesScalerMeanVariance<br><br>close_prices = stock_prices[&quot;Adj Close&quot;]<br>close_prices.index = pd.to_datetime(stock_prices[\<br>(     &#39;Date&#39;, &#39;Unnamed: 0_level_1&#39;)].values)<br><br>close_prices = close_prices.dropna(axis=1)<br><br>close_prices = close_prices[&quot;2018&quot;:]<br>close_prices.shape<br>#(1060, 479)<br><br>X_train = TimeSeriesScalerMeanVariance().fit_transform(close_prices.T)<br>X_train.shape<br>#(479, 1060, 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*SKQMcdXDPUFFEqfBDPPAaQ.png" /><figcaption>Data before scaling (top) and after(bottom)</figcaption></figure><h4>Will Sectors serve as a good clustering?</h4><p>Because stocks from the same sectors exhibit substantial similarity, one might argue that the Sector information presents an intrinsic notion of clustering. While this argument has some merit, EDA reveals that this clustering cannot capture similarities across sectors, say Health Care and IT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*TCJHzgzeF277mqDrJbtmZA.png" /><figcaption>scaled IT sector (top) vs health(bottom)</figcaption></figure><blockquote><em>The delta in the graph above is the difference in average value between February 2nd 2020 and June 30th 2020.</em></blockquote><p>Another demerit of using sector information as clusters is that some sectors present huge variability in rebound trajectories, for ex: Industrials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0SWkJcsO68OG476hViQsjA.jpeg" /><figcaption>scaled Industrial stock price data</figcaption></figure><h4>Final Clustering</h4><p>Due to high computing requirements, the final clustering has been performed on <a href="https://www.kaggle.com/code/ashish1610dhiman/sp500-time-series-cluster5-10/notebook">kaggle kernel</a>. The optimal number (8) of clusters has been found from the elbow plot of inertia.</p><pre>import pickle<br>from tslearn.clustering import TimeSeriesKMeans<br>from sklearn.metrics import adjusted_mutual_info_score as ami<br>from tqdm.notebook import tqdm<br><br>def ad_clustering(X_train,y_train,k, seed = SEED):<br>    model = TimeSeriesKMeans(n_clusters=k,<br>                          n_init=2,<br>                          metric=&quot;dtw&quot;,<br>                          verbose=True,<br>                          max_iter_barycenter=10,tol=1e-2,<br>                          random_state=seed,n_jobs=-1)<br>    y_pred = model.fit_predict(X_train)<br>    ami1 = ami(labels_true = y_train, labels_pred = y_pred)<br>    return (model,y_pred,ami1)<br><br>result_dict={}<br>for k in tqdm(range(5,20)):<br>    model,y_pred,ami2 = ad_clustering(X_train,y_train,k)<br>    result_dict[k] = (model,y_pred,ami2)<br>    print (f&quot;For k = {k}, ami = {ami2:.5%}&quot;)<br><br>with open(&#39;result_dict.pickle&#39;, &#39;wb&#39;) as handle:<br>    pickle.dump(result_dict, handle, protocol=pickle.HIGHEST_PROTOCOL)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*FFxUpMkvVKif4fUhP9PU9Q.png" /><figcaption>Elbow plot of Inertia | for k=8 we see the first elbow</figcaption></figure><h3>Results</h3><p>The clustering (#clusters = 8), does a pretty good job of finding similar rebound trajectories with maximal variability between the clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*cR-HRt9ri6-21pPSRyt24w.png" /><figcaption>Different clusters follow different rebound trajectories</figcaption></figure><p>The size of different clusters is given as:</p><pre>cross[&quot;predicted_label&quot;].value_counts()<br>#0    153<br>#4     72<br>#5     63<br>#7     58<br>#6     45<br>#2     38<br>#1     34<br>#3     16<br>#Name: predicted_label, dtype: int64<br><br>cross = pd.DataFrame()<br>cross[&quot;sector&quot;] = y_train<br>cross[&quot;sub_sector&quot;] = np.array([get_sub_sector(tick) for tick in close_prices.columns])<br>cross[&quot;predicted_label&quot;] = y_pred1<br><br>res1 = pd.crosstab(cross.sector, cross.predicted_label, margins=True, margins_name=&quot;Total&quot;)<br>res2 = pd.crosstab(cross.sub_sector, cross.predicted_label, margins=True, margins_name=&quot;Total&quot;)</pre><p>The crosstab of the Clustering vs the Sector information is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O1yTnButkQxVElLYfn6MEA.png" /><figcaption>Sector 0 is dominated by Healthcare/IT while Cluster 1 by Consumer Discretionary/Industrials</figcaption></figure><h4>Deep-dive into a few clusters:</h4><h4>Cluster 0 | 153 stocks:</h4><p>Stocks in cluster 0, on average, show a shallow drop at the onset of covid, but by June 2020, they have almost resurfaced to pre-pandemic levels. Also, following Covid, these stocks have shown strong growth and resurgence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IcAzd_P3mfFxGvgTtvlNYA.jpeg" /><figcaption>Cluster 0 regained almost all the hit between Feb 2020 and Jun 2020</figcaption></figure><blockquote>The delta in the graph above is the difference in average value between February 2nd 2020 and June 30th 2020.</blockquote><p>The top sub-sectors in cluster 0 include:</p><ul><li>Health Care Equipment (11 stocks)</li><li>Semiconductors (8 stocks)</li><li>Financial Exchanges &amp; Data (6 stocks)</li><li>Application Software (6 stocks)</li><li>Life Sciences Tools &amp; Services (6 stocks)</li></ul><h4>Cluster 1 | 34 stocks:</h4><p>Because of the economic slowdown accompanying covid, consumer spending and travel took a hit. Consequently, cluster 1, which is dominated by these sectors, has seen a significant drop during covid. Most of the stocks in this cluster continue to suffer from this shock, and with the onset of other economic crises like the Ukraine war, they continue to oscillate below their pre-pandemic price levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p53z0nP_2dnq6izNGudcNw.jpeg" /><figcaption>Cluster 1 suffers from the Covid drop even till now</figcaption></figure><blockquote>The delta in the graph above is the difference in average value between February 2nd 2020 and June 30th 2020.</blockquote><p>The top sub-sectors in the cluster include:</p><ul><li>Airlines (5 stocks)</li><li><strong>Biotechnology (3 stocks)</strong></li><li>Hotels, Resorts &amp; Cruise Lines (3 stocks)</li><li>Office REITs (2 stocks)</li><li>Aerospace &amp; Defense (2 stocks)</li></ul><p>The presence of 3 Biotech stocks seems counter intuitive in this cluster, as I the Biotech companies have been doing good following the healthcare crisis. These 3 stocks are:</p><pre>cluster_1_stocks = cross.predicted_label == 1<br>biotech_stocks = cross.sub_sector == &quot;Biotechnology&quot;<br>final_mask = cluster_1_stocks &amp; biotech_stocks<br>cross.loc[final_mask]</pre><ul><li>Biogen</li><li>Gilead Sciences</li><li>Incyte</li></ul><p>Deeper analysis reveals that these companies were not working on vaccines and each is poised with some intrinsic risks leading to this anomaly.</p><h3>Conclusion</h3><p>The patterns identified from this analysis serve not only to understand the economic downturn during covid but can add orthogonal value to investment strategies.</p><p><em>Feel free to connect me on </em><a href="https://www.linkedin.com/in/ashish1610dhiman/"><em>LinkedIn</em></a><em> if you would like to find out more about my project or experiences. I would also be glad to discuss other ideas to improve this analysis :)</em></p><p><a href="https://ashish1610dhiman.github.io/ashish_portfolio/">https://ashish1610dhiman.github.io/ashish_portfolio/</a></p><p><strong>Code Repo</strong>:</p><p>The implementation and complete results for this analysis can be found on <a href="https://github.com/ashish1610dhiman/time_series_practice/tree/master/notebooks/stocks_clustering">Github</a>.</p><p><strong>References</strong>:</p><p>[1] <a href="https://pypi.org/project/yfinance/#description">https://pypi.org/project/yfinance/#description</a></p><p>[2] <a href="https://tslearn.readthedocs.io/en/stable/user_guide/dtw.html#dtw">https://tslearn.readthedocs.io/en/stable/user_guide/dtw.html#dtw</a></p><p>[3] <a href="https://tslearn.readthedocs.io/en/stable/user_guide/clustering.html">https://tslearn.readthedocs.io/en/stable/user_guide/clustering.html</a></p><p>[4] <a href="https://www.psb.ugent.be/cbd/papers/gentxwarper/DTWalgorithm.htm">https://www.psb.ugent.be/cbd/papers/gentxwarper/DTWalgorithm.htm</a></p><p>[5] <a href="https://towardsdatascience.com/dynamic-time-warping-3933f25fcdd">https://towardsdatascience.com/dynamic-time-warping-3933f25fcdd</a></p><p>Subscribe to DDIntel <a href="https://ddintel.datadriveninvestor.com/">Here</a>.</p><p>Visit our website here: <a href="https://www.datadriveninvestor.com/">https://www.datadriveninvestor.com</a></p><p>Join our network here: <a href="https://datadriveninvestor.com/collaborate">https://datadriveninvestor.com/collaborate</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bd25b8c7a5" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/time-series-clustering-of-stock-behaviour-during-covid-9bd25b8c7a5">Time Series Clustering of stock behaviour during Covid</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>