<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Appala on Medium]]></title>
        <description><![CDATA[Stories by Ravi Appala on Medium]]></description>
        <link>https://medium.com/@appravi18?source=rss-c5efb00de0e0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*i1523ZG-BN_rMS6OYauNbg.png</url>
            <title>Stories by Ravi Appala on Medium</title>
            <link>https://medium.com/@appravi18?source=rss-c5efb00de0e0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:20:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@appravi18/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Inside the ETFs: Time to Rebalance?]]></title>
            <link>https://medium.com/@appravi18/inside-the-etfs-time-to-rebalance-2f0e57e1cedc?source=rss-c5efb00de0e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0e57e1cedc</guid>
            <category><![CDATA[etf]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[rebalancing]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[openbb]]></category>
            <dc:creator><![CDATA[Ravi Appala]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 00:47:18 GMT</pubDate>
            <atom:updated>2025-01-21T00:48:17.130Z</atom:updated>
            <content:encoded><![CDATA[<p>Belated Happy New Year!!!, time to re-balance your portfolio?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*yisIpyETgmCscceNb9pekA.png" /></figure><p>After what seems like a good three (four if you are lucky) weeks of break, going back to work and the usual grinds starts (again). Thankfully instead of just wasting my time before TV (I blame this on Netflix/Amazon) I decided to take a moment to ponder the future of the markets in 2025.</p><p>For the past few years my investment philosophy has been to just buy low cost S&amp;P 500 tracking ETF’s and completely forget about it. This worked really well given how little attention I have paid to it.</p><p>The returns have been surprisingly good, especially last year, but as we all know, those fantastic returns were largely driven by the rise of just a handful of mega-cap stocks. At first, I was fine with this — ignorance is bliss, right? But then, I had a mild existential crisis. Should I keep coasting along, or does it make sense to reassess my portfolio? <br>Spoiler alert: I opted for reassessment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*N_emjYD5aqTViqRskqJy_A.png" /><figcaption>*Top ~20 ETF returns (sorted on 2024 returns)</figcaption></figure><h4>Here is a <a href="https://photos.app.goo.gl/MBMHhRi1E5uLS15K6">link </a>to the full 100 ETF returns.</h4><p>So, here’s a wild thought experiment: What if, in 2025, you could avoid the mega-seven stocks (you know, the ones that everyone is talking about), and still see some decent returns? I’m talking about imagining a scenario where those few high-flying stocks flatline, but the rest of the market keeps performing as it has in the past few years.</p><p>Sounds crazy, right? It probably is. The reality is, if the “magnificent seven” flatline, other correlated stocks (like Palantir, Adobe, or even utility companies) probably wouldn’t be basking in the same level of success. <br>I get it.</p><p>But stick with me here. What I’m not suggesting is some doomsday scenario where tech stocks crash and the entire market crumbles into a heap of despair. What I’m really after is a better portfolio than the standard SPY ETF. No drama, just trying to be a little more strategic.</p><p>Obviously I am bit biased towards Python but the scenario’s like this are when you truly appreciate the power of Python. How would you feel if I told you that I can create the full “implied” year-to-date performance for all ETFs, using the latest weights, but <em>excluding</em> the M7 stocks from 2010 to 2024 — all in just two lines of code?</p><pre>M7 = [&#39;NVDA&#39;,&#39;AAPL&#39;,&#39;MSFT&#39;,&#39;AMZN&#39;,&#39;META&#39;,&#39;TSLA&#39;,&#39;GOOG&#39;,&#39;GOOGL&#39;]<br>t1=pd.DataFrame()<br><br>for i in range(2010,2025): <br>   t1[f&#39;{i}&#39;] = etf_imp_ytd.groupby(by=[&#39;ETF&#39;])<br>                .apply(lambda x: (x[~x[&#39;Symbol&#39;].isin(M7)][&#39;Weights&#39;]*x[i])<br>                .sum())</pre><p>This one little loop basically ignores the weights of M7 stocks in all etfs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PGRvQsaR46E-fuy9VTezNg.png" /><figcaption>ETF returns without M7 stocks</figcaption></figure><p>I know, I know — you’re skeptical. So, of course, I tested this with SPY to double-check the results. I also wondered: How did SPY perform without those M7 stocks in recent years?</p><pre>w = np.where(x[&#39;Symbol&#39;].isin(M7),0,x[&#39;Weights&#39;].values)<br>p = x[2024].values<br>np.nansum(w*p)<br>#output: 16.13352769477476</pre><p>Here is a little code snippet that does exactly that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3un1-PFb77e_7JsEFxp34g.png" /><figcaption>SPY returns without M7 stocks</figcaption></figure><p>To be 100% precise this works only with stock underlying&#39;s, sorry I am not really interested in Fixed Income or Crypto etfs, so I am happy to ignore them for now. But what I really wanted to see was how well the ETF’s would’ve performed if M7 stocks were excluded — and whether that performance was as good as it looked on paper. There’s probably a flaw here.</p><p>The cardinal sin that I have really done in this analysis is that the weights are based on the latest holdings of the ETF’s. The reason some stocks have heavier weightings in the ETF is that they’ve been performing exceptionally well. So, by using the most recent ETF weights, I’m essentially baking in the idea that those stocks will continue to shine i.e. we fell into the “survivor bias” trap here.</p><p>In hindsight, I should’ve looked at each year’s weights and averaged them over time. But hey, I thought using the end-of-2024 weights was good enough for now. Plus, this isn’t the last time I’ll dive into this topic. My next blog will focus on ETF analysis using “clustering,” so stay tuned!</p><p>Keep reading!!!</p><p><strong>References:</strong></p><ol><li>Github link: <a href="https://github.com/appravi18/ETF/blob/main/SPY_Plotly.ipynb">https://github.com/appravi18/ETF/blob/main/SPY_Plotly.ipynb</a></li><li>OpenBB: <a href="https://docs.openbb.co/platform/reference/etf/holdings">https://docs.openbb.co</a></li><li>Special thanks to PiperBatey for ETF Holdings DL code.<br><a href="https://github.com/PiperBatey/holdings_dl">PiperBatey/holdings_dl: A simple Python script that downloads the holdings of one or more ETFs into a csv file.</a></li><li>To all the open source python packages especially Pandas, Numpy, Jupyter &amp; Plotly. I cannot imagine working even a single day without these awesome libraries.</li><li>Top 100 ETF list from WSJ: <a href="https://www.wsj.com/">The Wall Street Journal — Breaking News, Business, Financial &amp; Economic News, World News and Video</a></li><li>ChatGPT: For the complete re-write of the blog to make it a bit more readable.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0e57e1cedc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yield Curve Modeling — Part 2]]></title>
            <link>https://medium.com/@appravi18/yield-curve-modeling-part-2-12820e9839d4?source=rss-c5efb00de0e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/12820e9839d4</guid>
            <category><![CDATA[yield-curve]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[interest-rates]]></category>
            <category><![CDATA[modeling]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ravi Appala]]></dc:creator>
            <pubDate>Mon, 07 Nov 2022 03:15:14 GMT</pubDate>
            <atom:updated>2022-11-07T13:39:49.873Z</atom:updated>
            <content:encoded><![CDATA[<h3>Yield Curve Modeling — Part 2</h3><h3>The Classical Models</h3><p>In the previous blog post, I discussed how we can use PCA analysis to transform the Yield curve into its three main components. We also looked at how we can ‘de-noise’ the yields by running a PCA model and using only the first three components to re-construct the yield curve.</p><p>In this blog, I want to explore other classical Yield curve models that could help us get a better picture/parameter that might be more helpful when combined with other techniques like Kalman Filters. But before we jump into that let&#39;s first take a quick look at the models.</p><p>The three main models I want to cover in this blog are Vasicek, <br>Cox-Ingress-Ross (CIR) and Nelson-Seigel-Svensson (NSS) models.</p><p>Also, I want to be very upfront that the code in this blog is not mine but is taken directly from the “<a href="https://www.coursera.org/learn/financial-engineering-computationalmethods">Computational Methods in Pricing and Model Calibration</a>” course from Coursera. This is a very interesting course and I highly recommend it. This course gives us the code for both Vasicek and CIR models. I made a few tweaks that help us to run this code for the US Treasury Yield curve data we have.</p><p>Here is a quick introduction to the models.</p><p><strong>Vasicek Model<br></strong>This is the first ‘base’ model for interest rates that use the available market yield curve to imply an instantaneous short rate. Here is the actual Vasicek stochastic differential equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*u7PzEBbCmplgj-0386kQPQ.jpeg" /><figcaption>Vasicek Equation</figcaption></figure><p>This model is mean-reverting. Theta is the long-term mean of the short rate and we mean revert to this rate from the current rate (r) with a mean reversion factor of k. If ‘k’ is large, then we get back to the mean quickly while a low ‘k’ value implies a long time to get back to the mean.</p><p>The best part of the Vasicek model in my view is that like the Black-Scholes model there is indeed a closed-form solution to the above PDE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*GblH09mdvRzRFWefG4lZEA.png" /><figcaption>Screenshot from “Interest Rate Models — An Introduction” by Andrew J. G. Cairns</figcaption></figure><p>Here is the code snippet to calculate this.</p><pre>def zero_coupon(tau, r0, kappa, theta, sigma, model):<br>  if model == &#39;Vasicek&#39;:<br>     B = (1- np.exp(-kappa * tau)) / kappa<br>     A = (theta - sigma**2 / (2* kappa**2)) * <br>               (B - tau) - (sigma**2 / (4 * kappa)) * B**2</pre><pre>      r = np.repeat(r0, len(A))<br>      p = np.exp(A - B * r)</pre><pre>  return p</pre><p>With the above closed-form solution, we can use scipy’s optimize module to find the best-fit parameters.</p><pre>def swapRates(tau, p, mat):<br>  tmax = mat[-1]<br>  ttemp = np.arange(0.5, tmax + 0.5, 0.5)<br>  ptemp = np.interp(ttemp, tau, p)<br>  dis = np.cumsum(ptemp)<br>  pmat = np.interp(mat, tau, p)<br>  index = (2 * mat).astype(int) - 1<br>  S = 100 * 2 * (1- pmat)/dis[index]</pre><pre>  return S</pre><pre>def objFunc(params, tau, SWAP, model):<br>  r0 = params[0]<br>  kappa = params[1]<br>  theta = params[2]<br>  sigma = params[3]</pre><pre>  p = zero_coupon(tau, r0, kappa, theta, sigma, model)<br>  S = swapRates(tau, p, SWAP[:,0])</pre><pre>  rel1 = (S - SWAP[:,1])/SWAP[:,1]<br>  mse = np.sum(rel1**2)<br>  return mse</pre><pre>def objFuncMaturityAdjusted(params, tau, SWAP, model):<br>  r0 = params[0]<br>  kappa = params[1]<br>  theta = params[2]<br>  sigma = params[3]</pre><pre>  p = zero_coupon(tau, r0, kappa, theta, sigma, model)<br>  S = swapRates(tau, p, SWAP[:,0])<br>  rel1 = ((S - SWAP[:,1])/SWAP[:,1])*SWAP[:,0]<br>  mse = np.sum(rel1**2)</pre><pre>  return mse</pre><pre>def calibration(fun,params, tau, SWAP, model):<br>  sol = minimize(fun,params, args= (tau,SWAP,model),<br>                 method=&#39;Nelder-Mead&#39;,<br>                 bounds=[[-1,10],[1e-15,10],[1e-15,10], [1e-15,1]],<br>                 options= {&#39;maxiter&#39;:100000,<br>                            &#39;maxfev&#39;:5e3,<br>                            &#39;adaptive&#39;:True})</pre><pre>  par = np.array(sol.x)<br>  return par</pre><p>For the keen code readers, yes I have two objective Functions defined above instead of the usual 1. The first one is what is considered the normal mean squared error based on the Vasicek model we just defined. The problem, as I see it was that all the tenor points got the same weights ie 10 basis points mispricing in 1 year, is the same as 10 basis points mispricing at a 30-year point. I think we should technically penalize more for the 30 year point (ideally by duration), but I used the maturity as a proxy.</p><p>Here is a sample calibration based on 2022–01–03 Yield curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZNg2kpmamgzvkcwDm86kHQ.png" /><figcaption>Vasicek Calibration</figcaption></figure><p>As you can see above the maturity-adjusted calibration seems like a much better fit than the normal one. Obviously, we need to do better, let&#39;s see how other models help with a better fit.</p><p><strong>Cox-Ingersoll-Ross (CIR)Model<br></strong>The second model we are going to look at is the CIR model. This is very similar to the Vasicek but was modified a bit to make sure we don’t end up with negative yields. As it was thought that we could never have negative yields before. This 1985 model certainly shows its age here.</p><p>The PDE for CIR is very similar to Vasicek but we have an additional sqrt of r(t) to make sure we don’t end up with negative short-term rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*CsYFTA6b_LbC5pOZ4-r2Zw.png" /><figcaption>Cox-Ingersoll-Ross Model</figcaption></figure><p>But fortunately, we still have a closed-form solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*x9AEYAGM4yt6mawwsQdv5Q.png" /><figcaption>Slide is taken from “<a href="https://www.coursera.org/learn/financial-engineering-computationalmethods">Computational Methods in Pricing and Model Calibration</a>” course from Coursera</figcaption></figure><p>So, without further ado, here is the comparison of the yield curve models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_HWmaH9ftqJ8qMH_V1xwA.png" /><figcaption>CIR Vasicek models</figcaption></figure><p>Just to be clear, we don’t always get a good fit esp. with simple Vasicek. Here are the model parameters for a random (50th row) from the final data frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SB9AIl3aWxPOXf_MBR6Hmg.png" /><figcaption>Model re-calibrated Yield Curve for final.iloc[50]</figcaption></figure><p>More details on the calibration process can be found in this Jupyter notebook.</p><p>One point to wonder about is that given how the calibration process is it makes us wonder if there is a better model than CIR.</p><p>The final model that I want to go over is the Nelson-Siegel-Svensson model.</p><h4><strong>Nelson-Siegel-Svensson</strong></h4><p>The final model we are going to look at is the Nelson-Siegel-Svensson model. Unlike the Vasicek &amp; CIR models, NSS is a parametric model that can use the Yield curve factors to create a much better fit than the smoothed bootstrap models.</p><p>This model uses four factors, and they can be interpreted as level, slope, first curvature, and second curvature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*4eDAOZlnAtZXXPZp1CySZQ.png" /><figcaption>From “Yield Curve modeling and forecasting the dynamic Nelson-Siegel approach” by Diebold &amp; Glenn</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*uCfwS1dxfeTNVme_DriLPA.png" /><figcaption>Factor-based model by Diebold &amp; Glenn</figcaption></figure><p>This means this model feels much more like PCA but with an additional factor for a better fit.</p><p>Instead of coding this up, we can use an existing <a href="https://github.com/luphord/nelson_siegel_svensson">python library</a>.</p><pre>!pip install nelson-siegel-svensson</pre><pre>from nelson_siegel_svensson.calibrate import calibrate_nss_ols</pre><pre>curve0, status0 = calibrate_nss_ols(tenors_in_years,<br>                                     np.array(tmp3),<br>                                     tau0=[2.0,5.0])</pre><p>This does all the work for us, and we have a nice NelsonSiegelSvennson curve object created by the library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HcUjzsJ7SJZK01fAUSLmVQ.png" /><figcaption>Nelson Siegel Svensson Yield curve fit</figcaption></figure><p>Here is a final comparison of all the “good” models we have till now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FeOLjTy8X5H_QQlqeJfqTA.png" /><figcaption>Yield Curve Modeling</figcaption></figure><p>As you can see, NSS gives us a much better fit than CIR and Vasicek. We will dive deeper and look at using Kalman filters to see how we can use NSS-based yield curve forecast to find trading opportunities and back-test these strategies in my next blog.</p><p>Keep reading !!!</p><p><strong>References:</strong></p><ol><li><a href="https://www.coursera.org/learn/financial-engineering-computationalmethods">Computational Methods in Pricing and Model Calibration</a> by Coursera</li><li><a href="https://github.com/appravi18/PCA/blob/main/YieldCurve_Classical_Models.ipynb">GitHub Python Notebook</a></li><li><a href="https://github.com/luphord/nelson_siegel_svensson">Nelson-Siegel-Svensson Python library</a></li><li><a href="https://www.amazon.com/Interest-Rate-Models-Andrew-Cairns/dp/0691118949">Interest Rate Models — An Introduction by Andrew J.G. Cairns</a></li><li><a href="https://press.princeton.edu/books/hardcover/9780691146805/yield-curve-modeling-and-forecasting">Yield Curve Modeling and Forecasting- The Dynamic Nelson-Siegel approach by Francis X. Diebold &amp; Glenn D. Rudebusch</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12820e9839d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yield Curve Modeling — Part 1]]></title>
            <link>https://medium.com/@appravi18/yield-curve-modeling-part-1-f25cd45ee546?source=rss-c5efb00de0e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f25cd45ee546</guid>
            <category><![CDATA[interest-rates]]></category>
            <category><![CDATA[yield-curve]]></category>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ravi Appala]]></dc:creator>
            <pubDate>Tue, 18 Oct 2022 01:07:21 GMT</pubDate>
            <atom:updated>2022-10-18T01:07:21.095Z</atom:updated>
            <content:encoded><![CDATA[<h3>Yield Curve Modeling — Part 1</h3><p>In this blog post series on Yield curve Modeling, I would like to dig a bit deeper into yield curve analytics using various methods.</p><p>In this first blog, we start the journey with the age-old, tried and tested Principal Component Analysis. We will dig into how we can use PCA to decompose the yield curve into components and reconstruct it back, thus minimizing the noise.</p><p>So, let&#39;s start with the basics.</p><h3><strong>What is a Yield Curve</strong></h3><p>A Yield Curve is simple a curve showing tenors or maturity on the x-axis and the corresponding yield on the y-axis. He is a simple yield curve graph based on the US Treasury rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9F-GntnxJMsDO1WYUloWyQ.png" /><figcaption>US Yield Curve rates from WSJ as of 9th Oct 2022</figcaption></figure><p>Obviously, we cannot do much analysis with just one day’s worth of data. We need a few years worth of data to start our analysis.</p><p>Here is a python code snippet that helps us to download this data from the US Treasury website. The full python notebook is <a href="https://github.com/appravi18/PCA/blob/main/PCA_USD.ipynb">here</a>.</p><pre>#Python code to download data from Treasury</pre><pre>def get_data():<br>  all_data = []</pre><pre>  for i in range(10,23):<br>    year = str(2000+i)<br>    url1 = &#39;<a href="https://home.treasury.gov/resource-center/data-chart-center/interest-rates/daily-treasury-rates.csv/&#39;+year+&#39;/all?type=daily_treasury_yield_curve&amp;field_tdr_date_value=&#39;+year+&#39;page&amp;_format=csv&#39;">https://home.treasury.gov/resource-center/data-chart-center/interest-rates/daily-treasury-rates.csv/&#39;+year+&#39;/all?type=daily_treasury_yield_curve&amp;field_tdr_date_value=&#39;+year+&#39;page&amp;_format=csv&#39;</a><br>    <br>    data = pd.read_csv(url1)<br>    all_data.append(data)</pre><pre>  pd_alldata = pd.concat(all_data,ignore_index=True)<br>  pd_alldata[&#39;Date&#39;] = pd.to_datetime(pd_alldata[&#39;Date&#39;])<br>  pd_alldata.index = pd_alldata[&#39;Date&#39;]<br>   <br>  req_cols = [&#39;3 Mo&#39;, &#39;6 Mo&#39;, &#39;1 Yr&#39;, &#39;2 Yr&#39;, &#39;3 Yr&#39;, &#39;5 Yr&#39;, <br>              &#39;7 Yr&#39;,&#39;10 Yr&#39;, &#39;20 Yr&#39;, &#39;30 Yr&#39;]</pre><pre>  final = pd_alldata[req_cols]</pre><pre>return final</pre><p>This code snippet gets data from 2010 to 2022. Here is the time series data visualized.</p><figure><img alt="US Yield Curve Time Series" src="https://cdn-images-1.medium.com/max/1024/1*VG_IKDaERvB1B5MgYKi_-g.png" /><figcaption>US Yields Time Series from Treasury</figcaption></figure><p>Before we jump into the actual PCA analysis, we need to understand why we need to do this in the first place. In today’s world where we have lots of data, it is very tough to figure out what components, columns, and features are required to efficiently understand the drivers and more importantly what we can ignore to predict the future.</p><p>On the one hand, we want to use only the relevant columns and features but on the other hand, we don’t want to throw away data that we worked hard to collect. What would be ideal in this case would be to use some kind of data transformation that would combine a lot of columns into a few. This technique is called <strong>Dimensionality reduction</strong>.</p><p><strong>Side Note: </strong>An interesting fact about Principal Component Analysis and a testament to its usefulness is actually how old the technique is. How many of you would be surprised to hear that the paper on PCA was published in <strong>1901? </strong>Even though this technique was first invented over 120 years ago the fact that we are still discussing and using it shows how important it is.</p><p>Now that we know what PCA is supposed to do I think one last example of what it actually does would give an intuitive understanding of this process.</p><p>Imagine you have a table that has a bunch of X components and Y components so basically a two-dimensional vector and if somebody asked you to plot them on a (one-dimensional) line, how would we do this?<br>Esp. while keeping the whole structure intact. A quick way of doing this is to calculate the distance between the points and map it accordingly on the one-dimension line. This way we are kind of capturing both the X and Y component but at the same time reducing the dimensionality to one from two dimensions.</p><p>Dist(z) = sqrt(x² + y²)</p><p>Intuitively PCA is doing exactly this. Trying to come up with a linear combination of columns that best captures the data i.e. explains the variance in the data.</p><p>Enough theory let&#39;s see how PCA works in practice.</p><p>Here is the code to create the PCA components using Scikit Learn.</p><pre>from sklearn.decomposition import PCA<br>from sklearn.preprocessing import scale, StandardScaler</pre><pre>#Step1: We calculate the mean of the <strong><em>scaled</em></strong> data.<br>df_normalized_mean = pd.DataFrame(scale(final))</pre><pre>#Step2: Remove any missing data points<br>df_normalized_mean.dropna(inplace=True)</pre><pre>instruments = final.columns</pre><pre>#Step3: We calculate the PCA using fit_transform<br>pca = PCA(n_components=len(instruments))<br>YC_PCA = pca.fit_transform(df_normalized_mean)</pre><pre>#Step4: Create a SCREE plot to check the weights of each component<br>per_var = np.round(pca.explained_variance_ratio_ * 100, decimals=1)<br>labels = [&#39;PC&#39; + str(x) for x in range(1, len(per_var)+1)]</pre><pre>plt.bar(x=range(1, len(per_var)+1), height=per_var, tick_label=labels)</pre><pre>plt.ylabel(&#39;Percentage of Explained Variance&#39;)<br>plt.xlabel(&#39;Principal Component&#39;)<br>plt.title(&#39;Scree Plot&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*M0ikpssLKkcLFK-cpFs1OA.png" /><figcaption>PCA Scree Plot</figcaption></figure><pre>#Here we look at the cumulative variance explained<br>plt.plot(labels, pca.explained_variance_ratio_.cumsum()*100)<br>plt.ylabel(&#39;Percentage of Explained Variance&#39;)<br>plt.xlabel(&#39;Principal Component&#39;)<br>plt.title(&#39;Cumulative Explained Variance&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*ZPqGOV4lLw0H9kRmQmSZ5A.png" /><figcaption>PCA Cumulative Variance</figcaption></figure><p>Here are the PCA components for each tenor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*DQxdhbgWc9GIqGFVVSiiDg.png" /><figcaption>PCA Components by Tenor</figcaption></figure><p>Instead of me coming up with an explanation of each component, here is an extract from Credit Suisse PCA paper.</p><ul><li>The <strong>first component</strong> has positive coefficients for all the variables: it can be seen as a proxy for the level of yields and explains most of the dispersion.</li><li>The <strong>second one</strong> has negative factors for the short end that become more neutral at the belly and then positive at the long end of the curve. It represents the slope of the swap curve: the short end is anti-correlated to the long end from this angle.</li><li>The <strong>third component</strong> has positive coefficients for the short- and long-ends, while it has negative coefficients in the belly of the curve. This indicates that the axis distinguishes the relative impact on a point from a curvature effect.</li></ul><p>Here is all of them together in one graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*gOqBw2tzzdTg2i97UjeIaA.png" /><figcaption>PCA components</figcaption></figure><p>Residuals i.e. differences between the original data and reconstructed data (using the three main PCA components) can be used to create new trading strategies. Please see this <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwji59XM59b6AhXolYkEHYqrBG8QFnoECB4QAQ&amp;url=https%3A%2F%2Fplus.credit-suisse.com%2Fr%2Fkv66a7&amp;usg=AOvVaw24qoCgEj2rS7KhwI7gItpV">paper</a> for more details.</p><p>Here is the code to re-construct the yields purely based on the first three components. The reason we do this, is to de-noise the yield curve.</p><pre>reConstructedData = pd.DataFrame(data=scaler.inverse_transform(np.dot(pca.transform(rescaledDataset)[:,:NumComponents],pca.components_[:NumComponents,:])),columns=final.columns,index = final.index)</pre><p>Here are the yields based on the re-constructed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WFT1b_xo7D4iObhoLun0IA.png" /><figcaption>Re-Constructed Yields</figcaption></figure><p>Here are the residuals</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yff1vXJRfKCM0BJLrVcreQ.png" /></figure><p>Here are the same residuals but for main 2yr, 5yr &amp; 10yr tenors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlxXqcxkpzP9gxEDPpcoDQ.png" /></figure><p>As you can see, PCA can be quite helpful to find some interesting trading ideas. In the next blog post, I want to get into other classical interest rate modeling techniques that we help us find curve trades like 2–5’s or 2–5–10 and other curve trades. To do this correctly we need a better model to fit and produce a full curve instead of the usual points we have. More details in the next blog.</p><p>Keep Reading !!!</p><p><strong>References:</strong></p><ol><li><a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwjW78bA6tb6AhXkkIkEHRa-BjMQFnoECB4QAQ&amp;url=https%3A%2F%2Fplus.credit-suisse.com%2Fr%2Fkv66a7&amp;usg=AOvVaw24qoCgEj2rS7KhwI7gItpV">PCA Unleashed — Credit Suisse | PLUS</a></li><li><a href="https://github.com/appravi18/PCA/blob/main/PCA_USD.ipynb">GitHub Python Notebook</a></li><li><a href="https://youtu.be/FgakZw6K1QQ">StatQuest: Principal Component Analysis (PCA), Step-by-Step</a></li><li><a href="https://www.wsj.com/market-data/bonds?mod=nav_top_subsection">WSJ Bonds &amp; Rates</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f25cd45ee546" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which Moving Average???]]></title>
            <link>https://medium.com/@appravi18/which-moving-average-eeb93c246670?source=rss-c5efb00de0e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/eeb93c246670</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[kalman-filter]]></category>
            <category><![CDATA[stock-market]]></category>
            <dc:creator><![CDATA[Ravi Appala]]></dc:creator>
            <pubDate>Fri, 27 Aug 2021 00:29:13 GMT</pubDate>
            <atom:updated>2021-08-27T00:32:03.641Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1f7d1inOagF0BsPa.jpg" /></figure><p>So you are looking at a simple trading strategy and want to buy the dips and sell when the stock is above certain moving average. Sounds simple and I am sure we can get an amazing looking back testing chart that shows how we could be billionaires by following this simple strategy.</p><p>Alas, only if life was this easy. So the big question now, how many number of days is this moving average going to be based on? Is this looking back 30days, 60days, 90days? With a little bit of patience and persistence we might even be able to figure out what worked the best.</p><p>Btw here is a quick code sample to do this in Python/Pandas.</p><pre>#Standard imports<br>import pandas as pd<br>import pandas_datareader as pdr<br>import matplotlib.pyplot as plt</pre><p>We use a library called pykalman to run our Kalman filters process.</p><pre># !pip install pykalman<br>from pykalman import KalmanFilter</pre><p>Lets just get Google’s stock and look at the 30d,60d &amp; 90d moving average</p><pre>#using DataReader to get Google stock prices<br>df = pdr.DataReader(<br>                    &#39;GOOG&#39;, &#39;yahoo&#39;, <br>                     start=&#39;2010-08-25&#39;,<br>                      end=&#39;2021-08-25&#39;)</pre><p>Once we have the stock prices, we can use Pandas build in rolling-functions to get the moving averages.</p><pre># Moving Averages using rolling mean, lookback windows<br>mean30 = x.rolling(window = 30).mean()<br>mean60 = x.rolling(window = 60).mean()<br>mean90 = x.rolling(window = 90).mean()</pre><p>And we finally plot this.</p><pre>plt.plot(x)<br>plt.plot(mean30)<br>plt.plot(mean60)<br>plt.plot(mean90)<br>plt.legend([&#39;Goog&#39;,<br>            &#39;30-day Moving Average&#39;,<br>            &#39;60-day Moving Average&#39;,<br>            &#39;90-day Moving Average&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*cGU_x80XzBJOonJCph6t1Q.png" /><figcaption>Google stock price with moving averages</figcaption></figure><p>Lets zoom in to see how Google stock did in last year.</p><pre>#Zooming into last years data<br>x1 = df[&#39;Adj Close&#39;][&#39;2020-08-21&#39;:]</pre><pre>x1_mean30 = x1.rolling(window = 30).mean()<br>x1_mean60 = x1.rolling(window = 60).mean()<br>x1_mean90 = x1.rolling(window = 90).mean()</pre><pre>plt.plot(x1)<br>plt.plot(x1_mean30)<br>plt.plot(x1_mean60)<br>plt.plot(x1_mean90)<br>plt.legend([&#39;Goog&#39;, <br>            &#39;30-day Moving Average&#39;,<br>            &#39;60-day Moving Average&#39;,<br>            &#39;90-day Moving Average&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*F5kXtRPi0XModmxknSio5A.png" /><figcaption>Google stock prices last year</figcaption></figure><p><strong>A side note:</strong> Why did I think Google and Tech in general was overpriced in 2020? What exactly was I thinking? Is this still overpriced? How about 6months ago, how about 3months ago?</p><p>Lets just say, I remind myself of this great quote by <strong>Oscar Wilde</strong>.</p><p>“ I can resist anything except temptation”</p><p>I will follow my professor <a href="https://pages.stern.nyu.edu/~adamodar/">Aswath Damodaran</a> and will be a “Lemming” with a life vest.</p><p>Now back to the moving averages, so which one is the best. How would I choose it? Also what makes me think any back testing I do to come up with a lookback window would hold for the future?</p><p>Well, as you might have already guessed it, my answer is none of the above. I want to use some other Bayesian algorithm that takes daily data and adjusts this “Special” moving average.</p><p>So without further ado, that is exactly what Kalman filter does for us. It starts with a random number and as it gets more observables it updates quickly and really catches up with the stock prices.</p><p>Here is the code for that.</p><pre>kf = KalmanFilter(transition_matrices = [1],<br>                  observation_matrices = [1],<br>                  initial_state_mean = 0,<br>                  initial_state_covariance = 1,<br>                  observation_covariance=1,<br>                  transition_covariance=0.01)</pre><pre># Use the observed values to get a &#39;implied&#39; rolling mean<br>state_means, _ = kf.filter(x1.values)<br>state_means = pd.Series(state_means.flatten(), index=x1.index)</pre><p>Once this is done, we get this wonderful chat with the “Kalman Average”.</p><pre>plt.plot(x1)<br>plt.plot(x1_mean30)<br>plt.plot(x1_mean60)<br>plt.plot(x1_mean90)<br>plt.plot(state_means)<br>plt.legend([&#39;Goog&#39;, <br>            &#39;30-day Moving Average&#39;,<br>            &#39;60-day Moving Average&#39;,<br>            &#39;90-day Moving Average&#39;,<br>            &#39;Kalman Average&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*8tSrIPy_ZaBHHF0QZNZW6A.png" /><figcaption>Kalman Moving Average- Google</figcaption></figure><p>As you can see the Kalman Average (brown line) starts off at a random point but quickly catches up to the stock price. This line tracks the price much better than any of the moving averages and the best part, I didn’t have to specify how many days to lookback, it figures that out on its own.</p><p>That&#39;s all for today. Happy reading !!!</p><p>References:</p><ul><li><a href="https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python">GitHub - rlabbe/Kalman-and-Bayesian-Filters-in-Python: Kalman Filter book using Jupyter Notebook. Focuses on building intuition and experience, not formal proofs. Includes Kalman filters,extended Kalman filters, unscented Kalman filters, particle filters, and more. All exercises include solutions.</a></li><li><a href="https://quantopian-archive.netlify.app/notebooks/notebooks/quantopian_notebook_49.html">NbPreview</a></li></ul><p>Github:</p><p><a href="https://github.com/appravi18/Kalman/blob/main/Kalman_Moving_Average.ipynb">Kalman/Kalman_Moving_Average.ipynb at main · appravi18/Kalman (github.com)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eeb93c246670" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Analysis with Classic Statistical Methods]]></title>
            <link>https://medium.com/@appravi18/time-series-analysis-with-classic-statistical-methods-fe6dfaf0cd4a?source=rss-c5efb00de0e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe6dfaf0cd4a</guid>
            <category><![CDATA[statsmodels]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[sarimax]]></category>
            <category><![CDATA[arima]]></category>
            <dc:creator><![CDATA[Ravi Appala]]></dc:creator>
            <pubDate>Fri, 20 Aug 2021 04:00:38 GMT</pubDate>
            <atom:updated>2021-08-20T04:00:38.180Z</atom:updated>
            <content:encoded><![CDATA[<p>Time series Analysis using Stats Models Arima and Sarimax in python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Fu7ad1RYGsu5UlGY.png" /></figure><p>In this blog we look at how to do Time Series analysis using classical statistical methods, specifically using Stas Models Arima and Sarimax models.</p><p>As we have seen in my previous <a href="https://medium.com/@appravi18/time-series-analysis-c4c60853d5ef">blog</a>, most Time Series Data can be decomposed into Trend, Seasonality and residual. Today we will look into how to use this and other functions to predict future.</p><p>Before we jump in here are a few terms and how I see them.</p><p><a href="https://otexts.com/fpp2/AR.html">Auto Regression (AR):</a> <br>We usually write regression problems as a linear combination of factors, <br> y = m x + b, this would be a how we would write a regression problem.</p><p>Now imagine instead of y being dependent on x, its dependent on y(t-n) <br>i.e., y at time t is based on y at time t-1 and y at t-2 etc. giving us the below formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*vRbpx0Ymg4fEIix0WoySOA.png" /><figcaption><a href="https://otexts.com/fpp2/AR.html">https://otexts.com/fpp2/AR.html</a></figcaption></figure><p><a href="https://otexts.com/fpp2/MA.html">Moving Average (MA): </a><br>A moving average model is based on past forecast errors instead of the past values, giving us the below formula for prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*9L7AAxpdvZ_eKMaM_8JE3Q.png" /><figcaption><a href="https://otexts.com/fpp2/MA.html">https://otexts.com/fpp2/MA.html</a></figcaption></figure><p>Obviously a better model would be one that combines both of them together. As you might have guessed that is the ARIMA model.</p><p><a href="https://otexts.com/fpp2/non-seasonal-arima.html"><strong>Arima Model (p,d,q)</strong></a><strong>: </strong>This model uses both the Auto Regressive (AR) and Moving Average (MA) models with differencing (d).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*bVIykD6Eu_kBE7ehd4Jlyw.png" /><figcaption><a href="https://otexts.com/fpp2/non-seasonal-arima.html">https://otexts.com/fpp2/non-seasonal-arima.html</a></figcaption></figure><p><a href="https://otexts.com/fpp2/seasonal-arima.html"><strong>Sarimax Model (P,D,Q,m)</strong></a><strong>: </strong>Seasonality is included in the Sarimax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*e6PSWkEZxtudUunR5yVxew.png" /><figcaption><a href="https://otexts.com/fpp2/seasonal-arima.html">https://otexts.com/fpp2/seasonal-arima.html</a></figcaption></figure><p>All this theory is wonderful but I prefer to see how its done in an actual notebook, esp. how does one get these values. The theoretical way of doing this is by looking at the autocorrelation (acf) and partial autocorrelation(pacf) graphs.</p><p>Here is an excellent article going into the details and specifying various rules. <br><a href="https://people.duke.edu/~rnau/411arim3.htm">Identifying the orders of AR and MA terms in an ARIMA model (duke.edu)</a></p><p><strong>Auto Arima: To get the actual p,d,q &amp; P,D,Q,s<br></strong>Now coming to the practical aspects of this, we just use pyramid-arima package that uses (a clever grid search type) algo to give us the best fit.<br> <br>PMDArima: <a href="https://alkaline-ml.com/pmdarima/index.html">https://alkaline-ml.com/pmdarima/index.html</a></p><p>Here are the steps to run prediction on a Time Series with seasonality.</p><pre>#Standard Imports<br>from statsmodels.tsa.seasonal import seasonal_decompose<br>from pmdarima import auto_arima<br>from statsmodels.tsa.arima_model import ARIMA<br>from statsmodels.tsa.statespace.sarimax import SARIMAX</pre><pre>import pandas as pd</pre><pre>import matplotlib.pyplot as plt<br>%matplotlib inline<br>plt.style.use(&#39;fivethirtyeight&#39;)</pre><p>I used the standard ‘airline_passengers’ to see how the forecasting works.</p><pre>df = pd.read_csv(&#39;airline_passengers.csv&#39;,index_col=&#39;Month&#39;)<br>df.index.freq = &#39;MS&#39;</pre><p>Now that the initial setup and data reading is done, we just run the auto arima algo to give us the best fit.</p><pre>am = auto_arima(<br>             df[&#39;Thousands of Passengers&#39;],<br>             seasonal=True, #our data is of course seasonal<br>             m=12,#this parameters sets cycle length as 12 months<br>             trace=True #set this param to see what values are tried<br>             )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*osIuu8uWClJOOzSKENTWnQ.png" /><figcaption>Auto Arima trails to find the best parameters</figcaption></figure><p>am.summary() gives us the details of the best fit model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*Ji3Qml5wCBUtApRVmsVwng.png" /><figcaption>Best fit model params (2,1,1) x (0,1,0,12)</figcaption></figure><p>Now we create a new SARIMAX model with the best fit params.</p><pre># after the model is created, we split the dataframe <br># into train and test sets</pre><pre>train = df[:120]<br>test = df[120:<br>]<br>model = SARIMAX(train,<br>                order=(2,1,1),<br>                seasonal_order=(0,1,0,12)<br>               )</pre><pre>results = model.fit()</pre><p>The model is now ready to run some predictions</p><pre>#run predictions now<br>predictions = results.predict(<br>                start=len(train),<br>                end=len(train)+len(test)-1,<br>                typ=&#39;levels&#39;).rename(&#39;SARIMA Predictions&#39;)</pre><pre>test.plot(figsize=(15,8),legend=True)<br>predictions.plot(legend=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*vnILWHYOwxbC6CAvuiZ0aA.png" /></figure><p>Can we improve on this further? Lets try to see if a first order difference makes this any better.</p><pre>#Trying to predict the change in Passengers from previous year</pre><pre>from statsmodels.tsa.statespace.tools import diff<br>df_diff = pd.DataFrame(<br>                 diff(<br>                      df[&#39;Thousands of Passengers&#39;],<br>                      k_diff=1,<br>                      seasonal_periods=12<br>                )<br>          )</pre><p>Now we run the same exact process as above but on the df_diff dataframe.</p><pre>am_diff = auto_arima(<br>                     df_diff[&#39;Thousands of Passengers&#39;],<br>                     seasonal=True,<br>                     m=12,<br>                     trace=True,<br>                     stationary=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*iw-U2JOApg_hQOhqEbWimA.png" /><figcaption>Best fix params for DF_Diff</figcaption></figure><pre>#train/test split for the df_diff<br>train_diff = df_diff[:120]<br>test_diff = df_diff[120:]</pre><pre>#New Sarimax model<br>model_diff = SARIMAX(<br>                     train_diff[&#39;Thousands of Passengers&#39;],<br>                     order=(2,0,1),<br>                     seasonal_order=(1,0,0,12))</pre><pre>results_diff = model_diff.fit()</pre><pre>#run predictions now<br>predictions_diff = results_diff.predict(<br>                start=len(train_diff),<br>                end=len(train_diff)+len(test_diff)-1,<br>                typ=&#39;levels&#39;).rename(&#39;SARIMA Predictions -- Diff&#39;)</pre><pre>#plot the diffs<br>test_diff.plot(figsize=(15,8),legend=True)<br>predictions_diff.plot(legend=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*P2mxuxprqm6roLE1Hrdyjg.png" /></figure><h3>Evaluate the Models</h3><pre>from sklearn.metrics import mean_squared_error<br>from math import sqrt</pre><pre>error = mean_squared_error(<br>                           test[&#39;Thousands of Passengers&#39;],<br>                           predictions)<br>print(f&#39;SARIMAX: RMSE Error: {sqrt(error):5.5}&#39;)</pre><p>We get a RMSE of 75.436</p><pre>error_diff = mean_squared_error(<br>                               test_diff[&#39;Thousands of Passengers&#39;], <br>                               predictions_diff)<br>print(f&#39;SARIMAX-Diff: RMSE Error: {sqrt(error_diff):5.5}&#39;)</pre><p>For the SARIMAX-Diff model we get a RMSE of 18.891.</p><p><strong>Git Hub:</strong></p><p><a href="https://github.com/appravi18/TimeSeriesAnalysis_Intro/blob/main/Arima_Sarimax.ipynb">TimeSeriesAnalysis_Intro/Arima_Sarimax.ipynb at main · appravi18/TimeSeriesAnalysis_Intro (github.com)</a></p><p><strong>References:</strong></p><p>Forecasting-Principles and Practice: <a href="https://otexts.com/fpp2/">https://otexts.com/fpp2/</a><br>StatsModels:<strong> </strong><a href="https://www.statsmodels.org/stable/index.html">https://www.statsmodels.org/stable/index.html</a><br>PMDArima: <a href="https://alkaline-ml.com/pmdarima/index.html">https://alkaline-ml.com/pmdarima/index.html</a></p><p>Pandas: <a href="https://pandas.pydata.org/">https://pandas.pydata.org/</a><br>Pandas Data Reader: <a href="https://pandas-datareader.readthedocs.io/en/latest/">https://pandas-datareader.readthedocs.io/en/latest/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe6dfaf0cd4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Analysis]]></title>
            <link>https://medium.com/@appravi18/time-series-analysis-c4c60853d5ef?source=rss-c5efb00de0e0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4c60853d5ef</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[seasonal-decompose]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[statsmodels]]></category>
            <dc:creator><![CDATA[Ravi Appala]]></dc:creator>
            <pubDate>Wed, 18 Aug 2021 01:59:32 GMT</pubDate>
            <atom:updated>2021-08-18T11:41:41.884Z</atom:updated>
            <content:encoded><![CDATA[<p>An introduction to time series analysis using python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*S5BbgZrihIStKrHe.png" /></figure><p>In this blog we look at basic time series and try to decompose this and check various parts. Also I would like to use this blog for a gentle introduction to two of what I consider really useful python libraries, Pandas and statsmodels.</p><p><strong>Time Series Data</strong></p><p>Time series data is any data where one of the columns, usually the index, is a time stamp. This could be a daily time stamp or an <a href="https://www.epochconverter.com/">epoch</a> for intraday time series. The big difference between time series data and other data is well the ordering. The time series data is by definition sequential and hence we can run various analysis that shows different patterns.</p><pre>#Here is an example of Time Series Data, a Google stock price chart</pre><pre>import pandas as pd<br>import pandas_datareader as pdr</pre><pre>df = pdr.DataReader(&#39;GOOG&#39;, &#39;yahoo&#39;, start=&#39;2019-09-10&#39;,<br>                    end=&#39;2021-08-17&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*dgKEx2QXyV65ojFIverKqw.png" /><figcaption>Google stock prices</figcaption></figure><p>And the obligatory chart showing how much you money you could have made by just buying Google in 2020!!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*5SoFjEC0D55RhHBN7Y5UZA.png" /><figcaption>Google Stock price — — Did you buy Google in 2020?</figcaption></figure><p>But before you jump on the Google bandwagon, at least in August 2021 lets see if there is anything else, may be with tons of leverage, how does 4x leverage sound? And on top of this, something that is what we in investment banking call “Accrual accounted position” or “Hold to Collect”.</p><p>I know this sounds too good to be true but thanks to Uncle Sam we can still do this by buying a house. You typically would put in only 20% upfront and there is no mark-to-market or margin calls if your house value drops.</p><p>Coming back to TSA, I use Pandas Data Reader library to download the US Housing Index from econdb.</p><pre>HOUUS = pdr.DataReader(&#39;ticker=HOUUS&#39;,&#39;econdb&#39;)</pre><pre>#A little bit of data wrangling</pre><pre>pdf_vals = pd.DataFrame(HOUUS[&#39;traditional&#39;,<br>                        &#39;purchase-only&#39;,&#39;sa&#39;].values)</pre><pre>pdf_Date = pd.DataFrame(HOUUS[&#39;traditional&#39;,<br>                        &#39;purchase-only&#39;,&#39;sa&#39;].index)</pre><pre>pdf_House = pd.concat([pdf_Date,pdf_vals],axis=1)<br>pdf_House.columns = [&#39;Date&#39;,&#39;US_House_Index&#39;]</pre><pre>pdf_House.index = pdf_House[&#39;Date&#39;]<br>pdf_House.drop(columns=&#39;Date&#39;,inplace=True)</pre><p>The above code does a little bit of data wrangling to get the details in this nice format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*tRThSD9ir6FeLeo8KpvKAQ.png" /></figure><p>Here is the last few step before we can run time series decomposition.</p><pre>pdf_House = pdf_House.resample(&#39;m&#39;).ffill()</pre><p>We use Statsmodels seasonal_decompose method to create the below chart.</p><pre>from statsmodels.tsa.seasonal import seasonal_decompose</pre><pre>res = seasonal_decompose(pdf_House[&#39;US_House_Index&#39;],   <br>                         model=&#39;additive&#39;)</pre><pre>plt.rcParams[&#39;figure.figsize&#39;] = (10, 8)<br>x = res.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*XqY8G9_DCoSHCntzCMlhGg.png" /></figure><p>The Statsmodels <a href="https://www.statsmodels.org/stable/generated/statsmodels.tsa.seasonal.seasonal_decompose.html#statsmodels.tsa.seasonal.seasonal_decompose">Seasonal_Decompose</a> nicely decomposes our Time Series into its three components.</p><p><strong>Trend</strong>: This is the general trend of how our values are charging over time.</p><p><strong>Seasonal</strong>: This gives us the effect of seasonality in our data.</p><p><strong>Resid</strong>: This is the residual value left after accounting for the trend and the seasonality in our data.</p><p>In future blogs we will get into more details and see how to use various features to help with our Time Series Analysis esp. ACF, PACF to check for autocorrelation and stationarity.</p><p><strong>Git Hub : </strong><a href="https://github.com/appravi18/TimeSeriesAnalysis_Intro/blob/main/TSA_Intro.ipynb">https://github.com/appravi18/TimeSeriesAnalysis_Intro/blob/main/TSA_Intro.ipynb</a></p><p><strong>References:<br></strong>StatsModels:<strong> </strong><a href="https://www.statsmodels.org/stable/index.html">https://www.statsmodels.org/stable/index.html</a><br>Pandas: <a href="https://pandas.pydata.org/">https://pandas.pydata.org/</a><br>Pandas Data Reader: <a href="https://pandas-datareader.readthedocs.io/en/latest/">https://pandas-datareader.readthedocs.io/en/latest/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4c60853d5ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>