<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ishan Shah on Medium]]></title>
        <description><![CDATA[Stories by Ishan Shah on Medium]]></description>
        <link>https://medium.com/@ishan.s_54240?source=rss-baef4b7681f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*rEY97aqrH_2elYldsTB3jg.jpeg</url>
            <title>Stories by Ishan Shah on Medium</title>
            <link>https://medium.com/@ishan.s_54240?source=rss-baef4b7681f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:52:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ishan.s_54240/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Pandemics Impact Financial Markets]]></title>
            <link>https://medium.com/data-science/pandemics-impact-financial-markets-9a4feb6951f5?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a4feb6951f5</guid>
            <category><![CDATA[pandemic]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Thu, 09 Apr 2020 02:53:50 GMT</pubDate>
            <atom:updated>2020-09-15T20:20:37.831Z</atom:updated>
            <content:encoded><![CDATA[<h4>Take cues from the pandemics in the past and quantitatively analyze the impact on financial markets. Visualize and analyze the data in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjC09khPc0rJPaUoN-jNlg.png" /></figure><p>We are under an uncertain environment which is also reflected in the financial markets. You would have many questions such as how COVID19 would impact financial markets, how much more will it fall, when it will end and how will it end. In this article, we will analyse and take cues from past pandemics to answer these questions and what we can expect in the future. <em>This article is written along the lines of the webinar conducted by QuantInsti on this topic on 2 April 2020. The recording and the codes can be accessed from the below.</em></p><p><a href="https://blog.quantinsti.com/pandemics-impact-financial-markets-webinar-2-april-2020/">How Pandemics Impact the Financial Markets - A Quantitative Analysis</a></p><p>Before we start our deep dive into impact on the markets, below are the news which we are seeing all around us or are expected.</p><ul><li>Business closures and manufacturing plant shutdowns</li><li>Job losses and an increase in the unemployment rate</li><li>Supply chain disruption</li><li>Shortages of daily essentials</li><li>Pressure on healthcare systems</li><li>Exponentially rising infection and death tolls</li><li>And many more……</li></ul><p>You have seen that shops are closed and the manufacturing plants are shut down. No one is producing anything and they don’t have a salary to pay so they are laying off people and thus, unemployment has increased. There is a disruption in the supply chain, and people are hoarding daily essentials including toilet papers. This will result in shortages. The number of cases related to infections is rising which is causing tremendous pressure on healthcare systems. The developed markets are coping somewhat with this unprecedented situation, but the developing markets such as India will find it very difficult to manage pressure on the healthcare system.</p><p>Combined with these are steep falls in financial markets. In these circumstances, the question which might come to you is when can we expect some good news, when will markets recover and many more questions.</p><h3>COVID19 Impact on Financial Markets</h3><blockquote>Risk velocity is elevated in today’s markets compared to 10 years ago. The social-media driven news cycle, the interconnectedness of global supply chains and a pricey stock market, make Wall Street more vulnerable to a black swan.</blockquote><blockquote>~ Seema Shah, chief strategist at Principal Global Investors</blockquote><p>Let’s begin by assessing the current impact on the financial markets due to COVID19. I have used Python freely available APIs and code to create all the charts in this article. All these codes are available on <a href="https://github.com/QuantInsti/webinars/tree/master/Impact%20of%20Pandemics%20on%20Financial%20Markets">GitHub</a>.</p><h4>If you are new to Python and haven’t used financial libraries such as NumPy and Pandas, then I would recommend you to brush up the concepts in the below free interactive course on Python for Trading.</h4><p><a href="https://quantra.quantinsti.com/course/python-trading-basic">Python for Trading Basic Free Course</a></p><p>Let me walk you through this piece of code which is used to plot the daily percentage change for the S&amp;P 500.</p><p>The code is given below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5892c95785bf10cd0615ac3a74b02b48/href">https://medium.com/media/5892c95785bf10cd0615ac3a74b02b48/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*NFaUhK8QQ8xofuaKlclUwg.png" /></figure><p>I have used pandas_datareader’s get_data_yahoo method to fetch the prices of S&amp;P500.</p><p>The get_data_yahoo method takes two arguments, first is the ticker which is “^GPSC” for S&amp;P 500 in Yahoo Finance, and the second is the date from which I want the data for. If you have checked, the first case of coronavirus was identified on 17th November. Therefore, I have used the same date.</p><p>When I run this code, I will get the dataset with six columns which are open, high, low, close, volume and adjusted close. Since we are interested in daily percentage change, I will calculate it by using pct_change() function of Python and it is called on the close column. For example, if the daily percentage change is changing from one hundred to hundred and two, then I will have a value of 0.02 in this data frame.</p><p>If you want to plot this series in Python, you simply have to pass the name of the series which in this case is “data_pc” and call the function plot and it will plot the data series. So this is how the daily return will look. Since it looks smaller, I have added a grid and done a few improvements to enhance its readability.</p><h4>Daily Percentage Change</h4><p>Apart from the S&amp;P500, I have also plotted the graphs for crude oil, Gold and TLT, which is the 20 years Treasury in the US markets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e408938e0bc0004345960d5c32c4b96/href">https://medium.com/media/5e408938e0bc0004345960d5c32c4b96/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*fGs9G40mvASttXKf" /></figure><p>I have plotted the daily percentage change for each of them. To give some more detail to these graphs, the blue line is the date on which the first case was identified and the red line is the date on which WHO declared a public health emergency.</p><p>So we can see the volatility or the change in the price on a daily basis for S&amp;P500 was very close to zero and we can see that a few days after the WHO declared emergency, there was a sharp change in its volatility. It is ranging from 5 to 10% in daily moments. Also, it is not just the S&amp;P500 which is a barometer for the US market, but it can also be seen in crude oil, gold and our TLT, which is 20 years Treasury bonds of US market. Thus, we can easily conclude that Corona is making the financial markets jittery.</p><p>Another way to look at these graphs would be to say there are a lot of down movements but there are a lot of up movements also. Thus, <strong>how do I actually assess how the markets would move?</strong></p><h4>Cumulative Returns</h4><p>For this, I have plotted the cumulative returns of the four cases. The start date is a few days before the first case was reported and the last day is 30 March.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3df9e2713421c96f1e9b48d48ec9c2f0/href">https://medium.com/media/3df9e2713421c96f1e9b48d48ec9c2f0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*356ANaJJgnRTNmHd" /></figure><p>As you can see the red line is for crude oil and it seems like it has been impacted the most. It’s a decline of almost 60% from the top. The blue line indicating the S&amp;P 500 has also shown a sharp decline. But if you check gold, you would realise that the impact is not as significant as the S&amp;P500 and Crude oil.</p><p><strong>Does that mean we should include gold in our portfolio as a reasonable hedge in these uncertain times?</strong></p><p>It seems like it but let us not jump to conclusions so fast. We will come back to it later. The last but not the least is the US treasury which saw a sharp decline but quickly recovered.</p><h4>Drawdowns</h4><p>Another way to look at this graph is the drawdown graph. In simple terms, drawdown measures the fall from the peak at any particular time period in time. Let’s see the graph now.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a27f017fe01341c5c10161b5237565d6/href">https://medium.com/media/a27f017fe01341c5c10161b5237565d6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*6EcsmJqah4sJOpNp" /></figure><p>For example, in the S&amp;P500 graph, there was a peak around February and then it fell close to 33%. The drawdown values for each are given below:</p><p>Maximum drawdowns<br>Crude Oil 67.07%<br>SP500 33.92%<br>Gold 11.80%<br>TLT 15.72%</p><h4>Relationship of SP500 with Crude Oil, Gold and TLT</h4><p>The drawdown helps us understand how much the asset class can fall but they are still different asset classes, right? <strong>How can we compare them, or in other words, what is the relationship between them?</strong></p><p>To answer that we will create a scatter plot. A scatter plot helps us to understand the relationship between the instruments we have.</p><p>First, we will plot a scatter plot of the different instruments with respect to the S&amp;P500. Let’s see how they look.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c36fbbfab019d362c53a068bc435a43d/href">https://medium.com/media/c36fbbfab019d362c53a068bc435a43d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*MxFy40W1Y_S8j08r" /></figure><p>The red scatter plot is the S&amp;P500 in relation to the TLT, which shows a negative relationship. We all know that in a moment of crisis, there is an economic slowdown and the Feds and other Govt. agencies step in and there is a reduction in the rate. Since there is an inverse relationship between the rates and price, the reduction in the rate results in an increase in price. Hence, we see this negative relationship between the S&amp;P500 and the TLT. However, we have a few data points and hence a conclusion would be unwise here.</p><p>With gold, we see no relationship with the S&amp;P500 as the data points are spread out in this plot. When it comes to Crude oil and S&amp;P500, it seems that the faraway points are negatively correlated but the nearer points are positively correlated. Hence we cannot really conclude anything at this point.</p><p>This was one way to establish a relationship between the different instruments. Based on these relationships, we could create a portfolio which will work for us in the time of pandemics.</p><h4>Sector Performance</h4><p>So far we were talking about the S&amp;P500 as a whole. But what about the individual sectors comprising the index?</p><p>One way to look at it is the sectoral performance. I have created a graph which takes into account the major sectors of the market.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b75e4e1dfdfc13942adecdece86f1d39/href">https://medium.com/media/b75e4e1dfdfc13942adecdece86f1d39/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*ZV9wPK7sSvtzLRL3" /></figure><p><strong>One observation from this graph is that you can see Pharma and Technology are two sectors which are relatively less impacted.</strong> In the graph above, Technology is this orange line which you can see at the very top which is followed by Pharma as the blue line. Overall, you can see that all the sectors are in the same direction but the relative magnitude, or the fall, is different. As I stated earlier, Pharma and Technology are relatively less impacted.</p><p><strong>You can see the pink line representing energy as the most impacted.</strong> This is logical considering we have seen earlier how crude oil was the most impacted, which in turn impacted the energy sector since they both are directly related. Another observation is the green line, which indicates financials, has also fallen. The reason for this is that whenever the economy slows down there are a lot of loan defaults which result in the rise of non-performing assets for a bank and eventually they will take a hit on their P&amp;L. The rise in NPAs makes it really difficult for a bank to survive. One classic example of this in the Indian market is Yes bank. Thus, in this economic crisis, financial sectors are bearing the brunt.</p><p><strong>Does this mean we should ignore financials and load up on, let’s say Technology and Pharma and what should my portfolio look like?</strong></p><p>Along with this, we all have these questions as shown below:</p><ul><li>When will it end?</li><li>How will it end?</li><li>Will markets fall more from here? And how it will move from here?</li><li>Should I buy Gold as safe haven? How should my portfolio look?</li><li>It is so contagious, will everyone be affected by it?</li><li>How the earnings will be impacted and which sector will be impacted the most?</li></ul><p>Let’s try to answer these questions by looking at past pandemics and how the world fared. I would like to add a small disclaimer here that I do not have any crystal balls which will give me a 100% correct answer but I believe data is always the best way to come to a reasonably good conclusion. Let us begin now.</p><h3>Past Pandemics</h3><blockquote>“Those who do not remember the past are condemned to repeat it.”</blockquote><blockquote>~ George Santayana</blockquote><p>If you think that this is a special scenario, well, you will be surprised to know that pandemics have been affecting us since the past few centuries. I have compiled some data in a simple table which is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*XrLlnhyqx8zThsuY2umrRw.png" /><figcaption>Source: <a href="https://www.weforum.org/agenda/2020/03/a-visual-history-of-pandemics/">World Economic Forum</a></figcaption></figure><p>I have highlighted the top three pandemics according to the death toll in red and the bottom four in green.</p><p><strong>There is one significant trend which is coming out here. Can you figure it out?</strong></p><p>There are actually two trends which are emerging here. One is that there is a significant fall in the number of deaths over the years. Another trend is the period has also significantly reduced. The reason for these is an improved healthcare system which is available to us and also a lot of effort which goes in finding and deploying vaccinations to the general public.</p><p>But this was a look at the past. <strong>How is Coronavirus affecting the infection and death toll?</strong> Let’s quickly understand how contagious is the Corona Virus</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*22n9HdK1nA-qONBX" /><figcaption>Source: <a href="https://www.weforum.org/agenda/2020/03/a-visual-history-of-pandemics/">World Economic Forum</a></figcaption></figure><p><strong>What is this infographic telling you?</strong></p><p>It shows us how quickly the infection can spread. Measles tops the list here at 16. It means that a person suffering from measles can infect 16 other people who are unvaccinated. In comparison, Coronavirus stands at 2.5, implying that a person infected with Coronavirus can impact on an average of 2.5 people.</p><p><strong>But then why do we see an exponential rise in the number of cases?</strong></p><p>I would like to draw an analogy with our investment world and bring your attention to compounding. A single person gets impacted on day one. So next day 2.5 will be impacted. The third day we will see a count of around six people who are getting infected. These six people will again impact (6*2.5) around 15. In this manner, the number of people infected rises exponentially.</p><p>The best way to contain this is is social distancing which is what is followed by most of the Governments in the world.</p><h3>So how will this thing end?</h3><blockquote>Past performance is the best predictor of success</blockquote><blockquote>~ Jim Simons</blockquote><p>There are actually two scenarios here which I would like to talk about. Let’s start with the first one.</p><h3>Scenario 1: Strict measures are taken</h3><h3>SARS</h3><p>There is one analogy in our history and it is regarding the SARS outbreak. SARS was of a similar nature and it started in southern China in 2002. It was brought under control by the measures which we are seeing around us which is through isolation, quarantine, and contact tracing.</p><p>In this pandemic, around 8,500 people were infected. Currently, there is no vaccine for this but they were able to contain this through social distancing and the number of new infections was flattened out and consequently, the number of deaths eventually reduced. But while it was easy to identify people who were infected with SARS, it is slightly trickier for the coronavirus carriers. Nevertheless, we can still use this as a reference to understand the impact of Coronavirus on the industry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5qlPeBwp-XEPuV3w" /><figcaption>Source: Financial Times (FT)</figcaption></figure><p>Over here, we the cumulative count and along with that the number of days since the 100th case. You can also see the trajectories if the cases double every day, the second day and so on.</p><p>For example, the U.S. took 24 days to reach 100,000 counts and similarly for other countries. The objective here is to make this exponential rising curve a flat as soon as possible. Here, let me direct your attention to South Korea dirt who made it quite flat and a similar thing was done in SARS. Let’s see that chart now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ez2DN4RkAVkHNpSR" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/Severe_acute_respiratory_syndrome">Wikipedia</a></figcaption></figure><p>This was taken from Wikipedia, and here you can see the cumulative cases in the form of the orange line, which stands at 8437. Initially, the number of cases rose exponentially, right from April to May, but because of social distancing and other measures, the number of incremental case started reducing. This created an “S” or a logistic curve which we see above. The death rate followed the orange curve and flattened out.</p><p>If we believe that SARS is a good analogy of what will happen with the Coronavirus, then the following timelines will help us understand how the current situation will pan out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*Q1qFwfQb5UTeRoxNCQyHAg.png" /><figcaption>Source: <a href="https://www.cdc.gov/about/history/sars/timeline.htm">CDC</a></figcaption></figure><p>Let’s see how the financial markets performed during the SARS outbreak.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*AyElmfKI_pcvvBhd" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>This is the same daily percentage change of the four instruments which we are working with. To recall, Crude oil, S&amp;P 500, Gold and 20 years US Treasury bonds. Since this was pretty much a disease epidemic which was in China so we see less impact on the financial markets but nevertheless the red line indicates when it was announced that this was a possible health emergency. You can see that there are some sharp movements in both directions indicating an increase in volatility. The following graph will make it clearer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*gaBsaaDRGyldbgQl" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>You can see Crude oil fell quite sharply in this period but the S&amp;P500 was relatively less impacted in comparison to the coronavirus scenario. The same is seen in the case of Gold and TLT.</p><p>Let’s see the drawdowns for this period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*jjPkyV7rH562Wzoz" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>You can see that the drawdowns were pretty bad. S&amp;P500 was around 15% during that period and Crude oil was 30%. But gold was again in mid-teens and US Treasury was down around 5%. I should point out that since this was managed very well by containing the epidemic, the fall was not as significant as we are seeing right now.</p><p>We will try the correlation analysis and we see that everything is negatively correlated with S&amp;P500.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*69_F3xRtZUisn3o3" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>This wouldn’t be the right data to work with since this epidemic happened in China. A better metric would be to compare it with the Shanghai index. But nevertheless, we are seeing a very high negative correlation with S&amp;P500 and the Treasury bonds, gold, and crude oil in this period of time.</p><p>Let’s see how the things pan out after the crisis is over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*_IMUM0IrRSs3Y3-o" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>The blue shaded region is when WHO announced that the crisis has been averted. You can see that while there is quite a fall after the WHO declared an emergency, but the index actually recovered quite sharply and the number of days since this happened was also quite short.</p><p>I have just taken four instruments for simplicity and you can use your own local index for any correlation analysis you might want to undertake. Do share your insights with me in the comments.</p><p>The measures taken for containing SARS are similar to the measures being undertaken for Corona Virus. Actually, it was very well handled and hence it didn’t spread very much away from China.</p><p>We will see the returns from the time the WHO declared emergency which is 26th March.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*8WephQhT82uGYNON" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>Do understand that it is the one month, three-month, six-month and one-year returns. You can see that the returns for the S&amp;P 500 are very robust like 5%, 10%, 15% and 25% for the respective time periods. It is a similar case for the others as well. Here, CL=F stands for crude oil futures, GC=F stands for gold futures and TLT is the treasury.</p><p>If you were to ask which sectors will be impacted the most and how the performance of this sector will be after the crisis, I have tried to answer that with this graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*oHzOdFBDU21eNGCn" /><figcaption>Source: FT and JPMorgan</figcaption></figure><p>You can see that the sector’s which are most impacted are at the top and sectors which have shown a positive return or the least impacted are shown at the bottom. This graph is of MSCI China sectors. As you can see the most impacted were consumer durables, Software, hotels, restaurants and leisure and Airlines. The least impacted were healthcare, food, beverage and tobacco. The dark blue indicates the performance of the sector during the crisis and the light blue indicates the performance of the sector after the crisis. You can see that food and beverage not only had a positive return during the crisis, but it continued the positive performance after the epidemic was over too. This gives us some cues like the products which are in FMCG or daily essentials are something which we can look for and similarly healthcare and pharma companies are showing similar trends. Interestingly, retailing showed a negative trend during and after the epidemic was over.</p><p>So if we think that this scenario is quite possible, then you will know which sectors will be showing a healthy recovery and which will take more time to recover.</p><p>This was a scenario where we are able to contain the epidemic by strong measures. We will now move on to the next scenario.</p><h3>Scenario 2: The disease spreads widely</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mXB15NJ0jj7--rOftuMl9A.png" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/Influenza_pandemic">Wikipedia</a></figcaption></figure><p>Something similar actually happened in the past too. We are talking about the Spanish flu which started during World War one and affected nearly 56% of the world population.</p><p>The other two are the Asian flu and Hong Kong flu which impacted around half a million. For comparison, I have given the numbers for coronavirus. So this numbers which might not be most recent but it can help compare the present scenario with the past.</p><h3>Swine Flu</h3><p>For this second scenario, we will draw an analogy with the Swine flu which was started in 2009 and impacted around 20% of the population. Let us take the financial impact during this time period.</p><p>This is the timeline for Swine Flu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*hwo8LkMEN0kgFrcl8_WsbQ.png" /><figcaption>Source: <a href="https://www.cdc.gov/flu/pandemic-resources/2009-pandemic-timeline.html">CDC</a></figcaption></figure><p>Let us look at the financial markets now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*xA1b4u7UPkZ5Ncwh" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>The blue line is when the first case was reported and the red line is when the WHO declared an emergency.</p><p>So you will see a stark difference to what is seen currently in the markets because the markets were recovering from the 2008 financial crisis and the stocks were significantly undervalued in that particular period in time. Therefore, the movements were not as sharp as the movements which we are seeing currently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*m-GtqNua3mnz4nXC" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>But nevertheless, when the pandemic was declared, there was this falling crude oil after the red line. This was the case with Gold and S&amp;P 500 as well. Only the TLT was rising and the reason was that the Fed had lowered the interest rate to support the economy.</p><p>Let’s check out the drawdowns now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*Mq4co_I4oK1RG2Yd" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>Maximum drawdowns</p><p>Crude Oil -21.68%<br>SP500 -16.00%<br>Gold -13.56%<br>TLT -14.37%</p><p>So S&amp;P 500 had a 6% drop after the pandemic was declared. Crude oil was down around 18%, Gold around 7% and TLT in the range of 13%. However, there were multiple waves and while it was not thought of as severe initially, it started spreading to a lot of people in the U.S. Hence, we see a lot of our fall not when the pandemic was declared but sometime later.</p><p>Let’s see what the scatter plot tells us now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*iPXwlG705lxiOdDT" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>Here you can see a clear relationship emerge from the asset classes. For example, the S&amp;P 500 and TLT is negative, which implies that the S&amp;P500 will fall if TLT rises. The relationship with gold is not exactly clear and thus we can’t be sure if Gold is a safe haven or not.</p><p>With crude oil, we are seeing a positive relationship between the two, implying that if S&amp;P500 falls, Crude oil will decline too.</p><p>To summarise, S&amp;P500 has a positive correlation with Crude oil, negative with TLT and no relation with Gold.</p><p>Let’s see how things turned out after WHO said everything is good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*LM2IMp5bFQMtmIvT" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>As we have seen before, The blue shaded region is after the WHO declared the pandemic is over. You can see the returns are pretty strong. The data is for a year after the declaration and these are the returns. Do keep in mind that the returns are calculated from the start date when the first case was reported.</p><p>Returns from the first date</p><p>Crude Oil 73.62%<br>SP500 42.20%<br>Gold 29.89%<br>TLT -10.46%</p><p>Thus, when everything gets sorted out, the market tries to catch up and enters into a bullish mode and everything looks perfectly normal. Let’s see the returns in a bar format for added emphasis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*xAhSwWb9E6V0p6j4" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>You can see the performance of all the instruments with respect to the time horizon here.</p><p>Let’s see the sector performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*HkoVtJbmO61S6wHN" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>It is interesting to note that unlike the previous scenario, here the financial sector performed relatively well and was actually at the top. The reason is basically the world had plunged into an economic recession a year from the swine flu epidemic, which had led to significant sell-offs and causing the financial market to be undervalued. It was a similar case for technology stocks as well.</p><p>The energy sector was significantly impacted during the epidemic but rallied up sharply after the situation turned for the better. This is similar to the crude oil rally which we saw earlier.</p><h3>Asian Flu</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*nFHf8lNmJdnG-GB4" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*Xd2ipKWXDqlYSCkh" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>I hope this gives some perspective on how the situation will turn out at present. In fact, just for an added exercise let us look at the Asian flu epidemic too.</p><p>The following is the performance of S&amp;P500.</p><p>Here, the red line indicates the time it started spreading in the U.S. The blue area is after WHO declared that it was over.</p><p>The same story is repeated for drawdowns where we see a maximum drawdown of 20% after WHO declared a public emergency.</p><h3>Hong Kong Flu</h3><p>Let’s see what Hong Kong Flu tells us now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*HZ3YiOsp84eZDfTO" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>The red line indicates the time when WHO declared an emergency while the blue shaded region shows us the period after WHO said it was over.</p><p>Coming to the drawdowns, we see a maximum drawdown of 35%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*joyd1Wz62u29f-mv" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>An important observation here is that the market does not like uncertainty. Thus, when the emergency is declared, the market declines sharply. But once we have taken the necessary steps to contain the situation and once the WHO has declared it is safe, the market immediately responds and rallies up to the pre-crisis levels or in some cases, even surpassing it.</p><h3>Spanish Flu</h3><p>Let’s see what the Spanish flu teaches us. The Spanish flu can be said to have occurred in three waves. June and July of 1918, then in October to December of 1918 and finally from January to March in 1919. I have used the Dow Jones Index here and the S&amp;P500 was created in 1927. You can see the performance below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/0*KJsFrzmIQjlyowfr" /><figcaption>Source: <a href="https://www.macrotrends.net/1319/dow-jones-100-year-historical-chart">macrotrend</a></figcaption></figure><p>You can see that the market was going up even through the worst phases of the Spanish flu. The reason is that, similar to swine flu, there was a bigger event which impacted the markets, and this was world war 1. The world was actually recovering from world war 1, which ended in 1918, when the Spanish flu struck. This is why there is an uptrend in the markets.</p><h3>World War II</h3><p>As an aside, here is how world war 2 impacted the financial markets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*VWS4hjQjr7lUj6eU" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>The V shape is similar to the past graphs we have seen, just time of recovery is different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*TACssKOkyYPq967P" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>We know that world war 2 ended in 1945, but the markets started rising from 1942. As I said earlier, the market does not like uncertainty. That is why the market dipped below, but in 1942 the allied forces had come together and started planning a coordinated plan to defeat the enemy. Thus, the market factored in this information and hence, once we were certain of the plans, the markets started recovering. The maximum drawdown here can be seen around 43%.</p><h3>Factor Investing</h3><p>How would you invest in the current situation? One emerging method is factor investing. One example of a factor is momentum which states that if anything rises for a year, it will continue to rise.</p><p>Let us see the performance of some factors during COVID19.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*N5IAWHsK3Q36tzbV" /><figcaption>Source: <a href="https://ssrn.com/abstract=3564688">van Vliet, Pim and Baltussen, Guido, Equity Styles and the Spanish Flu (March 30, 2020)</a></figcaption></figure><p>The monthly performance of various factors from the months of January to March 20 is shown above.</p><p><strong>Here you can see that while the others are generally negative, momentum-based factor investing is outperforming the market. To some extent, even low-volatility has outperformed the market. But will this continue?</strong></p><p>It is actually very difficult to answer this question based on the current table. Let’s try to see how things played out in the past.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/0*xvQjgDDagfrk77iM" /><figcaption>Source: <a href="https://ssrn.com/abstract=3564688">van Vliet, Pim and Baltussen, Guido, Equity Styles and the Spanish Flu (March 30, 2020)</a></figcaption></figure><p>We can see that while momentum is close to the market (in black) the small-cap is underperforming. But in the recovery phase, the small caps outperform all other forms of investing. You can also see momentum and low volatile stock during the recovery phase have underperformed the market.</p><p>But so far, we have seen data which was related to pandemics. I will try to present the data about market corrections happening before 1926, or more specifically 1872–1918. These corrections are not exactly related to pandemics but they were a time of great uncertainty, Let’s see how this plays out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3tUb1vYmXiHaN0o7" /><figcaption>Source: <a href="https://ssrn.com/abstract=3564688">van Vliet, Pim and Baltussen, Guido, Equity Styles and the Spanish Flu (March 30, 2020)</a></figcaption></figure><p>As we have seen earlier, the low volatility and momentum-based investing do not fall as low as the market but do not outperform the market in the recovery phase. In contrast, small caps decline more than the market during the correction phase but outperform it in the recovery phase.</p><h3>When will this end?</h3><blockquote>Research on pandemics by Hackett Financial shows that most major viral outbreaks, such as the Great Plague of 1665 and the Spanish flu a little over a century ago, played out within 3 months. In other words, those bad events came and went mainly within 12 weeks.</blockquote><p>Considering the past data which shows that the number of days till WHO declares it is safe has reduced, we could be looking at anywhere between three to six weeks.</p><p>Swine Flu: 116 days<br>SARS: 101 days</p><p>However, I cannot stress enough the importance of social distancing. The governments across the world should be proactive in imposing social distancing. In fact, the following is the chart of various cities in America at the time the Spanish flu was spreading across the US.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/0*yC5nwrWBjm2263W7" /><figcaption>Source: N<a href="https://api-nationalgeographic-com.cdn.ampproject.org/c/s/api.nationalgeographic.com/distribution/public/amp/history/2020/03/how-cities-flattened-curve-1918-spanish-flu-pandemic-coronavirus">ational Geographic</a></figcaption></figure><p>The shaded region is the time when the lockdown was imposed and you can see the death toll declining rapidly. Thus, social distancing is really effective to overcome this crisis.</p><h3>How to protect your equity heavy portfolio?</h3><p>If you’re thinking about your portfolio, then TLT makes sense since it has a negative correlation to the S&amp;P500. You should avoid crude oil as the demand for it could weaken due to the current crisis.</p><p><em>Note: You should consult with a financial advisor before adopting any of the recommendations.</em></p><h3>Summary</h3><p>Phew! We have gone through quite a few scenarios. Let us try to summarise these in a table somehow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*eu5zhQGHEtjLYPA4n0JE5w.png" /><figcaption>Source: Yahoo finance and GitHub Code</figcaption></figure><p>You can see in terms of drawdown, COVID is in third place at 33%. But if you check the returns after a period of six months, they look really promising. This will give you some idea on how the returns will look after 6 months and a year after COVID 19.</p><p><strong><em>Note from Towards Data Science’s editors:</em></strong><em> While we allow independent authors to publish articles in accordance with our </em><a href="https://towardsdatascience.com/questions-96667b06af5"><em>rules and guidelines</em></a><em>, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our </em><a href="https://towardsdatascience.com/readers-terms-b5d780a700a4"><em>Reader Terms</em></a><em> for details.</em></p><h3>Source and References</h3><ol><li><a href="https://www.britannica.com/event/Hong-Kong-flu-of-1968">Britannica</a></li><li><a href="https://www.cdc.gov/flu/pandemic-resources/index.htm">CDC</a></li><li><a href="https://www.who.int/">World Health Organization: WHO</a></li><li>Yahoo finance</li><li>Financial Times</li><li>World Economic Forums</li><li><a href="https://www.medicinenet.com/script/main/art.asp?articlekey=26428">Medicinenet</a></li><li><a href="https://github.com/QuantInsti/webinars/tree/master/Impact%20of%20Pandemics%20on%20Financial%20Markets">Source code</a></li><li>Wikipedia</li><li>JPMorgan</li><li><a href="https://www.oaktreecapital.com/insights/howard-marks-memos">Howard Marks Memos</a></li><li><a href="https://www.ncbi.nlm.nih.gov/books/NBK92460/">NCBI</a></li><li><a href="https://www.weforum.org/agenda/2020/03/a-visual-history-of-pandemics/">World Economic Forum</a></li><li>JPMorgan</li><li><a href="https://ssrn.com/abstract=3564688">van Vliet, Pim and Baltussen, Guido, Equity Styles and the Spanish Flu (March 30, 2020)</a></li><li><a href="https://api-nationalgeographic-com.cdn.ampproject.org/c/s/api.nationalgeographic.com/distribution/public/amp/history/2020/03/how-cities-flattened-curve-1918-spanish-flu-pandemic-coronavirus">National geographic</a></li><li>Pandas Datareader</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a4feb6951f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pandemics-impact-financial-markets-9a4feb6951f5">How Pandemics Impact Financial Markets</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Historical Stock Price Data in Python]]></title>
            <link>https://medium.com/data-science/historical-stock-price-data-in-python-a0b6dc826836?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0b6dc826836</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Fri, 07 Feb 2020 14:47:48 GMT</pubDate>
            <atom:updated>2024-09-18T09:18:56.930Z</atom:updated>
            <content:encoded><![CDATA[<h4>Get daily and minute level historical stock data using yahoo finance. And get the list of stock tickers in stock index such as S&amp;P500.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UY9SOZOx4mZntUjB" /><figcaption>Photo by <a href="https://unsplash.com/@austindistel?utm_source=medium&amp;utm_medium=referral">Austin Distel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Financial assets such as stocks, commodities and ETFs can be unpredictable and volatile. Prices can swing up and down, and that’s why it’s important to dig into how they behave before making any trading decisions. To do that, we need to gather and analyze historical prices carefully. And this is where Python comes in — it’s a great tool to help us quickly fetch and analyze all that price data. It helps in making our decision-making process smoother and more informed.</p><p>The stock data can be downloaded from different packages such as yahoo finance, quandl and alpha vantage. In this article, we will look at fetching the daily and minute level data from yahoo finance.</p><p>If you’re interested in learning about getting news and stock fundamental data, below course can be helpful.</p><p><a href="https://bit.ly/3MT6BL6">Getting Market Data: Stocks, Crypto, News &amp; Fundamental | Free Quantra Course</a></p><h3>Yahoo Finance</h3><p>We’ll use the yfinance package to fetch stock data. If you don’t have it installed yet, just run !pip install yfinance. Once you’ve got it set up, the code below shows how to grab data for Apple (AAPL) from 2016 to 2024 and plot the adjusted closing price. It’s a quick way to visualize how the stock performed over time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef1b2859e115e953e17c161d20de95b2/href">https://medium.com/media/ef1b2859e115e953e17c161d20de95b2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*Mww1gTV26AARPNTt1tKeSQ.png" /><figcaption>Daily Price of AAPL</figcaption></figure><h3>Data for multiple stocks</h3><p>In the below code, we will fetch the data of multiple stocks and store it in a dataframe data. Then we will calculate the daily returns and plot the cumulative returns of all the stock prices using matplotlib package.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55b593b90edf89a8ffbf80da89dfe4a6/href">https://medium.com/media/55b593b90edf89a8ffbf80da89dfe4a6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*jul_T2DWy913W4pro0z3zQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e0da1590e9ac21984e0a9f59f311e73/href">https://medium.com/media/2e0da1590e9ac21984e0a9f59f311e73/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*bh8Xqoy4XMzB5f4Z34Alrw.png" /></figure><h3>Minute level data</h3><p>Through yfinance, you can also fetch the data of minute frequency. You can download for other frequency by just tweaking the interval parameter on line no 8 below. Following values are supported in the interval: 1m, 5m, 15m, 30m, 60m.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ac0b41aa07c786adb7b2801512d9542/href">https://medium.com/media/0ac0b41aa07c786adb7b2801512d9542/href</a></iframe><p>You can analyze this data, create a trading strategy and analyze the performance of the strategy using the pyfolio package. It computes the Sharpe ratio, Sortino ratio, maximum drawdowns and many other metrics.</p><p>I hope you find this useful and happy trading and analysis.</p><h3>List of stock tickers in S&amp;P500</h3><p>The next logical requirement is how to get the tickers that make up your favourite index or are listed on a specific exchange. For that, you can get the tickers from the Wikipedia page and use those tickers to get data from yfinance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b8d6f76998d4dfb0dbf433d93760922/href">https://medium.com/media/5b8d6f76998d4dfb0dbf433d93760922/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*ea-QQZLAEuqcvLmc1gsLDA.png" /><figcaption>List of stock tickers and other information.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*YPRpL9ooGjMeo0g1mUzWQg.png" /><figcaption>Close price of the stocks in S&amp;P500.</figcaption></figure><h4>Sources and References</h4><ol><li>yfinance</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0b6dc826836" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/historical-stock-price-data-in-python-a0b6dc826836">Historical Stock Price Data in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to Data Science]]></title>
            <link>https://medium.com/data-science/intro-to-data-science-531079c38b22?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/531079c38b22</guid>
            <category><![CDATA[epl]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[premier-league]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Mon, 07 Oct 2019 14:32:07 GMT</pubDate>
            <atom:updated>2020-01-13T09:39:41.921Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step guide to learn data science</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*OaCfUKMz2ir04o-C.jpg" /><figcaption>Source: Freepik</figcaption></figure><p>Sports are always interesting to watch and we all have our favourites clubs. And we always discuss with our friends and colleagues on which club will win the EPL These discussions are based on our intuition and maybe on the recency bias. A better way to make an informed guess is to involve the data to predict the EPL winner.</p><h3><strong>Data Science</strong></h3><p>But first, let’s start by understanding what is <a href="https://quantra.quantinsti.com/course/introduction-to-data-science">data science</a>?</p><ol><li>Have you ever wondered how Amazon, eBay suggest items for you to buy?</li><li>How Gmail filters your emails in the spam and non-spam categories?</li><li>How Netflix predicts the shows of your liking?</li></ol><p>How do they do it? These are the few questions we ponder from time to time. In reality, doing such tasks are impossible without the availability of data. Data science is all about using data to solve problems. The problem could be decision making such as identifying which email is spam and which is not. Or a product recommendation such as which movie to watch? Or predicting the outcome such as who will be the next President of the USA? So, the core job of a data scientist is to understand the data, extract useful information out of it and apply this in solving the problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tOgakR76KLgvGOslDXuOiw.png" /><figcaption>What is Data Science?</figcaption></figure><blockquote>A person who is better at statistics than any software engineer and better at software engineering than any statistician.</blockquote><blockquote>— Josh Wills</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fcsj6RE-EmSU%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dcsj6RE-EmSU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fcsj6RE-EmSU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/568fb5d7e76fa5860c18cc979bcf2457/href">https://medium.com/media/568fb5d7e76fa5860c18cc979bcf2457/href</a></iframe><p>After reading this article, you will be able to</p><ol><li><em>Explain the steps in </em><a href="https://quantra.quantinsti.com/course/introduction-to-data-science"><em>data science</em></a></li><li><em>Apply these steps to predict the EPL winner</em></li><li><em>Explain the importance of data quality</em></li><li><em>Define data collection methods</em></li></ol><h3>Data Science Life Cycle</h3><h3><strong>Step 1: Define Problem Statement</strong></h3><p>Creating a well-defined problem statement is a first and critical step in data science. It is a brief description of the problem that you are going to solve.</p><p>But why do we need a well-defined problem statement?</p><blockquote>A problem well defined is a problem half-solved.</blockquote><blockquote>— Charles Kettering</blockquote><p>Also, all the efforts and work you do after defining the problem statement is to solve it. The problem statement is shared by your client. Your client can be your boss, colleague or it can be your personal project. They would tell you the problems they are facing. Some examples are shown below.</p><ol><li><em>I want to increase the revenues</em></li><li><em>I want to predict the loan default for my credit department,</em></li><li><em>I want to recommend the job to my clients</em></li></ol><p>Most of the times, these initial set of a problem shared with you is vague and ambiguous. For example, the problem statement: “I want to increase the revenue”, doesn’t tell you how much to increase the revenue such as 20% or 30%, for which products to increase revenue and what is the time frame to increase the revenue. You have to make the problem statement clear, goal-oriented and measurable. This can be achieved by asking the right set of questions.</p><blockquote>“Getting the right question is the key to getting the right answer.”</blockquote><blockquote>– Jeff Bezos</blockquote><p>How can you ask better or right questions to create a well-defined problem statement? You should ask open-ended rather than closed-ended questions. The open-ended questions help to uncover unknown unknowns. The unknown unknowns are the things which you don’t know you don’t know.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*sT9ZC3xjsejXb5imSGglQQ.png" /><figcaption>Source: USJournal</figcaption></figure><p><em>We will work on a problem statement “Which club will win the EPL?”</em></p><h3><strong>Step 2: Data Collection</strong></h3><p>You need to collect the data which can help to solve the problem. Data collection is a systematic approach to gather relevant information from a variety of sources. Depending on the problem statement, the data collection method is broadly classified into two categories.</p><p>First, when you have some unique problem and no related research is done on the subject. Then, you need to collect new data. This method is called as primary data collection. For example, you want information on the average time that employees spend in a cafeteria across companies. There is no public data available of these. But you can collect the data through various methods such as surveys, interviews of employees and by monitoring the time spent by employees in cafeteria. This method is time-consuming.</p><p>Another method is to use the data which is readily available or collected by someone else. These data can be found on the internet, news articles, government census, magazines and so on. This method is called as secondary data collection. This method is less time-consuming than the primary method.</p><p><em>For our problem statement on EPL. You can collect and aggregate the data from various open-source websites such as Github, Kaggle, and datahub.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IavYVxbJkmN9zk1c" /><figcaption>Table 1: Snapshot of data collected from web sources</figcaption></figure><h3><strong>Step 3: Data Quality Check and Remediation</strong></h3><p>One of the most important and often ignored aspects by data scientists is ensuring the data that is used for analysis and interpretation is of good quality.</p><p>After collecting the data, most people start the analysis on it. Often, they forgot to do a sanity check on the data. If the data is of bad quality, it can give misleading information. Simply said: “Garbage in, garbage Out”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*Vyc9L1D_-uqysG3N" /></figure><p>For example, if you start the analysis without ensuring data quality. Then you might get unexpected results such as the Crystal Palace club will win the next EPL. However, your domain knowledge on EPL says that the result looks inaccurate as Crystal Palace has never even finished in the top 4.</p><h3><strong>Step 4: Exploratory Data Analysis</strong></h3><p>Before you model the steps to arrive at a solution, it’s important to analyse the data. It is the most exciting step as it helps you to build familiarity with the data and extract useful insights. If you skip this step then you might end up generating inaccurate models and choosing the insignificant variables in your model.</p><blockquote>As quoted by John Tukey, developer of Exploratory Data Analysis,</blockquote><blockquote><em>“It is important to understand what you CAN DO before you learn to measure how WELL you seem to have DONE it.”</em></blockquote><p>But it can be strenuous and difficult. Are there tools or techniques to explore data in a meaningful way?</p><p>Yes, you can use descriptive statistics such as central value measures and variability measures. Also, visualisation methods such as graphs and plots can be used for analysis.</p><p>For example, you can calculate the average goals scored per match. You can also check if home advantage is a real thing?</p><p>The bar chart below shows the home advantage of Liverpool. It shows that of all home matches of Liverpool, they have won 17 and drawn 2. But have never lost any home match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*BUkPYnYglQm6Yb4FsM1eHQ.png" /><figcaption>Chart 1: Home Advantage for Liverpool [1]</figcaption></figure><p>However, there is no home advantage of Leicester. It shows that of all home matches of Leicester, they have won 8, lost 8 and drawn 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*woMCF3TxVKnHkN4HCoLeLQ.png" /><figcaption>Chart 2: No home Advantage for Leicester [1]</figcaption></figure><p><em>Source: Intro to Data Science by Quantra</em></p><p>Data analysis is an iterative process that helps you to get closer to the solution. Every iteration has a cost associated with it. Therefore, it is advisable that as a data scientist you plan properly so that the number of iterations is reduced. You can play with data, create your own graphs and learn to derive inferences. All these analyses which you performed are commonly known as exploratory data analysis (EDA).</p><h3><strong>Step 5: Data Modelling</strong></h3><p>Modelling means formulating every step and gather the techniques required to achieve the solution. You need to list down the flow of the calculations which is nothing but modelling steps to the solution. The important factor is how to perform the calculations. There are various techniques under Statistics and Machine Learning that you can choose based on the requirement.</p><p>For EPL data, we chose to use statistical techniques to predict the winner in the current season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*4aYvYQE1nicruLs3" /><figcaption>Figure 1: Data Modelling Steps [1]</figcaption></figure><p><strong>Top 6 teams</strong></p><p>As per the data of the last three seasons, we have learnt that the winner of EPL is changing but the top 6 clubs remain the same. These clubs are Arsenal, Chelsea, Liverpool, Manchester City, Manchester United and Tottenham Hotspur. The winner in the next season is very likely to be from these top 6 teams.</p><p><strong>Rank Clubs on Player Skills</strong></p><p>As we know, in football a player can be an Attacker, Defender, Midfielder or a Goalkeeper. Each of these positions requires a different set of physical skills. A player is assigned a score based on his physical skills and the position he is playing in. The scores are given on a scale of 0–100. And each of these skills is assigned a weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*Kd1f_fytg5G7kqjMM3OlUA.png" /><figcaption>Table 3: Player skills by Club. [1] and [2]</figcaption></figure><p><strong>Rank Clubs on Past Performance</strong></p><p>We rank the performance of the teams based on the number of wins and difference in goals in the last season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*pdHklkZG5TKTPHL0PmDfGg.png" /><figcaption>Table 4: Club last season performance.</figcaption></figure><p><strong>Combine the Ranks to predict the winner</strong></p><p>Considering 80% weight to the players’ skills and 20% weight to the past performance of clubs we get the final ranking of the top 6 clubs. <br>Based on these calculations, Tottenham Hotspur is likely to win this EPL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*2gXHmgi5O5-kpeT2OXd3Bg.png" /><figcaption>Table 5: Combining Player skills and Past Performance</figcaption></figure><h3><strong>Step 6: Data Communication</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*GZw0Qk2PObYT8CbU.jpg" /><figcaption>Source: Freepik</figcaption></figure><p>This is the final step where you present the results from your analysis to the stakeholders. You explain to them how you came to a specific conclusion and your critical findings.</p><p>Most often you need to present your findings to a non-technical audience, such as the marketing team or business executives. You need to communicate the results in a simple to understand manner. And the stakeholders should be able to chalk out an actionable plan from it.</p><p>But what are the important things you need to keep in mind while communicating your results?</p><ol><li><strong>Know your audience and speak their language</strong></li></ol><p>You should know your audience and speak their language. For example, you are presenting the prediction of EPL winner to a football fan. They do not understand the statistics you have used but they can relate to the steps you followed to decide the winner.</p><p><strong>2. Focus on values and outcomes</strong></p><p>You should focus on value and outcomes. Your stakeholder would not be interested in how you got the data but from where you got the data. The usage of credible sources helps to build confidence in your prediction.</p><p><strong>3. Communicate assumptions and limitations</strong></p><p>You should clearly communicate the important assumptions and limitations you have made. For example, to calculate the overall rating of the team you have assumed 3–4–3–1 formation for all the teams. It’s important that you communicate these to the stakeholders.</p><p>Although the goal is to predict the winner, there might be some other critical findings that are relevant. For example, the teams with the best attack, midfield, defence and goalkeeping. The best player for each position in the league based on the players’ skills. Such a unified view of the data in the form of graphs, charts and numbers is called a Dashboard. you can use Excel to create the dashboard.</p><p>Now, it is your turn. Take a problem and solve it as a data scientist.</p><p>Happy learning!</p><p><strong>Sources and References</strong></p><ol><li><a href="https://quantra.quantinsti.com/course/introduction-to-data-science">Intro to Data Science</a></li><li><a href="https://www.premierleague.com/stats">Premier League Stats</a></li><li><a href="https://en.wikipedia.org/wiki/Data_science">Wikipedia: Data Science</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=531079c38b22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/intro-to-data-science-531079c38b22">Intro to Data Science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Quality: Importance and Remediation]]></title>
            <link>https://medium.com/@ishan.s_54240/data-quality-importance-and-remediation-9011c53dd197?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9011c53dd197</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Fri, 13 Sep 2019 10:39:45 GMT</pubDate>
            <atom:updated>2019-09-13T10:47:35.989Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*Zf5YH8WblvpKvuIE.jpg" /></figure><p>After reading this article, you will be able to</p><ol><li>Explain the importance of data quality</li><li>Explain the causes of data quality issues</li><li>Identify data quality issues</li><li>Fix the data quality issues</li></ol><p>One of the most important and often ignored aspects by data scientists is ensuring the data that is used for analysis and interpretation is of good quality.</p><p>After collecting the data, most people start the analysis on it. Often, they forgot to do a sanity check on the data. If the data is of bad quality, it can give misleading information. Simply said: “Garbage in, garbage Out”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*YcqayqMZSI5HAkjH" /></figure><p>For example, if you start the analysis without ensuring data quality. Then you might get unexpected results such as the Crystal Palace club will win the next EPL. However, your domain knowledge on EPL says that the result looks inaccurate as Crystal Palace has never even finished in the top 4.</p><p>A surprising fact and also strengthens the claim for the importance of the data quality is that a professional data scientist spends approximately 60% of his time, ensuring that data is of high quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*MOen0FviK3pVw17l" /></figure><p>Source: Forbes</p><p>But why the data quality issue occurs? It can occur during data collection or data integration.</p><p><strong>Data collection</strong></p><p>Consider that you are recording the average time that employees spend in a cafeteria weekly across companies. You recorded 100 hours instead of 10 hours, or the unit of measurement recorded incorrectly. Also, if you are interviewing and someone may choose not to respond to certain questions which lead to missing values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*YocOPl7yUyUQa6Mv" /></figure><p><strong>Data integration</strong></p><p>Another cause is when data is collected from different sources and merged. For example, you require the weight of all players of EPL in a single file. You extract the player’s weight data from source X. But weight data of some new players are not available, so you get it from source Y. The unit of weight measurement in source X is in kilograms, and source y is in pounds. If data collected from both sources are combined as it is, then there will be inconsistency in the data, and it will result in inaccurate data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*yniZbi1BhuLFwTe4" /></figure><p>The common data quality issues that are easy to spot are missing values, duplicate values, and inconsistent data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m6rvvJwUe-ePNgxY" /></figure><p>However, some issues are difficult to spot. For example, can you spot what is wrong in this dataset?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AWXUdIP6YXmlPICT" /></figure><p>If you are following EPL, then there is no club with the name of Real Madrid in EPL. This infers that detecting and handling such problems would require domain knowledge. Therefore as a data scientist, you should develop a good understanding of the domain and the problem you are solving.</p><p>But how to fix these data quality issues?</p><p>Once you identify the inaccurate and missing data, you can use the alternate source of data, if available.</p><p>For example, if Bernd Leno’s age is missing then you can get it from the alternative source such as Wikipedia. However, this approach is not always possible as you can’t find an alternative source for every dataset or data point.</p><p>In that case, a simple approach is to remove the inaccurate data.</p><p>This can work well if you have a few inaccurate data points. But if there are many numbers of records with data quality problems then this approach can reduce the data size. Thus resulting in a poor analysis.</p><p>A better approach would be to impute the incorrect or missing values. The mean, mode, and the median of attributes can be used for this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*JyuoepquftZJGse-" /></figure><p>For example, in this dataset, you can impute the missing weight of Joe Hart by the mode of 185 or mean of 178.3 or median of 178.5. We will discuss these methods in detail in the reading document. But there are chances that imputed values are inaccurate. Another approach is to estimate the missing weight based on the player whose height and age is similar to John Hart. That is you look at attributes that can better help in estimating the incorrect or missing value. For example, Matthew Lowton has the same age and height as Joe Hart so you can assign 171. However, not all values can be estimated from the values of other attributes.</p><p>Thus, the approach to remediate the data quality issues depends on the type of data you are dealing with and the domain understanding of the data.</p><p>Source: <a href="https://bit.ly/2m6XT24">Intro to Data Science [Free Course]</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9011c53dd197" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stock Data and Analysis]]></title>
            <link>https://medium.com/@ishan.s_54240/stock-data-and-analysis-529aa9aee60?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/529aa9aee60</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[stock-data]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Thu, 08 Aug 2019 08:23:17 GMT</pubDate>
            <atom:updated>2020-02-07T09:17:24.095Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stock Data and Analysis in Python</h3><h4>Get historical stock data for different timeframes and from different python packages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*bW_TFszfpDEsr9TK.jpg" /><figcaption>Source: Freepik</figcaption></figure><p>Many of us aspire to be quant traders or want to add data analytics to our investment and/or trading process. This is where fetching the right set of data and in right format becomes most important.</p><p>You can get the stock data using popular data vendors. And then you can <a href="https://bit.ly/2YKC0Y1">perform analysis</a> on it. In python, there are many libraries which can be used to get the stock market data. The most common set of data is the price volume data. These data can be used to create quant strategies, technical strategies or very simple buy-and-hold strategies.</p><h3><strong>Yahoo Finance</strong></h3><p>Yahoo Finance is one of the free sources to get stock data. You can get the data either using pandas datareader or can get using yfinance library. The method to get data from yfinance library is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef1b2859e115e953e17c161d20de95b2/href">https://medium.com/media/ef1b2859e115e953e17c161d20de95b2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*99r-3jir5vQ01ux-8YexpA.png" /><figcaption>Daily Price of AAPL</figcaption></figure><h3><strong>Quandl</strong></h3><p>Wiki is one of the free source available on quandl to get the data for the 3000+ US equities. This is a community maintained data. Recently it is stopped being maintained but however, it is a good free source to backtest your strategies.</p><p>To get the data, you need to get the free API key from quandl and replace the &lt;Your_API_Key&gt; in the below code with your API key.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/046f08e0322d5f4962135e07fcae3106/href">https://medium.com/media/046f08e0322d5f4962135e07fcae3106/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*yRCdgg-3rCoglCvyi_ag1w.png" /><figcaption>Daily Data for Coca-Cola</figcaption></figure><p>Note:</p><ol><li>Quandl requires NumPy (v1.8 or above) and pandas (v0.14 or above) to work.</li><li>To get your API key, sign up for a free Quandl account. Then, you can find your API key on Quandl account settings page.</li><li>To fetch data for different markets and types</li></ol><h3><strong>Data for multiple stocks</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55b593b90edf89a8ffbf80da89dfe4a6/href">https://medium.com/media/55b593b90edf89a8ffbf80da89dfe4a6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*jul_T2DWy913W4pro0z3zQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e0da1590e9ac21984e0a9f59f311e73/href">https://medium.com/media/2e0da1590e9ac21984e0a9f59f311e73/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*PDWJkpXMal1i94HfL_YvqA.png" /></figure><h3><strong>Minute level data using the Alpha vantage</strong></h3><p>Until so far we have fetched the data with a frequency of daily. That is the end of the day data. Using the below code, you get the stock data for each minute. This is very useful to backtest intraday trading strategies and for risk management purpose.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58c4d4b119cbfe8ddc853957b601e027/href">https://medium.com/media/58c4d4b119cbfe8ddc853957b601e027/href</a></iframe><p>Note:</p><ol><li>For the time interval between two consecutive data points in the time series, the following values are supported: 1min, 5min, 15min, 30min, 60min</li><li>By default, outputsize=compact. Strings compact and full are accepted with the following specifications: compact returns only the latest 100 data points in the intraday time series; full returns the full-length intraday time series. The &quot;compact&quot; option is recommended if you would like to reduce the data size of each API call.</li></ol><p>Once you have data the next step is to analyze the data. This can be done using pyfolio library. This helps to compute Sharpe ratio, Sortino ratio, equity curve and other analytics useful to analyze the performance of your strategy.</p><p>I hope you find this useful and happy trading and analysis.</p><h3><strong>Sources and References</strong></h3><ol><li><a href="https://bit.ly/2YKC0Y1">Stock Data and Analysis Blog</a></li><li>Quandl</li><li>yfinance</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=529aa9aee60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Regression To Predict Gold Prices In Python]]></title>
            <link>https://medium.com/@ishan.s_54240/machine-learning-regression-to-predict-gold-prices-in-python-e359b0f29973?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e359b0f29973</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[quantitative-strategies]]></category>
            <category><![CDATA[gold]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Thu, 08 Feb 2018 09:25:23 GMT</pubDate>
            <atom:updated>2018-02-08T09:25:23.165Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*XO_A6TXE-7b3tvtt.jpg" /></figure><p>Is it possible to predict where the Gold price is headed?</p><p>Yes, let’s use <a href="https://www.quantinsti.com/blog/machine-learning-trading-predict-stock-prices-regression/">machine learning regression techniques</a> to predict the price of one of the most important precious metal, the Gold.</p><p>We will create a <a href="https://quantra.quantinsti.com/course/introduction-to-machine-learning-for-trading">machine learning</a> linear regression model that takes information from the past Gold ETF (GLD) prices and returns a prediction of the Gold ETF price the next day.</p><p><em>GLD is the largest ETF to invest directly in physical gold.</em> (source:<a href="http://www.etf.com/GLD"> http://www.etf.com/GLD</a>)</p><h3>Steps to predict gold prices using machine learning in python</h3><ol><li>Import the libraries and read the Gold ETF data</li><li>Define explanatory variables</li><li>Define dependent variable</li><li>Split the data into train and test dataset</li><li>Create a linear regression model</li><li>Predict the Gold ETF prices</li></ol><h4>Import the libraries and read the Gold ETF data</h4><p>First things first: import all the necessary libraries which are required to implement this strategy.</p><pre># LinearRegression is a machine learning library for linear regression </pre><pre><strong>from</strong> sklearn.linear_model <strong>import</strong> LinearRegression </pre><pre># pandas and numpy are used for data manipulation </pre><pre><strong>import</strong> pandas as pd </pre><pre><strong>import</strong> numpy as np </pre><pre># matplotlib and seaborn are used for plotting graphs </pre><pre><strong>import</strong> matplotlib.pyplot as plt </pre><pre><strong>import</strong> seaborn </pre><pre># fix_yahoo_finance is used to fetch data </pre><pre><strong>import</strong> fix_yahoo_finance as yf</pre><p>Then, we read the past 10 years of daily Gold ETF price data and store it in Df. We remove the columns which are not relevant and drop NaN values using dropna() function. Then, we plot the Gold ETF close price.</p><pre># Read data </pre><pre>Df = yf.download(&#39;GLD&#39;,&#39;2008-01-01&#39;,&#39;2017-12-31&#39;)</pre><pre># Only keep close columns </pre><pre>Df=Df[[&#39;Close&#39;]] </pre><pre># Drop rows with missing values </pre><pre>Df= Df.dropna() </pre><pre># Plot the closing price of GLD </pre><pre>Df.Close.plot(figsize=(10,5)) </pre><pre>plt.ylabel(&quot;Gold ETF Prices&quot;)</pre><pre>plt.show()</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*2Yjw2g_m7HyVtklf.jpg" /></figure><h4>Define explanatory variables</h4><p>An explanatory variable is a variable that is manipulated to determine the value of the Gold ETF price the next day. Simply, they are the features which we want to use to predict the Gold ETF price. The explanatory variables in this strategy are the moving averages for past 3 days and 9 days. We drop the NaN values using dropna() function and store the feature variables in X.</p><p>However, you can add more variables to X which you think are useful to predict the prices of the Gold ETF. These variables can be technical indicators, the price of another ETF such as Gold miners ETF (GDX) or Oil ETF (USO), or US economic data.</p><pre>Df[&#39;S_3&#39;] = Df[&#39;Close&#39;].shift(1).rolling(window=3).mean() </pre><pre>Df[&#39;S_9&#39;]= Df[&#39;Close&#39;].shift(1).rolling(window=9).mean() </pre><pre>Df= Df.dropna() </pre><pre>X = Df[[&#39;S_3&#39;,&#39;S_9&#39;]] </pre><pre>X.head()</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/0*ma6AZkjjJm8SKIzF.jpg" /></figure><h4>Define dependent variable</h4><p>Similarly, the dependent variable depends on the values of the explanatory variables. Simply put, it is the Gold ETF price which we are trying to predict. We store the Gold ETF price in y.</p><pre>y = Df[&#39;Close&#39;]</pre><pre>y.head()</pre><p><strong>Output:</strong></p><p>Date</p><p>2008–02–08 91.000000</p><p>2008–02–11 91.330002</p><p>2008–02–12 89.330002</p><p>2008–02–13 89.440002</p><p>2008–02–14 89.709999</p><p>Name: Close, dtype: float64</p><h4>Split the data into train and test dataset</h4><p>In this step, we split the predictors and output data into train and test data. The training data is used to create the linear regression model, by pairing the input with expected output. The test data is used to estimate how well the model has been trained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZWTGS8aAXADSFhli.jpg" /></figure><ol><li>First 80% of the data is used for training and remaining data for testing</li><li>X_train &amp; y_train are training dataset</li><li>X_test &amp; y_test are test dataset</li></ol><pre>t=.8 </pre><pre>t = int(t*len(Df)) </pre><pre># Train dataset </pre><pre>X_train = X[:t] </pre><pre>y_train = y[:t]  </pre><pre># Test dataset </pre><pre>X_test = X[t:] </pre><pre>y_test = y[t:]</pre><h4>Create a linear regression model</h4><p>We will now create a linear regression model. But, what is linear regression?</p><p>If we try to capture a mathematical relationship between ‘x’ and ‘y’ variables that “best” explains the observed values of ‘y’ in terms of observed values of ‘x’ by fitting a line through a scatter plots then such an equation between x and y is called linear regression analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vCnUJt4GPCXpbeIS.jpg" /></figure><p>To break it down further, regression explains the variation in a dependent variable in terms of independent variables. The dependent variable — ‘y’ is the variable that you want to predict. The independent variables — ‘x’ are the explanatory variables that you use to predict the dependent variable. The following regression equation describes that relation:</p><pre><strong><em>Y = m1 * X1 + m2 * X2 + C</em></strong></pre><pre><strong><em>Gold ETF price = m1 * 3 days moving average + m2 * 15 days moving average + c</em></strong></pre><p>Then we use the fit method to fit the independent and dependent variables (x’s and y’s) to generate coefficient and constant for regression.</p><pre>linear = LinearRegression().fit(X_train,y_train) </pre><pre><strong>print</strong> &quot;Gold ETF Price =&quot;, round(linear.coef_[0],2), \ </pre><pre>&quot;* 3 Days Moving Average&quot;, round(linear.coef_[1],2), \ </pre><pre>&quot;* 9 Days Moving Average +&quot;, round(linear.intercept_,2)</pre><p><strong>Output:</strong></p><p>Gold ETF Price = 1.2 * 3 Days Moving Average — 0.2 * 9 Days Moving Average + 0.39</p><h4>Predicting the Gold ETF prices</h4><p>Now, it’s time to check if the model works in the test dataset. We predict the Gold ETF prices using the linear model created using the train dataset. The predict method finds the Gold ETF price (y) for the given explanatory variable X.</p><pre>predicted_price = linear.predict(X_test)  </pre><pre>predicted_price = pd.DataFrame(predicted_price,index=y_test.index,columns = [&#39;price&#39;])  </pre><pre>predicted_price.plot(figsize=(10,5))  </pre><pre>y_test.plot()  </pre><pre>plt.legend([&#39;predicted_price&#39;,&#39;actual_price&#39;])  </pre><pre>plt.ylabel(&quot;Gold ETF Price&quot;)  </pre><pre>plt.show()</pre><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*nTc4IQgu3LoUMsYi.jpg" /></figure><p>The graph shows the predicted and actual price of the Gold ETF.</p><p>Now, let’s compute the goodness of the fit using the score() function.</p><pre>r2_score = linear.score(X[t:],y[t:])*100  </pre><pre>float(&quot;{0:.2f}&quot;.format(r2_score))</pre><p><strong>Output:</strong></p><p>95.81%</p><p>As it can be seen, the R-squared of the model is 95.81%. R-squared is always between 0 and 100%. A score close to 100% indicates that the model explains the Gold ETF prices well.</p><p>Congrats! You just learned a fundamental yet strong<a href="https://www.quantinsti.com/blog/overview-machine-learning-trading/"> machine learning</a> technique. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e359b0f29973" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Classification Strategy In Python]]></title>
            <link>https://medium.com/@ishan.s_54240/machine-learning-classification-strategy-in-python-c641e38e4f40?source=rss-baef4b7681f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/c641e38e4f40</guid>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[quantitative-strategies]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ishan Shah]]></dc:creator>
            <pubDate>Thu, 08 Feb 2018 09:13:17 GMT</pubDate>
            <atom:updated>2018-02-08T09:14:05.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*VjoGFkjsccmCGe0e.jpg" /></figure><p>We will step by step implement a <a href="https://goo.gl/NEQWMw">machine learning classification</a> algorithm on S&amp;P500 using Support Vector Classifier (SVC).</p><p>SVCs are supervised learning classification models. A set of training data is provided to the machine learning classification algorithm, each belonging to one of the categories. For instance, the categories can be to either buy or sell a stock. The classification algorithm builds a model based on the training data and then, classifies the test data into one of the categories. Let’s implement the machine learning classification strategy in Python.</p><h3>Step 1: Import the libraries</h3><p>In this step, we will import the necessary libraries that will be needed to create the strategy.</p><pre># machine learning classification</pre><pre><a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html">from sklearn.svm import SVC</a></pre><pre>from sklearn.metrics import scorer</pre><pre>from sklearn.metrics import accuracy_score</pre><pre># For data manipulation</pre><pre>import pandas as pd</pre><pre>import numpy as np</pre><pre># To plot</pre><pre>import matplotlib.pyplot as plt</pre><pre>import seaborn</pre><pre># To fetch data</pre><pre>from pandas_datareader import data as pdr</pre><h3>Step 2: Fetch data</h3><p>We will download the S&amp;P500 data from google finance using pandas_datareader.</p><p>After that, we will drop the missing values from the data and plot the S&amp;P500 close price series.</p><pre>Df = pdr.get_data_google(&#39;SPY&#39;, start=&quot;2012-01-01&quot;, end=&quot;2017-10-01&quot;)        </pre><pre>Df= Df.dropna()</pre><pre>Df.Close.plot(figsize=(10,5))</pre><pre>plt.ylabel(&quot;S&amp;P500 Price&quot;)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*p1faREF-LzXYe_Yo.jpg" /></figure><h3>Step 3: Determine the target variable</h3><p>The target variable is the variable which the <a href="https://goo.gl/NEQWMw">machine learning classification algorithm</a> will predict. In this example, the target variable is whether S&amp;P500 price will close up or close down on the next trading day.</p><p>We will first determine the actual trading signal using the following logic — if next trading day’s close price is greater than today’s close price then, we will buy the S&amp;P500 index, else we will sell the S&amp;P500 index. We will store +1 for the buy signal and -1 for the sell signal.</p><pre>y = np.where(Df[&#39;Close&#39;].shift(-1) &gt; Df[&#39;Close&#39;],1,-1)</pre><h3>Step 4: Creation of predictors variables</h3><p>The X is a dataset that holds the predictor’s variables which are used to predict target variable, ‘y’. The X consists of variables such as <em>‘Open — Close’</em> and <em>‘High — Low’</em>. These can be understood as indicators based on which the algorithm will predict the option price.</p><pre>Df[&#39;Open-Close&#39;] = Df.Open - Df.Close</pre><pre>Df[&#39;High-Low&#39;] = Df.High - Df.Low</pre><pre>X=Df[[&#39;Open-Close&#39;,&#39;High-Low&#39;]]</pre><p>In the later part of the code, the <a href="https://www.quantinsti.com/blog/machine-learning-trading-predict-stock-prices-regression/">machine learning</a> classification algorithm will use the predictors and target variable in the training phase to create the model and then, predict the target variable in the test dataset.</p><h3>Step 5: Test and train dataset split</h3><p>In this step, we will split data into the train dataset and the test dataset.</p><ol><li>First 80% of data is used for training and remaining data for testing</li><li>X_train and y_train are train dataset</li><li>X_test and y_test are test dataset</li></ol><pre>split_percentage = 0.8</pre><pre>split = int(split_percentage*len(Df))</pre><pre><em># Train data set</em></pre><pre>X_train = X[:split]</pre><pre>y_train = y[:split]</pre><pre><em># Test data set</em></pre><pre>X_test = X[split:]</pre><pre>y_test = y[split:]</pre><h3>Step 6: Create the machine learning classification model using the train dataset</h3><p>We will create the machine learning classification model based on the train dataset. This model will be later used to predict the trading signal in the test dataset.</p><pre>cls = SVC().fit(X_train, y_train)</pre><h3>Step 7: The classification model accuracy</h3><p>We will compute the accuracy of the classification model on the train and test dataset, by comparing the actual values of the trading signal with the predicted values of the trading signal. The function accuracy_score() will be used to calculate the accuracy.</p><p><strong>Syntax:</strong> accuracy_score(target_actual_value,target_predicted_value)</p><ol><li>target_actual_value: correct signal values</li><li>target_predicted_value: predicted signal values</li></ol><pre>accuracy_train = accuracy_score(y_train, cls.predict(X_train))</pre><pre>accuracy_test = accuracy_score(y_test, cls.predict(X_test))</pre><pre>print(&#39;\nTrain Accuracy:{: .2f}%&#39;.format(accuracy_train*100))</pre><pre>print(&#39;Test Accuracy:{: .2f}%&#39;.format(accuracy_test*100))</pre><p>An accuracy of 50%+ in test data suggests that the classification model is effective.</p><h3>Step 8: Prediction</h3><p>We will predict the signal (buy or sell) for the test data set, using the cls.predict() function. Then, we will compute the strategy returns based on the signal predicted by the model in the test dataset. We save it in the column ‘Strategy_Return’ and then, plot the cumulative strategy returns.</p><pre>Df[&#39;Predicted_Signal&#39;] = cls.predict(X)<br># Calculate log returns</pre><pre>Df[&#39;Return&#39;] = np.log(Df.Close.shift(-1) / Df.Close)*100</pre><pre>Df[&#39;Strategy_Return&#39;] = Df.Return * Df.Predicted_Signal</pre><pre>Df.Strategy_Return.iloc[split:].cumsum().plot(figsize=(10,5))</pre><pre>plt.ylabel(&quot;Strategy Returns (%)&quot;)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/0*FXDkVxRWoosE7MO2.jpg" /><figcaption>Strategy Performance</figcaption></figure><p>As seen from the graph, the <a href="https://goo.gl/NEQWMw">machine learning classification</a> strategy generates a return of around 15% in the test data set.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c641e38e4f40" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>