<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Srijit Mukherjee on Medium]]></title>
        <description><![CDATA[Stories by Srijit Mukherjee on Medium]]></description>
        <link>https://medium.com/@srijit-mukherjee?source=rss-a4bc34279e8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*w7jB1O-JeDY0ffzXJq7zzQ.jpeg</url>
            <title>Stories by Srijit Mukherjee on Medium</title>
            <link>https://medium.com/@srijit-mukherjee?source=rss-a4bc34279e8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:28:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srijit-mukherjee/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Quant Questions: The Sum Is Right]]></title>
            <link>https://srijit-mukherjee.medium.com/quant-questions-the-sum-is-right-a87dad706863?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a87dad706863</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[probability]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 02:31:07 GMT</pubDate>
            <atom:updated>2026-02-17T02:39:40.427Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, we will solve this problem in Order Statistics + Continuous Random Variables from Quant Questions step by step. If it helps you, please clap &amp; share it to others. Follow for more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y0qY5GXXFclP9t5TXpzBKQ.png" /><figcaption>Source: <a href="https://quantquestions.io/problems/the-sum-is-right">https://quantquestions.io/problems/the-sum-is-right</a></figcaption></figure><h3>Step by Step Solution:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JX5VqKUcLYWFduzXvesOzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lA9SEJqE0-fDwlnNKo96Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_Y783-oefQkEPITrGPGMQ.png" /></figure><p>Beautiful right?</p><h4>Step 4: Next time Calculations =&gt; Symmetry. So recognize it before.</h4><p>Solve the next problem! See you soon.</p><p><strong>Note</strong>: I first wrote the following text as my solution and used Gemini to generate the above solution in latex for better visibility. Feel free to copy the following text, and understand it in more details from your favourite LLM. All the best. Enjoy problem solving</p><h4>Step 1: Recognize the Underlying Concept</h4><p>This is an <strong>order statistics</strong> problem. You recognize this from the fact that you have to find some information regarding the distribution U and V, which are the min and max of the random sample {X_i}.</p><h4>Step 2: Probability Properties of U and V</h4><p>Once you recognize that this is an order statistics problem, you know the power of the following result.</p><ul><li>P[U ≤ u] = 1-(1-u)^n, and P[V ≤ v] = v^n</li><li>f_V(v) (probability density of V) = nv^(n-1)</li><li>f_U(u) (probability density of U) = n(1-u)^(n-1)</li></ul><p>for appropiate u &amp; v in the support. (if you don’t know, revise order statistics).</p><h4>Step 2: Find the distribution of U and V</h4><p>Observe that you have find the Probability that U+V &gt; 1. Do some range-based thought experiments like what is the range of U and V? Is there any mutual relationships? These may not be useful in solving directly, but can trigger ideas for solution. Observe that U (min) ≤ V (max) ≤ 1.</p><ul><li>P[U+V &gt; 1] = P[V &gt; 1 — U] = then conditional probabilities come up, and a lot of integration will come up naturally, and you will soon realize that this is not the right approach.</li><li>You will see that 1/2 is coming out to be final answer, if you dare to calculate so. I would suggest you to do that, atleast once, because, it will help you revise your conditional probability fundamentals &amp; also a lot of inequalities will be necessary to adjust the joint distribution range. So, it will teach you a lot.</li></ul><h4>Step 3: Think in terms of Symmetry</h4><p>When you ask your mind about symmetry, you will naturally tend to find symmetry in terms of the base distribution that is Uniform(0,1).</p><ul><li>The beautiful property of uniform is if X ~ U[0,1] =&gt; 1-X ~ U[0,1].</li><li>How can I incorporate this inside the min and max.</li></ul><p>Observe this simple property.</p><ul><li>min(1-x,1-y) = 1-max(x,y) and max(1-x,1-y) = 1-min(x,y)</li><li>P[U + V &gt; 1] can be written now in the following way.</li><li>P[U &gt; 1 -V] = P[min(Xi) &gt; 1-max(Xi)] = P[min(Xi) &gt; min(1-Xi)]</li><li>Let’s call Yi = 1-Xi. Note that Yi ~ U[0,1], and Xi = 1-Yi. This leads to a beautiful result.</li><li>We are required to find P[min(Xi) &gt; min(Yi)]. Now this should be exactly same as P[min(Yi) &gt; min(Xi)].</li><li>Why? The distribution (Xi,Yi) is same as (Yi,Xi) because of the symmetric property. More generally,</li><li>The vector (X1, … , Xn) and the vector (1-X1,…, 1-Xn) are identically distributed.</li><li>P[min(Xi) &gt; min(Yi)] + P[min(Yi) &gt; min(Xi)] + P[min(Yi) = min(Xi)] = 1 because that are the only possibilities.</li><li>Since min(Xi) and min(Yi) are continuous distributions, min(Yi) = min(Xi) has probability 0.</li><li>So P[min(Xi) &gt; min(Yi)] + P[min(Yi) &gt; min(Xi)] = 1, and each term is equal so P[min(Xi) &gt; min(Yi)] = P[U &gt; 1-V] = P[U+V &gt; 1] = 1/2 = 0.5</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a87dad706863" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who is stealing your attention?]]></title>
            <link>https://srijit-mukherjee.medium.com/who-is-stealing-your-attention-900a98748e71?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/900a98748e71</guid>
            <category><![CDATA[life-lessons]]></category>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[personal-growth]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 12:32:50 GMT</pubDate>
            <atom:updated>2023-01-22T12:32:50.349Z</atom:updated>
            <content:encoded><![CDATA[<p>Attention is valuable. Why? Because it is equivalent to your time. But, what’s drawing your focus away?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eByyNbfpyue5OZusdbSFEw.png" /></figure><p>Your two most important resources.<br>Time &amp; Money.</p><p>Everybody is seeking your attention.<br>Why? <br>Because they earn money from your attention.<br>How?<br>Simple.</p><p>You give attention to something.<br>They know your psychology quite well.<br>The “something” keeps you hooked.<br>It takes you on a journey.<br>The journey is never-ending.<br>In the journey.<br>They share ads.<br>The ads go deep into your mind.<br>After 2 months, you will buy it.<br>Probably. Or not!<br>And?<br>You lost 3 hours of your time.<br>Congratulations.</p><p>Imagine how many ads are shared.<br>Before a film starts.<br>In a cinema hall.</p><p>They are not wrong.<br>It is their duty. <br>To make their product attractive.<br>To make it the best.</p><p>It’s your fault.<br>You failed to keep your attention to yourself.<br>You act as if.<br>They have cast a magic spell on you.<br>It’s BS.</p><p>You should ask yourself?<br>Why am I not attentive?<br>Am I avoiding something?<br>What am I looking for?<br>Do I have some other important goals?<br>Why am I not working on the goals?<br>Do I have some goals without tech?</p><p>Go ahead.<br>Do them.<br>Take action.<br>Do something.<br>Don’t scroll.<br>Be responsible.</p><p>Love,<br>Srijit.</p><p>If you like reading this, support my work by clapping for this article, and sharing your views in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=900a98748e71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What to do before asking a question?]]></title>
            <link>https://srijit-mukherjee.medium.com/what-to-do-before-asking-a-question-5c5da9b1bbcf?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c5da9b1bbcf</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[success]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[life-lessons]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Fri, 20 Jan 2023 04:17:55 GMT</pubDate>
            <atom:updated>2023-01-20T04:17:55.247Z</atom:updated>
            <content:encoded><![CDATA[<p>Every day. Many people get in touch with me. I find a typical pattern.</p><p>They get in touch with me.<br>For doubts, help, and queries.</p><p>I find a typical pattern.<br>They come to me with a problem.<br>The solution is just one click away.</p><p>Instead of being calm.<br>and thinking rationally.<br>Out of panic.<br>They text me.</p><p>I understand them.<br>I understand where they are coming from.</p><p>I ask them.</p><p>“Did you search this in google?”<br>and they reply back to me.<br>“No”</p><p>I have to tell someone every day at least once.<br>“Please search in Google.”<br>with the phrase ‘….’</p><p>Are you being curious enough?<br>Are you asking the right questions?<br>Are you asking the right person?<br>Are you asking questions in a proper way?<br>Are you using the resources properly?</p><p>Next time, before asking someone a question<br>Stop.<br>First, search for it on google.<br>80% of the time.<br>I get the solution.</p><p>Have a great day.<br>Love,<br>Srijit.</p><p>If you like to read this, do share your support by clapping for the piece of writing. You can subscribe to my newsletter <a href="https://www.srijitmukherjee.com/newsletter/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VMryV7hOZ0qsudAhcuOJZQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c5da9b1bbcf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My mind is going crazy. I cannot focus. Please help.]]></title>
            <link>https://srijit-mukherjee.medium.com/my-mind-is-going-crazy-i-cannot-focus-please-help-2cdeee2ea7d8?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cdeee2ea7d8</guid>
            <category><![CDATA[spirituality]]></category>
            <category><![CDATA[thoughts]]></category>
            <category><![CDATA[mind]]></category>
            <category><![CDATA[mindfulness]]></category>
            <category><![CDATA[brain]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Sun, 26 Sep 2021 06:36:57 GMT</pubDate>
            <atom:updated>2021-09-26T06:36:57.740Z</atom:updated>
            <content:encoded><![CDATA[<p>We were all there.</p><p>We still experience this almost often.</p><p>The unhappiness and dissatisfaction stems from the fact that</p><blockquote>We want to focus.</blockquote><blockquote>We know where to focus.</blockquote><blockquote>But we cannot focus.</blockquote><p>Let me give you a similar real-life example, and find a solution from the real-life only.</p><p>You want to drink pure boiled water.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*ufDFLNjK2cJlyBcZ.gif" /></figure><p>So, you put a pan full of water, turn the heat on, till the moment, it starts boiling.</p><p>Now, do you drink at this point in time?</p><h3>No, right?</h3><p>Your mind is exactly behaving like this boiling water. Suddenly, boiling bubbles of thoughts come to the surface of your mind out of nowhere, and it keeps on coming and coming.</p><blockquote>You cannot effin’ stop it.</blockquote><p>Your goal/focus is to drink the water. But, you cannot drink it, because it’s boiling and the hot steamy bubbles don’t stop coming.</p><h3>What will you do to cool down the boiling water?</h3><h4>A Slow Method</h4><p>It is very simple. You turn off the heat. You wait for some time, till the water comes to a normal temperature.</p><h4>A Fast Method</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*tREuhJLKBRy1kVMY.gif" /></figure><p>After turning off the heat, you try to dissipate the heat into the atmosphere. We pour the water back and forth between two pans.</p><h3>We will exactly do the same thing with our minds.</h3><p>We have to map the corresponding objects over here.</p><ul><li>Heat =?</li><li>How to reduce the Heat =?</li><li>Water = Mind</li><li>Bubbles = Thoughts</li><li>Dissipating the Heat =?</li></ul><h3>Heat</h3><p>The heat is nothing but your surroundings, the atmosphere around you. The mind gets heat (information) from the surroundings and starts to slowly heat itself up, till it reaches that boiling point.</p><h3>How to reduce the Heat? → Step 1</h3><p>You just shut off everything that distracts you. The mind gets 80% of the information from the eyes (visual aspect). So, the best thing to reduce the heat is to close the eyes, and just let the thoughts subside. Don’t think over the thoughts, just let them subside.</p><blockquote>Imagine the bubbles of thoughts slowly subsiding, and the water becoming silent and calm. Imagine. Imagine.</blockquote><p>This will take some time (15 minutes). Your mind is calm now.</p><blockquote>Master this step 1.</blockquote><h3>You can do it faster. Dissipating the Heat. → Step 2</h3><p>Dissipating the heat is an intelligent, scientific, and smart way of removing the heat.</p><p>Once you are practicing step 1, add another layer to it.</p><p>The core idea is that your heat needs to dissolve. It will only dissolve if you dilute it faster.</p><p>The ideal way to dilute the external information is by reducing it, by transforming your focus on <strong>breathing</strong>.</p><blockquote>Imagine, you breathe in from the left nose, the oxygen goes up into the brain, the mind, it dissolves all the heat, all the thoughts, all the data. Breathe out through the right nose, and imagine it is taking all the heat out with it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*tREuhJLKBRy1kVMY.gif" /><figcaption>Imagine this picture in your mind.</figcaption></figure><h4>Keep on doing this, till your mind becomes calm.</h4><blockquote>Be grateful for everything.</blockquote><blockquote>Thank everyone and everything around you.</blockquote><blockquote>Then, again focus on your goal, and keep moving ahead.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cdeee2ea7d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to be a Happy Edupreneur?]]></title>
            <link>https://srijit-mukherjee.medium.com/how-to-be-a-happy-edupreneur-49a787d83a86?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/49a787d83a86</guid>
            <category><![CDATA[business-strategy]]></category>
            <category><![CDATA[enterpreneurship]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Tue, 01 Jun 2021 07:01:50 GMT</pubDate>
            <atom:updated>2021-06-02T14:53:02.838Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FIYIdl8Ttos4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DIYIdl8Ttos4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FIYIdl8Ttos4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/75771225b8519d2a6842ff65dcc77caf/href">https://medium.com/media/75771225b8519d2a6842ff65dcc77caf/href</a></iframe><p>I have given a live webinar on this topic here with Mathematical Spaces.</p><p>You can watch it here for a better impact.</p><p>I am an edupreneur.</p><p>I am happy.</p><p>This article is all about the theory and the practical steps in becoming a happy edupreneur.</p><h3>Who is an edupreneur?</h3><p>An entrepreneur, who works in the education domain.</p><h3>Who is an entrepreneur?</h3><p>An entrepreneur is someone, who</p><ul><li>Solves a problem in society and, adds value to the society.</li><li>Builds a proper team for the goal, and creates <strong>employment opportunities.</strong></li><li>Makes a dedicated process towards the growth of the product and the team.</li></ul><h3>What should be the mentality of a Happy Entrepreneur?</h3><p>To impact the lives of the people positively, with an abundance mindset. This will make you happy. This will make sow the seeds of an entrepreneur in you.</p><h3>Two Equivalent Theories of Business, I follow</h3><h4>Simon Sinek’s Start With Why</h4><ul><li>Why → The Goal</li><li>What → Specify the Goal</li><li>How → The Process and the Team</li></ul><h4>3 Ps of Business</h4><ul><li>Product → Your Why, The Problem’s Solution</li><li>People → Internal People, External People</li><li>Process → A Well Defined Process</li></ul><h3>3D1P →The Theory of Resource Management</h3><p>There are two resources:</p><ul><li>People</li><li>Time</li></ul><p>Make sure you properly use it.</p><h4>The 3D1P Theory for Process and People Management</h4><ul><li>Define → Define the tasks explicitly, aligned with the goal.</li><li>(People) Divide → Distribute the tasks among the workforce.</li><li>(Time) Deadline → Give a deadline to the tasks.</li><li>Progress → Check Progress. (This is boring, but a decisive step to evaluate the team.)</li></ul><h3>How to make the product better?</h3><ul><li>Hit and Trial with the ideas.</li><li>Take feedback from the customers and people around you.</li><li>Be consistent.</li></ul><h3>How to startup?</h3><h3>Step 0</h3><h3>Don’t stay hungry.</h3><p>Make sure, when you are starting up, you have a consistent “okay” flow of income. Good things take time. You have to keep your bowl non-empty, every day till things start happening. If not, make sure it happens, before you move on to the next step.</p><p>I made sure that I have a minimal steady flow of income by teaching students, while on the other side, I started exploring the opportunities.</p><h3>Step 1</h3><h3>Find a Problem.</h3><p>It is really easy to find a problem.</p><p>Look around. Observe what can be made better.</p><p>What is done in the wrong way?</p><ul><li>Geography is not taught in the best possible illustrative way.</li><li>Mathematics can be made fun using animation and studies.</li><li>Music is not taught online in an effective and interesting way.</li><li>Preparation for jobs is not fun and enjoyable.</li></ul><p>All the fellow teachers, remember the following</p><blockquote>Every job opportunity builds an opportunity for a teacher to teach a person on how to get into that job.</blockquote><p>I found that statistics were not taught in a proper way for the entrances in masters, and not at an undergraduate level. In fact, students called me up and told me the same.</p><ul><li>Talk to people to understand the problems.</li><li>Build an account on Linkedin, to connect to such people.</li><li>Observe what people are doing around you.</li><li>Observe what people are not doing around you, but it is important.</li><li>Bravo, go ahead and start experimenting.</li></ul><h3>Step 2</h3><h3>Test your ideas.</h3><p>This is the most important. This is the most exciting. This is the hardest of all.</p><p>This step will make you understand, whether your idea is good or not.</p><h4>How will a shopkeeper understand whether a new product is good or bad?</h4><ul><li>He/She must set up a shop so that people can easily find it out.</li><li>He/She must test it out himself/herself.</li><li>He/She must make the product available for a minimal cost, to check it out, if the customers are liking it or not. Is it really effective?</li><li>He/She must go ahead on making the product available more, attract more customers, place better competitive prices on it.</li></ul><h3>How can an edupreneur test and build it out?</h3><h4>Build a shop, people can find it out easily.</h4><p>Build an online shop. Make a website. Give a domain name. Learn SEO. Apply SEO, so that people can find it out, easily, when they need something.</p><h4>Build Content for Free according to your niche.</h4><p>The shopkeepers can test out a new biscuit, or drink. Then, he/she can sell it with assurity.</p><p>But, here you have to test your ideas and innovative methodologies with the people. This is sort of Beta Testing of your ideas, services, and product.</p><p>I built content for free for 100+ days, with SEO to understand, whether it really works.</p><h4>Accept the Truth</h4><p>If people love it, bravo! If not make your product better. Test it out. But, don’t fall into sunk cost bias.</p><p>If it doesn’t really work out, go ahead to test another idea.</p><p>I had been making content for free. I had a mindset, that if it doesn’t work out, I have added something positive to the world. This abundance mindset will help a lot during this phase or any phase of testing the product and services.</p><h4>Go ahead, and build it.</h4><p>In the process, you will reach a lot of enthusiastic persons, who will comment, who will like your work. You can engage them in the team somehow. They are well-wishers of your ideas.</p><p>Don’t start building a team just from the beginning, unless you know someone for a long time.</p><p>Go ahead and build a paid + volunteering team to help the idea, product and service grow for the people, for the customers.</p><ul><li>Ask them, what are mistakes you are making.</li><li>Ask them, what can be improved.</li><li>Take feedback.</li></ul><h3>Remember the following.</h3><blockquote>Content Build Relationships.</blockquote><blockquote>Relationships build Trust.</blockquote><blockquote>Trust generates Revenue.</blockquote><h4>Build the customer database by a Call To Action.</h4><p>To investors, the customer database is the most important criterion.</p><p>When you are creating something for free, always make sure, you keep a call to action, like fill-up the form, for building up the database.</p><p>This helps, when you are planning to start a course or something interesting, you can reach out to them.</p><ul><li>Build Whatsapp Groups</li><li>Build Telegram Groups</li><li>Build Email List</li><li>Build Facebook Groups</li><li>Build your Followers</li></ul><p>All these things are the same.</p><p>This database behaves like compound interest when you offer a good service or product.</p><h4>Measure your Progress</h4><blockquote>If you have lost your focus, then you are not measuring your progress.</blockquote><blockquote>If you are not measuring your progress, you have lost focus.</blockquote><p>Measure your progress by the numbers and progress bars I have mentioned above.</p><h3>Build yourself, and your product.</h3><p>Build yourself. Inspire others. Make the world a better place. Don’t forget about this.</p><p>This will also pave the way for other ideas, other products, and will make your existing product better.</p><p>Take feedback. Talk to people to understand your product better.</p><p>Go on experimenting with your ideas, till you find the one.</p><h3>All the best.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xIJRPUCHKpKwRqEx.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49a787d83a86" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AR, MA, and ARIMA Models: A Comprehensive Guide]]></title>
            <link>https://medium.com/srijit-mukherjee/ar-ma-and-arima-models-a-comprehensive-guide-46b1db00083?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/46b1db00083</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Sat, 22 May 2021 10:29:48 GMT</pubDate>
            <atom:updated>2021-05-22T10:29:48.463Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/srijit-mukherjee/time-series-a-comprehensive-guide-577e6debf8a3">first part of this series</a>, I have discussed the following questions.</p><ul><li>What is Time Series?</li><li>What is the main focus of Time Series?</li><li>How is Time Series different from Regression?</li><li>How to mathematically model Time Series?</li><li>Why stationarity of the Time Series?</li><li>What is the Central Idea (the two fundamental steps)?</li><li>What is ARIMA modeling in short?</li></ul><h3>There are two broad steps in Time Series.</h3><h3>Step 1</h3><pre>Exploratory Data Analysis and Transform data into stationary data.</pre><h3>Step 2</h3><pre>Model and Predict the dependence structure of the errors.</pre><h4>In this article, I will discuss Step 2, using AR, MA, and ARIMA.</h4><p>I will discuss the following questions:</p><ul><li>What is stationary data?</li><li>What are the components behind prediction?</li><li>How future data depend on the past errors made?</li><li>What is MA(q)?</li><li>What is AR(p)?</li><li>How to get a good estimate of q and the coefficients in MA(q)?</li><li>Why do we need PACF?</li><li>How to get a good estimate of p and the coefficients in AR(p)?</li><li>What is ARMA(p,q)?</li><li>How to estimate the parameters of ARMA(p,q)?</li><li>What is Ljung Box Test?</li><li>What is ARIMA(d,p,q)?</li><li>How to estimate the parameters of ARIMA(d,p,q)?</li></ul><h3>What is Stationary Data?</h3><p>After step 1, we get stationary data. Let’s say Y is a “stationarized” data</p><p>A “stationarized” time series has</p><ul><li>no trend,</li><li>a constant variance over time, and</li><li>constant “wiggliness” over time, i.e., its random variations have a qualitatively similar pattern at all points in time if you squint at the graph.</li></ul><p>In technical terms, this means that its autocorrelations are constant over time.</p><p>At this point, we have stationary data (say zero mean), with no seasonality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*p8AIjM7_zpSt7rNy0rEk3A.png" /><figcaption>is the stationary data, we have now.</figcaption></figure><p>We also have the white noise in the prediction, the uncorrelated errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*lV0IP3c7xwxNqrjYzIjK6Q.png" /><figcaption>is the sequence of white noise present in prediction.</figcaption></figure><h4>Aim: Predict the future, given past data and past prediction errors.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*pAd1GvVRB7V1bHrSFRgX1g.png" /><figcaption>The future | past data, past errors in prediction</figcaption></figure><h3>What are the components behind prediction?</h3><p>Understand that prediction needs to take care of the following.</p><ul><li>Future Data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/89/1*oNPOcEUV6SnXBHxvTyX9DA.png" /><figcaption>Future Data</figcaption></figure><ul><li>To how much extent the future data is dependent on the past data?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*p8AIjM7_zpSt7rNy0rEk3A.png" /><figcaption>past data</figcaption></figure><ul><li>To how much extent the past predictions are doing well and correspondingly change the prediction style?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*1SA_L6i3m4Q5VvDaPIkJhQ.png" /><figcaption>Prediction errors</figcaption></figure><h3>How future data depend on the past errors made?</h3><p>Let’s say, you have observed that in the past 2 days, you have observed a consistent error in prediction around negative 20%, so in the next day, you must also predict not the actual predicted value, but the actual predicted value with a reduction in 20% following the behavior of the errors.</p><p>Hence, we come to the Moving Average method.</p><h3>What is MA(q) (Moving Average)?</h3><p>In moving average, we check how the stationary time series is dependent on the errors, in an additive way.</p><p><strong>q</strong>: denotes the number of past errors the future is dependent upon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*ome4IirWunzQ28wPaSrHuw.png" /><figcaption>The q, thetas are the parameters, we need to estimate.</figcaption></figure><p>We will come to the ideas of how to estimate this <strong>q </strong>and the<strong> thetas </strong>from data<strong>.</strong></p><h3>What is AR(p) (Autoregressive)?</h3><p>In autoregressive, we check how the stationary time series is dependent on the past data, in an additive way. This is exactly like the multivariate regression step, hence the name autoregressive.</p><p><strong>p</strong>: denotes the number of past data the future is dependent upon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*uEOcLe-ZsKnc6HfBYc-CrQ.png" /><figcaption>The p, thetas are the parameters, we need to estimate.</figcaption></figure><p>We will come to the ideas of how to estimate this <strong>p</strong> and the<strong> thetas</strong> from data<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*0_drs8GQ1WjjRktxktiqwQ.png" /><figcaption>See how the positive effect on just the previous data results in a continuation of similar paths.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*b70z9-ZO-EQiMY_iSI5W5A.png" /><figcaption>See how the negative effect on just the previous data results in a change of signs so often.</figcaption></figure><h3>How to get a good estimate of q and the coefficients in MA(q)?</h3><p>If you calculate the ACF function of MA(q), it will be 0 after time lag = q.</p><p>The cutting off of ACF(h) after q lags is the MA&#39;s signature (q) model.</p><p><strong>Examples</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*kGaxBEAhHg2EbMkCeR8AUA.png" /><figcaption>Observe that the ACF lies outside the two confidence interval bars only till MA(1).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ShD-HIqWEJovEkyfqtvwPg.png" /><figcaption>Observe that the ACF lies outside the two confidence interval bars only till 2. Hence MA(2)</figcaption></figure><p>The coefficient estimation is done by</p><ul><li>Ordinary Least Square</li><li>Fast Filtering Algorithm, etc</li></ul><h3>Why do we need the Partial Autocorrelation Function (PACF)?</h3><p>If we try to get a good estimate of p of the AR(p) model, the ACF doesn’t give any insight, because it behaves similarly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*izW_EnEa4ltYJ_veZLPX6Q.png" /></figure><p>So, we need another tool, which can capture the relationship between the future data and the past data points.</p><h4>What is PACF (Partial Autocorrelation Function)?</h4><p>In general, a partial correlation is a conditional correlation. It is the correlation between two variables under the assumption that we know and takes into account the values of some other set of variables.</p><blockquote>For a time series, the partial autocorrelation of lag h, between the data point at <strong>t</strong>, and time<strong> t-h, </strong>is defined as the conditional correlation between the data point at <strong>t</strong>, and time<strong> t-h, </strong>conditional on the set of observations that come between the time points t and t−h.</blockquote><p><strong>2nd Order Lag PACF</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*ZeRTdWXkuXlBPtm6eOdi5g.png" /><figcaption>2nd order lag PACF</figcaption></figure><p><strong>3rd Order Lag PACF</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*VdCGPnm3Q-EaEAeLY-UDhQ.png" /><figcaption>3rd order lag PACF</figcaption></figure><p>For an AR model, the theoretical PACF “shuts off” past the order of the model.</p><pre>Exercise: Think, why PACF gives a better idea than ACF?</pre><h3>How to get a good estimate of p and the coefficients in AR(p)?</h3><p>If you calculate the PACF function of AR(p), it will be 0 after time lag = p.</p><p>The cutting off of PACF(h) after p lags is the AR’s signature (p) model.</p><p><strong>Examples</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*Oed9fw1_naEshAVrLDmKGQ.png" /></figure><p>The coefficient estimation is done by</p><ul><li>Ordinary Least Square</li><li>transient Kalman gain, etc</li></ul><h3>What is ARMA(p,q)?</h3><p>When the AR(p) and the MA(q) models are combined together to give a general model, we call it ARMA (p,q) to model stationary nonseasonal time series data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*JNWp9xhLz12W2xPC6BMbpA.png" /><figcaption>p past data and q prediction errors -&gt; future data</figcaption></figure><p>We need to estimate the parameters of ARMA (p,q) now.</p><h3>How to estimate the parameters of ARMA(p,q)?</h3><p>We have understood the parameters p, q by observing the ACF and PACF plot.</p><p>But, we will discuss a general algorithm now.</p><p>This is a model selection problem. We minimize the BIC (Bayesian Akaike Criterion) and select among all the models, the one with minimum BIC.</p><p>In order to determine which order p,q of the ARMA model is appropriate for a series, we need to use the AIC (or BIC) across a subset of values for p,q.</p><blockquote>But, now after the fitted model with ARMA, how do we know that the errors are not again autocorrelated?</blockquote><h3>What is Ljung Box Test?</h3><p>The <strong>Ljung-Box test</strong> is a classical hypothesis test that is designed to test whether a set of autocorrelations of a fitted time series model differ significantly from zero.</p><p>The test does <em>not</em> test each individual lag for randomness but rather tests the randomness over a group of lags.</p><p>We define the null hypothesis H0 as The time-series data at each lag are iid that is, the correlations between the population series values are zero.</p><p>We define the alternate hypothesis Ha as The time series data are not i.i.d. and possess serial correlation.</p><p>So, after fitting the ARMA(p,q) model, we must apply the Ljung-Box test to determine if a good fit has been achieved, <em>for particular values of p,q</em>.</p><h3>How to estimate the parameters of ARIMA(d,p,q)?</h3><h3>ARIMA = AR + I + MA = I + ARMA</h3><p>ARIMA is actually to model a time series with a trend added with stationary errors.</p><h4>Step 1</h4><p>By differencing in I step, first we detrend the time series to get the stationary time series errors.</p><h4>Step 2</h4><p>Then, we apply ARMA modeling to this remaining portion.</p><blockquote>Simple and Elegant.</blockquote><p>I will next share the practical approach to do Time Series Analysis which will include Step 1 and the fitting of the ARIMA model.</p><p>Stay Tuned. Stay Blessed.</p><p>Don’t forget to clap and follow, if you have enjoyed reading this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46b1db00083" width="1" height="1" alt=""><hr><p><a href="https://medium.com/srijit-mukherjee/ar-ma-and-arima-models-a-comprehensive-guide-46b1db00083">AR, MA, and ARIMA Models: A Comprehensive Guide</a> was originally published in <a href="https://medium.com/srijit-mukherjee">Srijit Mukherjee Content</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series: A Comprehensive Guide]]></title>
            <link>https://medium.com/srijit-mukherjee/time-series-a-comprehensive-guide-577e6debf8a3?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/577e6debf8a3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Fri, 21 May 2021 20:08:06 GMT</pubDate>
            <atom:updated>2021-10-26T08:39:57.375Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_6CPyF4OifM&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_6CPyF4OifM&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F_6CPyF4OifM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7806350a8e9c6cb1eebc85c49f6123d/href">https://medium.com/media/e7806350a8e9c6cb1eebc85c49f6123d/href</a></iframe><h4>You can listen to this youtube tutorial (with clarity and better intuition) of the same thing, and/or read the whole article.</h4><h3>We discuss the following</h3><ul><li><a href="#0c38">What is Time Series?</a></li><li>What is the main focus of Time Series?</li><li>How is Time Series different from Regression?</li><li>How to mathematically model Time Series?</li><li>Why stationarity of the Time Series?</li><li>What is the Central Idea?</li><li>What is ARIMA modeling in short?</li></ul><h3>What is Time Series?</h3><h4><strong>Time series data is a sequence of data over discrete time points.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*fpVWlwWflMoEV1-82lgpiw.png" /><figcaption>are the data points over time, where 1,2,3,..,n are the time points.</figcaption></figure><p>In terms of probability, X is called a random variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eN7RLe6LZHPh_-Hh.png" /><figcaption>The data are collected at discrete time points, which are very close.</figcaption></figure><pre>Time series is an application of the Stochastic Process in probability.</pre><h3>What is the main aim of Time Series Analysis?</h3><ul><li>Forecasting is the main focus of time series analysis by measuring and estimating the dependence among the data points.</li></ul><p>This leads us to the next question.</p><h3>How is Time Series different from usual Regression?</h3><h4>Regression Framework</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*8YL8utZdZ7GaMjXdKjFcZg.png" /><figcaption>There are n such data points. The aim is to approximate such a function f from the data.</figcaption></figure><p>One important assumption is that the errors are independent of one another, which in turn means that the data points are independent (<strong><em>white noise</em></strong>).</p><p><strong>In time series, the errors are in fact dependent upon one another, as time proceeds, which is a really genuine assumption.</strong></p><blockquote>Thus, we cannot apply the usual regression techniques in Time Series Analysis.</blockquote><h3>How to mathematically model Time Series?</h3><h4><strong>We need the following instruments and ideas.</strong></h4><h4>1. White Noise (to model random fluctuations)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/1*ylmZ6BkktVjfLmwyBOoNEw.png" /><figcaption>White noise is independent and they are identically distributed.</figcaption></figure><h4>How does normal white noise look like over time?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*gLeevd0LCyRZKbcqLTnr2Q.png" /></figure><h4>2. Trend, Seasonality, and then Detrending of Time Series</h4><p>The trend is actually the f(x), which captures the general overall behavior.</p><p>There are mathematical methods to identify the trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*YvsGGY8ut6IKYBc9VeDryA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*fNloxhxWA5xrEWbQWLhYdQ.png" /></figure><p>The relationship of the trend can be super complex why?</p><p>Because, one time series can be dependent upon other time series, with some extra component of its own trend, added with the relationship with other time series.</p><p>Yeah, it is complex in nature.</p><p><strong>How to handle this?</strong></p><p>The way to handle this is that we have to detrend (remove the trend) to get residuals and analyze it.</p><blockquote>The most important aspect of the time series is not to model the trend, but to model the heavily dependent errors.</blockquote><p>I will discuss one way of doing it.</p><ul><li><strong>Differencing</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*fNloxhxWA5xrEWbQWLhYdQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*sGMQEu5t-1TzYZAjSEPFMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*7upMu4NAKev7TlaV8klYuA.png" /></figure><p>We now try to remove the trend by subtracting the trend equation, and by difference.</p><p>Look into the pictures below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*lCF6P1GEpX--gqiBF45pMA.png" /></figure><p>Observe still there is some order in the detrended (by linear equation) while the first difference looks more random like an error.</p><p>This is because, by difference, we have also accounted for the <strong>seasonality </strong>aspect of the time series.</p><p>This forms the <strong>I </strong>aspect of the <strong>ARIMA </strong>model.</p><p>I will discuss this in detail later on in a different post.</p><blockquote>Now, after detrending the whole time series, we are left with the residuals/sample errors, which have a dependence structure and they must have some good properties, which we will explore.</blockquote><h4>3. Dependence Structure (of errors of data points unlike regression)</h4><p>The lack of independence between errors of two data points can be assessed numerically, as in classical statistics, using the notions of covariance and correlation.</p><h4>Autocovariance Function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*amnprE73W0ZvH-HfGvOznQ.png" /><figcaption>Autocovariance Function of two data points</figcaption></figure><p>The autocovariance function measures the dependence between errors of two data points at time t and time s.</p><p>But, covariance can be very large in magnitude. So, we need to normalize it, to make it autocorrelation function.</p><h4>Autocorrelation Function (ACF)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*y9unH_CG7hPiLl7tggb8Ig.png" /><figcaption>Autocorrelation Function of two data points</figcaption></figure><p>The autocorrelation function (ACF) measures the normalized dependence between two data points at time t and time s. It lies between 1 and -1. (Why?)</p><p>Understand, that we need to estimate the autocorrelation function from data using the sample correlation.</p><h4>3. Properties of Good (Predictable) errors of Time Series data</h4><h4>- Good(Predictable) errors of Time Series is stationary</h4><p>A time series (of errors) are behaving in a nice way if any collection of them are invariant over time (stationary) in a probabilistic sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*FFIJBppiDdJM4wfCAPoCgQ.png" /><figcaption>This is the initial collection of data points.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*RZKa9ybKCRZvXMg94QNoyQ.png" /><figcaption>This is the collection of data points of h time point shifted.</figcaption></figure><p>These two sets of data points must behave similarly (stationary) in a probabilistic framework.</p><p>But, this hardly happens in real life and it is very hard to check.</p><blockquote>So, we will focus on two conditions, where we can understand the first-order moment (mean) and the second-order dependence moment (covariance) behaviour.</blockquote><ul><li>The mean value function must remain <strong>constant </strong>over time. We can subtract a constant from it to make it 0 always.</li><li>The autocovariance function (the dependence structure) of two data points at time<strong> t and s</strong>, must be only dependent on the time lag <strong>|t-s|.</strong></li></ul><pre>Trend is not stationary, since the mean function is changing with time. So, we need to detrend to make it stationary.</pre><p>This implies that the dependence function of the errors is based on a good behavioral notion of <strong>stationarity</strong>.</p><p>We will learn various interesting methods to model this dependence using ACF.</p><p>In the case of stationarity, the ACF function reduces to a function of the time lag (h) only.</p><p>Why? Watch the video. :)</p><h4>- Autocorrelation Function (ACF) decreases over time.</h4><blockquote>This intuitively means, far apart data points are hardly correlated and dependent upon one another.</blockquote><p><strong>Example</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*ImkMCOuqG5SXGO4wTw_OXw.png" /><figcaption>Example 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*povn_WoTqYHbVTYf_5o3RQ.png" /><figcaption>Lag = t-s, where we understand the dependence of data points, Xt and Xs by ACF. Observe that the estimated ACF function decreases with an increase in the magnitude of lag.</figcaption></figure><h3>Why stationarity of the Time Series?</h3><p>The stationarity of time series helps us to estimate the dependence structure and the mean behavior of all the possible sample collection.</p><p>This is turn helps us to understand to how many degrees the data points are connected to each other (in terms of lag) and this forms the basis of the <strong>AR </strong>and <strong>MA </strong>aspect of the <strong>ARIMA </strong>model.</p><h3>What is the Central Idea?</h3><h4>→ Time Series = Trend + Season + Dependent Error</h4><h4>Step 1</h4><pre>Transform data into stationary data.</pre><ul><li>In the case of polynomial trend, we can difference many times to detrend.</li><li>In the case of exponential trend, we need to non-linearly transform the data into polynomial trend and then difference.</li><li>There are other methods to detrend.</li><li>There are other smoothing methods to understand the trend and then detrend (Nadarya Watson Estimator, etc).</li><li>Check various plots to understand graphically and do some statistical checks to check stationarity.</li></ul><p>We will learn these in detail in another post. Here, we essentially prepare the data for the understanding of the dependence structure of step 2.</p><h4>Step 2</h4><pre>Model and Predict the dependence structure of the errors.</pre><p>The modeling of the dependence is done by the famous models like</p><ul><li>AR</li><li>MA</li></ul><h3>What is the ARIMA model in short?</h3><ul><li>Change data into stationary data by transformation and differencing. (<strong>I</strong>)</li><li>Apply <strong>AR </strong>and <strong>MA </strong>models to estimate the dependence structure of the differenced time series and predict.</li><li>That’s AR + I + MA = ARIMA for you.</li></ul><h3>Read about AR, MA, and ARIMA below.</h3><figure><a href="https://medium.com/srijit-mukherjee/ar-ma-and-arima-models-a-comprehensive-guide-46b1db00083"><img alt="" src="https://cdn-images-1.medium.com/max/180/0*2eYumgwk-8-uFvzH" /></a></figure><h3>Click here on the left.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=577e6debf8a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/srijit-mukherjee/time-series-a-comprehensive-guide-577e6debf8a3">Time Series: A Comprehensive Guide</a> was originally published in <a href="https://medium.com/srijit-mukherjee">Srijit Mukherjee Content</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID19’s effect on airport traffic: A Data Analysis in R]]></title>
            <link>https://medium.com/srijit-mukherjee/covid19s-effect-on-airport-traffic-a-data-analysis-in-r-6175c0f0901a?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6175c0f0901a</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Fri, 21 May 2021 04:58:36 GMT</pubDate>
            <atom:updated>2021-05-21T05:02:29.479Z</atom:updated>
            <content:encoded><![CDATA[<p>Download the data from <a href="https://drive.google.com/drive/folders/1LWFC344Ng-M8JborpJ2qoeQ8vIkesYi2?usp=sharing">here</a>.</p><h3>Load your data (csv format)</h3><pre>data = read.csv(&quot;location/filename.csv&quot;)</pre><h3>Exploration of Data</h3><h3>Observing the data</h3><pre>head(data) <br>tail(data)</pre><h3>Variable Types</h3><pre>str(data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*Q-eQhTZgVqM7J-e2hqGrQw.png" /></figure><h3>Understanding the Data</h3><h4>Important Variables</h4><h4>Numerical Variables</h4><ul><li>PercentOfBaseline</li></ul><h4>Categorical Variables</h4><ul><li>Date</li><li>AirportName</li><li>Centroid (of airports)</li><li>City</li><li>State</li><li>Country</li><li>Geography (of countries)</li></ul><h4>Comments</h4><ul><li>This is time-series data.</li><li>The centroid and geography will be only helpful if we want to visualize the data on the world map.</li><li>Country → State → City → AirportName are factor variables inside the next one.</li></ul><h3>Focus</h3><ul><li>Data Visualization is the main focus of the project.</li><li>Due to the structure of Country → State → City → AirportName, the best way to visualize it in an app with dropdown options. (R Shiny App, Tableau, Power BI, etc.)</li><li>Analyze each of the individual time series and forecast for the next “n” number of days.</li><li>Data visualization of categorical variables and their relationship is not important, because they are interrelated and give no valuable insight.</li></ul><h3>Exploratory Data Analysis</h3><h3>Univariate (Kernel Density of PercentOfBaseline)</h3><pre>library(ggplot2)<br>ggplot(data, aes(x = PercentOfBaseline, fill = &quot;red&quot;))+geom_density(alpha = 0.2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*akSquEL6-RJ35mztoXi31A.png" /></figure><p><strong>Exercise</strong>: Plot individual kernel density of each factor of Country, State, City, and AirportName.</p><h3>Bivariate</h3><h3>Numerical vs Categorical</h3><h4>- PercentofBaseline vs Country</h4><pre>ggplot(data, aes(x = PercentOfBaseline, fill = Country))+geom_density(alpha = 0.2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*zOGHO8U-pp2MUK5MXgq_OQ.png" /></figure><h4>Observations:</h4><ul><li>All the countries are behaving like normal mixtures with multiple modes.</li><li>Australia has two distinct modes of normal mixtures. (Why?)</li><li>Canada has a higher PercentOfBaseline than most of the other countries.</li></ul><p><strong>Exercise</strong>: Plot and compare kernel density of PercentOfBaseline of factors of Country, State, City, and AirportName.</p><h3>- PercentofBaseline vs Date (Time Series)</h3><pre>ggplot(data, aes(x =Date, y = PercentOfBaseline, group = Country, colour = Country)) + geom_line() +facet_wrap(~ Country)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*mzoDBQYYQTGfAFymQcCa6Q.png" /></figure><h4>Observations:</h4><ul><li>Except for Australia, the other countries have an increasing trend.</li><li>Australia has two distinct modes of normal mixtures because, over time, the PercentOfBaseline has a major shift.</li></ul><p><strong>Exercise</strong>: Plot and compare the time series of PercentofBaseline of Country, State, City, and AirportName.</p><h3>Time Series Forecast</h3><p>Time Series is a specific data structure in R.</p><p>We have to convert data into time series data structures to apply time series algorithms.</p><h3>Chile Data</h3><pre>library(dplyr)<br>data_chile = data %&gt;% filter(Country == &quot;Chile&quot;) <br>chile = data_chile[,c(2,5)]<br>head(chile)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/0*Cis1uzxMHpxgaz-R.png" /></figure><h3>Convert into time-series data structure*</h3><ul><li>1,2,3,4,… as time index with no NA values.</li></ul><pre>library(zoo)<br>z &lt;- read.zoo(chile, format = &quot;%Y-%m-%d&quot;)#zoo series for dates<br>time(z) &lt;- seq_along(time(z))#sequential data as 1,2,3,4,...<br>ts_chile = as.ts(z) #conversion into time series data structure<br>head(ts_chile)</pre><ul><li>Date as a time index with NA values to be removed.</li></ul><pre>library(zoo)<br>library(tseries)#for removing na from ts()<br>z &lt;- read.zoo(chile, format = &quot;%Y-%m-%d&quot;)#zoo series for dates<br>ts_chile = as.ts(z) <br>ts_chile = na.remove(ts_chile)#remove na from time series<br>head(ts_chile)</pre><p>If you convert here, like this it will create weird numbers. I prefer to do it as 1,2,3,4,… time index.</p><h3>Plot the time series</h3><pre>plot(ts_chile)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*NVG1XFcsPrgEClR8.png" /></figure><p>Two parts of the Time Series, we’ll discuss are</p><ul><li>Decomposition (Trend + Seasonal + Error)</li><li>Forecasting</li></ul><h3>Decomposition</h3><pre>decompose(ts_chile)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*dc_feuFE7va-FJdn" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*yahbYzY6IowRxOUT.png" /></figure><p>This is happening because the data has no seasonal trend</p><h3>Forecast</h3><h3>Model (ARIMA model)</h3><pre>library(forecast)<br>model = auto.arima(ts_chile)<br>model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/60/0*j3_6uac7y6d7Xgbe" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*x539M2xly9fXoDuW.png" /></figure><h3>ACF Plot</h3><pre>acf(model$residuals, main = &quot;Correlogram&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*9QoJmnI-Eqo31aeZ.png" /></figure><h3>PACF Plot</h3><pre>pacf(model$residuals, main = &quot;Partial Correlogram&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*KFAcD_aKxpEm5RJq.png" /></figure><h3>Ljung Box Test</h3><p>The test determines whether or not errors are iid (i.e. white noise) or whether there is something more behind them; whether or not the autocorrelations for the errors or residuals are non zero.</p><pre>Box.test(model$residuals, type = &quot;Ljung-Box&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/0*fF2p0r9KgNs2OVq2.png" /></figure><p>We do not reject the hypothesis, that the model shows a good fit.</p><h3>Normality of Residuals</h3><pre>hist(model$residuals, freq = FALSE)<br>lines(density(model$residuals))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*vndFK1-cfLHKB12W.png" /></figure><h3>Forecast</h3><pre>forecast = forecast(model,4)#4 = number of units you want to <br>library(ggplot2) <br>autoplot(forecast) #plot of the model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/0*G3TkElNvSX33gQ_b.png" /></figure><pre>accuracy(forecast) #performance of the model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/0*Yi8wEgjm_EXN3bV2.png" /></figure><pre>forecast #forecast values</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/0*P_pnKJXuuju60tkr.png" /></figure><p><strong>Exercise</strong>: Apply it to other countries as well.</p><p>All the best. :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6175c0f0901a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/srijit-mukherjee/covid19s-effect-on-airport-traffic-a-data-analysis-in-r-6175c0f0901a">COVID19’s effect on airport traffic: A Data Analysis in R</a> was originally published in <a href="https://medium.com/srijit-mukherjee">Srijit Mukherjee Content</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Analysis in R]]></title>
            <link>https://medium.com/srijit-mukherjee/time-series-analysis-in-r-355597bc7536?source=rss-a4bc34279e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/355597bc7536</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Srijit Mukherjee]]></dc:creator>
            <pubDate>Wed, 19 May 2021 19:40:51 GMT</pubDate>
            <atom:updated>2021-05-19T19:40:51.954Z</atom:updated>
            <content:encoded><![CDATA[<p>Download the data from <a href="https://drive.google.com/drive/folders/1LWFC344Ng-M8JborpJ2qoeQ8vIkesYi2?usp=sharing">here</a>.</p><h3>Load your data</h3><p>Time Series is a specific data structure in R.</p><p>We have to convert data into time series data structures to apply time series algorithms.</p><h3>Chile Data</h3><pre>library(dplyr)<br>data_chile = data %&gt;% filter(Country == &quot;Chile&quot;) <br>chile = data_chile[,c(2,5)]<br>head(chile)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/1*jivbUDXzYyGKRpPh-Y2_7Q.png" /></figure><h3>Convert into time-series data structure*</h3><ul><li>1,2,3,4,… as time index with no NA values.</li></ul><pre>library(zoo)<br>z &lt;- read.zoo(chile, format = &quot;%Y-%m-%d&quot;)#zoo series for dates<br>time(z) &lt;- seq_along(time(z))#sequential data as 1,2,3,4,...<br>ts_chile = as.ts(z) #conversion into time series data structure<br>head(ts_chile)</pre><ul><li>Date as a time index with NA values to be removed.</li></ul><pre>library(zoo)<br>library(tseries)#for removing na from ts()<br>z &lt;- read.zoo(chile, format = &quot;%Y-%m-%d&quot;)#zoo series for dates<br>ts_chile = as.ts(z) <br>ts_chile = na.remove(ts_chile)#remove na from time series<br>head(ts_chile)</pre><p>If you convert here, like this it will create weird numbers. I prefer to do it as 1,2,3,4,… time index.</p><h3>Plot the time series</h3><pre>plot(ts_chile)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*Dni9SpxGPraX9yZEnPtNiw.png" /></figure><p>Two parts of the Time Series, we&#39;ll discuss are</p><ul><li>Decomposition (Trend + Seasonal + Error)</li><li>Forecasting</li></ul><h3>Decomposition</h3><pre>decompose(ts_chile)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*Q7ikJel8nVqyzGNN3w3mTg.png" /><figcaption>This is happening because the data has no seasonal trend</figcaption></figure><h3>Forecast</h3><h4>Model (ARIMA model)</h4><pre>library(forecast)<br>model = auto.arima(ts_chile)<br>model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*dOhlP3zyYgvcEaOcrmP0pg.png" /></figure><h4>ACF Plot</h4><pre>acf(model$residuals, main = &quot;Correlogram&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*Z-xLa5DzNPSVVDXKjPTa_w.png" /></figure><h4>PACF Plot</h4><pre>pacf(model$residuals, main = &quot;Partial Correlogram&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*NOKUHxE4-80NdgqA2zasPw.png" /></figure><h4>Ljung Box Test</h4><p>The test determines whether or not errors are iid (i.e. white noise) or whether there is something more behind them; whether or not the autocorrelations for the errors or residuals are non zero.</p><pre>Box.test(model$residuals, type = &quot;Ljung-Box&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*-1gtBKdy0OHT2eWLnW_aEQ.png" /></figure><p>We do not reject the hypothesis, that the model shows a good fit.</p><h4>Normality of Residuals</h4><pre>hist(model$residuals, freq = FALSE)<br>lines(density(model$residuals))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*a15NjWZKZG0BJkdgfI0Zmw.png" /></figure><h4>Forecast</h4><pre>forecast = forecast(model,4)#4 = number of units you want to <br>library(ggplot2) <br>autoplot(forecast) #plot of the model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*qaeGAPJex6ZmsXU9frZKkA.png" /></figure><pre>accuracy(forecast) #performance of the model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*JA6T3j_mcpKtER8KO0O_xg.png" /></figure><pre>forecast #forecast values</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*0l3wzj3hMjiuJthVXo2YqQ.png" /></figure><p>Exercise: Apply it to other countries as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=355597bc7536" width="1" height="1" alt=""><hr><p><a href="https://medium.com/srijit-mukherjee/time-series-analysis-in-r-355597bc7536">Time Series Analysis in R</a> was originally published in <a href="https://medium.com/srijit-mukherjee">Srijit Mukherjee Content</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>