<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naveen Mysore on Medium]]></title>
        <description><![CDATA[Stories by Naveen Mysore on Medium]]></description>
        <link>https://medium.com/@nbitzz?source=rss-63f8d992e93a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jG6E-WHF0TeZ2ac1cQiZ9A.jpeg</url>
            <title>Stories by Naveen Mysore on Medium</title>
            <link>https://medium.com/@nbitzz?source=rss-63f8d992e93a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:36:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nbitzz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reinforcement learning .]]></title>
            <link>https://medium.com/data-science/reinforcement-learning-79ffd92886a7?source=rss-63f8d992e93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/79ffd92886a7</guid>
            <category><![CDATA[monte-carlo]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Naveen Mysore]]></dc:creator>
            <pubDate>Fri, 08 Sep 2017 02:35:44 GMT</pubDate>
            <atom:updated>2017-09-08T20:58:46.947Z</atom:updated>
            <content:encoded><![CDATA[<p>We humans learn many things from our day to day activities. We observe our environment, take some actions and see how our actions affected our environment and take our next actions. Reinforcement learning is similar way of modeling a systems or writing a computer program where a system sees or observe its environment and takes some actions to achieve some goal.</p><p>In this blog we will start from scratch, learn Monte Carlo estimation methods and train an agent to achieve a goal. Particularly, we will learn to train an agent to balance a cart pole!! We will also learn how to convert equations to code and see for ourself how it all works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*OkwYj4C70I2roKn-wsvW7g.gif" /><figcaption>Pushing cart towards left.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*4_6smfudN4hnYM2-T4K0ow.gif" /><figcaption>pushing cart towards right.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Ew41SO5DCQfrZQ5DCrozxg.gif" /><figcaption>Trained agent taking correct decions.</figcaption></figure><p>The classic text book by <a href="http://people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf">Sutton</a> and <a href="https://www.youtube.com/watch?v=2pWv7GOvuf0&amp;t=2s">lectures</a> by David silver are best way to get detailed understanding of the topics however one can learn better if he/she applies those concepts and build something and see it work for itself and that`s the primary motive of this blog. This blog will be super useful if you have watched his lectures but if not, don`t worry I have tried my best to explain those concepts in a simple way.</p><p>We will take the classic Cart pole game environment and learn Reinforcement Learning (RL) through it. In RL setup we call the thing which is trying to learn something an <em>agent</em>. In this blog I will be using agent/we interchangeably. For example in animals the brain is an agent. In computers your RL code is an agent. Humans have evolved so complex that even our top researchers are no where close to understand how it learns everything. When people say AI you need to realize that its not some super conscious thing that is able to learn everything and communicate and act like human!! its just a code running as expected. Neural networks are just tiny component of it and what matters is how your design system using these nets. Similarly our RL programs can only learn what we will program to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*7cuAqjQ97x1H_sBIeAVVZg.png" /><figcaption>RL Setup</figcaption></figure><p>In a typical RL setup our agent interacts with environment, by taking some actions (<em>At</em>) at time <em>t</em>. Also the actions the agent took has changed environment by modifying environment or if agent has moved then its camera/sensor perspective would have changed, so agent will see the new state(<em>St+1</em>). Just like us you see something, close your eyes and take a step forward and then you see what you see is bit different from what you saw earlier. If agent`s actions were good, environment will give a reward (<em>Rt+1</em>). Using those rewards agent will learn something and seeing this new state agent will take new action. And it goes on. Lets understand this with example, first lets setup our environment.</p><p><strong>Setup</strong></p><p>OpenAI provides RL environments thought their OpenAI gym. Take a <a href="https://gym.openai.com/">look</a>. We will be using Cartpole environment. Here is the <a href="https://gym.openai.com/docs">link</a> to setup your environment. Once you have setup try out this code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8baabd2fbf766fde66419270f45297d5/href">https://medium.com/media/8baabd2fbf766fde66419270f45297d5/href</a></iframe><p>You will see a cart pole game environment will be displayed and as per the code. We will randomly sample an action (<em>0 or 1</em>) 0 for left and 1 for right and we take that action by calling env.step. Then we get results for taking that action. results will be new state, reward, terminal, info. terminal denotes if game has ended or not (True if game ended). Now if you print the new_state values you will see something like [ 28.53394423 4.91374739 -43.39076723 -9.09823416]. Which is a 1 X 4 vector. Each value in this vector represents something like angle with base or distances moved from center. Each unique number in this vector represents unique states for that cart pole. Now our objective is to take these vectors, pick actions by decisions instead of random actions and see how our action make cart pole move. If one of these values goes extreme like taking only action left (pushing left) will result in new states moving towards left and the game will end if cart crosses some threshold. Similarly if pole falls or makes extreme angles game ends. At every time step we/agent have two possible actions to take (push left or push right). we need to observe how our cart pole is moving and adjust our actions so the pole stays in same place. Here`s how it will look if you push only towards left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*OkwYj4C70I2roKn-wsvW7g.gif" /><figcaption>Pushing cart to left.</figcaption></figure><p><strong>States</strong></p><p>A state can be anything which describes the current condition of an agent. For example let`s say you are playing a corn maze game. What parameters can uniquely describe your location if you consider locations as states? we can choose person`s latitude, longitude and orientations as states and we can represent them as an vector array [37.338, -121.88, 0.234] or lets say you are teaching a robot to move then we can consider like sensor readings, its camera feed, location etc as state representation. Obviously more sensors more uniquely we can identify its states.</p><p><strong>Goals</strong></p><p>If you consider any animal, everything thing it does, it does it to maximize something. For example, you study hard for exams to maximize your grades. In this case your school, class, home, exam are all part of environment. You take actions to study and environment gives you rewards inform of grades. If you get poor grades you will study hard so in then next exam you can improve your grades. Eventually you are trying to maximize your grades. Note that you wont get rewards immediately as soon as you study but the rewards are delayed. So when we say we got rewards we need to be consider everything from your study time to exam time to results day. Lets quantify this goals and rewards in next section.</p><p>In case of cart pole balancing the pole is our goal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Q1S-FZSkWYwynjxlECoofw.gif" /><figcaption>initial agen`t effort</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Ew41SO5DCQfrZQ5DCrozxg.gif" /><figcaption>Goal achieved</figcaption></figure><p><strong>Rewards</strong></p><p>In the code if you print values for reward you will see its 1. Actually you can define whatever rewards you want but lets understand the term called <em>discounted returns</em>.</p><p>Every time step you are in a state, you take an action, you will get some reward (lets say 1). In the game setup, if you keep pushing cart only to left you will see like something carting moving right and pole falling down (try it).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*4_6smfudN4hnYM2-T4K0ow.gif" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afff5d40937b6fc82f67b034d41e2477/href">https://medium.com/media/afff5d40937b6fc82f67b034d41e2477/href</a></iframe><p>As per the code you only pushed toward left/right and at each time setup you get reward of 1 and at some point pole falls above/below threshold and the game ends. So, what is the total rewards we got?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*rAzG2qReGTNblbdw5rW9qg.png" /></figure><p>We say the total return at timestep t=1 was 8. Similarly if you want to calculate return from time step t3. Just add all returns from time step t3 onwards. You can think of it like this. Return from game frame 1 is sum of all rewards until end of the game. Return for game frame 4(time_step = 4) is sum of all rewards from game frame 4 on wards. Or in general if a game ends at time step tn and you want to calculate return form time step t onwards. we can represent it like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/1*U9f6R1QhdjMKrXZyCrcYAQ.png" /></figure><p>Now here is a problem. Some environments like this cartpole is terminal i.e at some point game ends (if you take wrong actions). However some environments might not be terminal, lets say you are teaching an agent to read english, here there is no finite terminal point. we need our returns to generalize to all agents. To generalize we add extra <em>terminal state</em> called <em>ST</em> . The special property of terminal state is its never ending ie. if you enter terminal state we will come back to same terminal state. Also entering terminal state will give you 0 reward. Don`t worry about terminal states now you will understand them eventually. But, now with this slight modification we can write equation as.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/95/1*hT4h5lHFmYHV2bEwsjAzHg.png" /></figure><p>where t runs from 0 to inifinity. If you notice the value of Gt has not changed at all since at terminal state we get 0 rewards. even if we sum toward infinity.</p><p>Now there is one more problem for environment with non terminal states. like that english reading problem as we can see GT values will blow up if we keep adding rewards. Besides for actions taken at time t the rewards we get at next time step (immediate reward) matters much, and rewards from next next time step(delayed rewards) matter much less and so on. To generalize this we introduce <em>discounted rewards</em>. In some cases rewards only come after you have achieved some goal. So we need to design our system keeping these things in mind. So, putting weights on those rewards is a start. ie rewards would be discounted by some factor gamma at every time step. Now we can visualize all these .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6RI3vR0NzcQDlbBsfG70bQ.png" /><figcaption>state to state transitions with rewards and terminal state.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/144/1*xdajS9OzqaZNx7V3xQy9Ww.png" /><figcaption>Discounted return</figcaption></figure><p>S0 in some initial state we take some action, we get reward R1 then our new state is S1 from which we take some action and we get reward R2 and so on. At S2 we/agent did some bad action which pushed us/agent to some terminal state and hence from then onwards we get rewards as 0.</p><p>Now to calculate returns Gt we take next reward(immediate reward) and add all other rewards by discounting them by gamma. Typically we keep 1 as positive reward and for last state before terminal we give negative reward because in that state our agent took bad action and ended up in terminal state.</p><p>Lets run a simple code to see it in action.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2977d9a58b3fb5943d8af292bcfa21d0/href">https://medium.com/media/2977d9a58b3fb5943d8af292bcfa21d0/href</a></iframe><p>Go through the code and see how states are transitioning. At time step 1 we were in initial state. We deterministically took action 1 and we ended up in new state. From that state we again deterministically took action 1 and so on. at time step t=11 we were in state [ 0.15274551, 1.70880722, -0.17696351, -2.69724965] and took action 1 and we reached last state [ 0.18692165, 1.90472068, -0.2309085 , -3.03828657]. Even though reward says 1 we will set it to -10 when counting returns. And after that we can calculate returns using discounting 0.9</p><p><em>Notice something important here, in order to calculate returns for a state we had to wait till the game ends.</em> we can run one game as an episode. If we run two game we can say we ran two episodes. In literature its usually referred as rollouts.</p><p>Lets modify that code a little bit to see returns for first state and last state for 10 different episodes deterministically taking one action. Notice how each episode length varies!! and that`s the reason we derived generic return function Gt using terminal states. Irrespective of episodes length we can sigma from time t to infinity and since terminal states will have zero rewards our returns will be uniform across episodes. here a code for total returns calculation for state s1 and st(last state) for 10 different episodes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fb23e41e7a0adda43c366fff709db32/href">https://medium.com/media/0fb23e41e7a0adda43c366fff709db32/href</a></iframe><p><strong>Markov property and Markov Decision process.</strong></p><p>So far we have deterministically chosen a single action (either left/right). We didn`t consider any intuitive way of deciding which actions to take. Before we can jump to decisions we need to know how the whole state actions transitions are happening. In a real world its the universe decides how transitions should happen. For example if you make a paper plane and you shoot it to a target, even if your target was perfect its the universe`s factor like wind, humidity, friction etc will decide if your plane should hit the target or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*LbyplQbNbDzPHYf7yWkeWQ.png" /><figcaption>Finite MDP</figcaption></figure><p>The cartpole is a simple environment. let us consider a generic environment. Let this environment have only 3 states and at each state you can take either action a0 or a1. In a real world everything happens probabilistically i.e say you are in state S0 then may be there is a 30% chance you can take action a0 and 70% chance you can take action a1. lets say you chose action ao then may be there is 50% chance you can transit to state S2 or may be you might come back to same state, any thing can happen!!. The example provided above is a finite MDP because there are finite number of states ie. 3. In case of cartpole there can be any number of states, so we can use something called stick diagrams and Bellman`s equations to capture them later. In each state, its the environment which controls how you transition to next state. Given a state, there will be some probabilities associated with each action and as we know only after we take an action we will get the rewards.</p><p>We should some how encode information to each states so when we are at a particular state we can see the probabilities and values of taking those actions. If states can retain such values they are termed to have Markov property. Once we reach sections in this blog for value of state V(s) and q values Q(s,a) you will understand better on how each states retains values and we can use Markov Decision process to take right actions. Before we jump to finding value of these states lets give these probability terms some names.</p><p><strong>Policy</strong></p><p>In a given state there will be probabilities associated with each actions which governs our agent`s behavior. For, example if robot is an agent and lets say in a particular state(top of the mountain) there are high winds blowing, sometime pushing the robot of the cliff then robot should avoid going to that state. The robot can avoid that state by decreasing probabilities associated with those actions which led to that state.</p><p>Policy is nothing but probability of taking an action in a given state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/47/1*3_6c1KiAnh33C48A0ziG0w.png" /></figure><p>pi is probability, a is action, | is given that, s is state.</p><p><strong>Transition probability</strong></p><p>If you are in a state, you take a particular action. Then you could end up in any state. If your environment has n number of states then each state will have some probability associated with it given that you were in state and took some action.</p><p>Transition probability is nothing by transiting to state from a given state taking some action a and we discussed above its the universe which has encoded these probabilities and we need to recover it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/101/1*xO0d5hxX9bDnsgpxamqzLA.png" /></figure><p>P is probability, St+1 is next state, St is your current state, At is the action you took. Probability of going to state St+1 from state St taking action At.</p><p><strong>Math ahead</strong>!! :)</p><p><strong>Transition stick diagram.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*RgYJybo8LRQ73oGukkXROg.png" /><figcaption>State Transitions.</figcaption></figure><p>From the stick diagram its now easy to infer that at the beginning we in state So. In that state we had options to take <em>n+1</em> number of actions. We randomly decided to take action <em>An</em>. After taking action <em>An</em> the environment now may push us back to state So or S1 and any other states. Since we have so many actions to take and we might end up in any states lets associate probabilities to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2SkDuYjNbKEGJsNUQ4Byag.png" /><figcaption>State Transitions with probabilities over state and action space.</figcaption></figure><p>Lets walk through this diagram. Initially agent/we were in state <em>So</em>, in that state we could pick <em>n+1</em> number of actions from Action Space <em>A:{A0, A1, A2….An, An+1}</em> and for each of those action there is some probability of picking it and that defines agents behavior. A good agent will have high probability for picking right action. Medium does&#39;t let me use math notation for <em>PI</em> so I will be using <em>PI</em>. <em>PI(At|St)</em> represents probability of picking action At given that we were in state St and that`s our policy.</p><p><em>Note that the actions agent chooses is conditioned on that state its in. Just like you!! The speed at which you drive your car(action=press gas?) is conditioned on if your mom/dad is sitting next to you in the car(St=mom,dad)!!</em></p><p>Guess.. PI(action=press gas?| St=best buddies)? :)</p><p>Ok, Some how the agent chose to pick action <em>An</em> now our environment is stochastic (“wind might blow us anywhere analogy”) so there is probability our agent could end up in state <em>S0, S1 ..Sn</em>. These states come from state space <em>S:{S0, S1, S2…Sn}</em>. We can write it as <em>Pr(St+1| St, At)</em>, which means probability agent might end up in state St+1 given that it was in state St and took action <em>At</em>. If agent ends up in right state it will get good reward (positive reward), if it ends up in bad state it will get bad reward (negative reward). So agent`s reward is conditioned on that state it ends up in so we can put probability on that Pr(Rt+1| St, At, St+1) which means probability that agent will get reward Rt+1 given that we was in state St, took action At and landed in state St+1.</p><p><strong>Value of a state V(s)</strong></p><p>Lets say we have a robot agent and its in state <em>S0</em>. If it takes action <em>A0</em> it will end up in fire pit which is bad, if it takes action A1 it will send up on smooth road which is good. Now, we need to mathematically quantify what good/bad means. In other world we need to estimate the values for these states. We can estimate values using <em>Bellman`s equation.</em></p><p>Bellman`s equation is a dynamic programming equation. To understand it let say we have estimated some value for say state <em>S0</em>, next time we visit same state <em>So</em> we can use the old estimate value, compute new estimate and update value for this same state. Next time you visit same state <em>So</em> you would have a better estimate.</p><p>One thing we can do is we can simply setup a dictionary with states as key and state-values as its values. We can use total returns we get as a value for that state. But, if you look at the code captioned “Total returns for state s1 and st(last state) for 10 different episodes.” posted above we can see the the returns for state S1 keeps changing episode to episode and for each episode we can have any number of states. We can take a simple average of total returns for a particular state and that would be a good way to start estimating.</p><p>So, we will take that code above run few episodes say 10 and we will average it. Now, lets think how can we do that? we could keep a counter. We will run 10 episodes every time we visit a particular state we can just update the counter for that state. After 10 episodes for all the states we can take sum and update them.</p><p><em>Note when we say same state keep in mind that all values must be same. For instance if in episode the first game frame is state S0:[1.1, 1.2, 1.3, 1.4] and in second episode the first game frame is sate So:[0.1, 0.1, 1.1, 0.4] then they both don’t represent same state. If in second episode 5th game frame is state S5:[1.1, 1.2, 1.3, 1.4] then they both represent same state.</em></p><p>here is a simple <strong>pseudo</strong> code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3976add0d0d367c1ce91e42b1dd8598c/href">https://medium.com/media/3976add0d0d367c1ce91e42b1dd8598c/href</a></iframe><p><strong>Monte Carlo Learning</strong></p><p>This way of estimating values for states where we run entire episodes end to end. Collect returns for all states and run many such episodes. Finally stop running game and update state values by taking mean of collected states value is Monte Carlo approach. This could be game episodes.</p><p>episode 0: S00 -&gt; S01 -&gt; S02 -&gt; …… -&gt; Sot</p><p>episode 1: S10 -&gt; S11 -&gt; S12-&gt;S1t</p><p>episode 2: S20 -&gt;S2t</p><p>…</p><p>episode N: Sn0 -&gt; Sn1 -&gt; Sn2 -&gt; Sn3 -&gt; Sn4 -&gt;……… -&gt; Snt</p><p>Few things to note here. S10 represents state 0 of episode 1, similarly S01 represents state 1 of episode 0.</p><p>S00 S10 might not be same!! May be S01 and S12 same state!!</p><p>In a stochastic environment transition to state governed by Pr(St+1|St, At) as we saw before.</p><p>lets say S10 and S12 are both same states and lets call is Sb. lets initially estimate value of sb ti be 0.1 ie. V(sb) = 0.1. Over two episodes 0,1 we know the true returns Gt(S10) and Gt(S12) lets says its 0.2 and 0.3 then we can keeps a counter and update it N(sb) += 1 so it will be 2. To estimate new value add all returns for that state Gt(S10)+G(S12) and compute mean Gt(S10)+G(S12)/N(sb) thats the new estimate.</p><p>So we can update V(sb) ← Gt(S10)+G(S12)/N(sb) where S10 and S12 represents same state across different episodes. If you don`t increment counter for within a state ie. if you visit the same state within an episode and you don`t update the counter it called <em>first visit Monte Carlo</em>. If you increment counter every time you visit a state then it is called <em>every visit monte Carlo</em>.</p><p>Now here we see a small problem. In order to update V(s) we had to run say n number of full episodes then stop running, compute mean and then update V(s) and then we proceed to do the same thing. Can be find some solution? instead of stopping and doing batch update can we do it on the go? Lets do a simple derivation. We know V(s) is our estimated state value when running a episode, After we run the episode we know the true return for that state ie. Gt(s). So, Gt(s)-V(s) is the error we made in estimating the value of the state.</p><p>Let our initial state estimate be V(s). After we run k number if episodes of N(s)k represents number of times we visited state s. We know the value of state V(s)k is mean of total returns until episode k over N(s)k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/1*EOJfHEA19VT5uuUbyMgquA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*MHuhm_9q67g76w6TnbOnsg.png" /></figure><p>We can split Sigma to current true return + sum of all previous return. We know at k-1 episode V(S)k-1 must have been mean of returns. So</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*nogNI5S1isaF5R-owmV2qQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*V4ughlOLMAiGJhHesdgEcw.png" /></figure><p>We can replace sigma term in V(S)k with N(S)k-1 in V(S)k-1 and N(S)k-1 is (N(S)k)-1. So</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*b8tqHpv79gyu_JIMlg9o1A.png" /></figure><p>Rearranging</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*bJ_vz3tTsH-n8Lp6ATG9LA.png" /></figure><p>We can count all occurrences of states with N(s) or we can relax by considering only a small fraction of episodes so we can get rid of counter. lets fix it with alpha.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*iP9CcGSWy5FCsSUMdNzRgA.png" /></figure><p>Now we have a Monte Carlo update. what this is saying is, every time you visit a state you can get the estimate we have done so far for that state and we can get the true return for that state after episode ends and we find the error we made and update our old estimate in the direction of error. So next time we visit the same state we would have better estimate.</p><p>Note to update we still have to run full episodes, but to update value we don`t need to stop and do batch update but we can update after every episode. But still we have to run one full episode to compute total return and this problem can be solved by Temporal Difference Learning which we will visit later.</p><p><strong>Enough with math lets code!!</strong></p><p>But wait, how do we store state-values ? previously I had said we could use some dictionary to put state as key and its value as state value but there can be so many states!! lets use a function approximat0r.</p><p>A function approximator takes array vector as input and multiplies we weights and result will be approximate value. Of course initially value will be wrong but since we know the true value its supposed to be we can find error and change the weights. This function approximator can be a neural networks or a simple numpy multiplication.</p><p>From one of our previous code lets pick one of the state s =[-0.02553977, -0.5716606 , 0.03765532, 0.90520343], lets pick random weights w = [0.1, 0.21, 0.14, 0.18] then V(s) = -0.02553977 * 0.1 + -0.5716606 * 0.21 + 0.03765532 * 0.14 + 0.90520343 * 0.18 + 0.3. where 0.3 is a bias term.</p><p>So V(s) is 0.3456056592. But if True return Gt(S) is 0.234 then true value of this state should be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*iP9CcGSWy5FCsSUMdNzRgA.png" /></figure><p>0.345605659 + 0.9 * [0.234–0.3456056592] i.e 0.24516056572. But our estimator predicted it as 0.345605659 so the error it made was -0.10044. Now we just need to change weights accordingly to make right prediction. I will not go in detail about how to make changes to weights as it a separate topic but what we will do is use python packages for that (we will be using Tensorflow)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d601ca3c3e295d7e7b3415f8c195b850/href">https://medium.com/media/d601ca3c3e295d7e7b3415f8c195b850/href</a></iframe><p>The first array represents state and second array represents function approximator`s value. Now we know the true returns so lets computer true value estimates for those states.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a329cf297eadc52411de87d76477fb6/href">https://medium.com/media/1a329cf297eadc52411de87d76477fb6/href</a></iframe><p><strong>Expected Values</strong></p><p>Now earlier we visited this diagram, now lets look at it from value of a state perspective to see what exactly value is state we are taking into consider. As we know our environment is stochastic. Our policy decides what actions we choose our environment decides what state we should land in and we have probabilities to capture them. Lets write equations to capture them. What I am trying to say here is the value we are estimating are not true estimates. Ideally if you consider universe as a real deal then every state has some true universe encoded state value!! our model makes prediction what that state value might be by exploring and updating its view of states values. When our states values becomes equal to universe encoded state values then our model would converge to true ideal state value. We don`t know what true ideal state value is but using this diagram we can come up with a math equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*XJh2eJGzMfaG3OtklYz7YA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*pvd7eBrr49OtnFo7M-SUTg.png" /></figure><p>Under a policy pi, lets say your policy is to always take left action(push left). Lets say you are in state <em>S0</em>, then for all actions you can take from that state <em>S0</em>. we consider the probability of that action <em>pi(a|s)</em>, for all those actions you can land in any states controlled by universe probability <em>p(s’|s,a)</em> we add immediate reward and discounted values of next state. Which means our actions influences our value estimates.</p><p><strong>Decisions for actions with Q values Q(S,A)</strong></p><p>So far we have estimated Values of states V(S). In the previous code you might have noticed that we saw what our function approximator approximated value of state and we also computed what true value of state should be. We could have computed error between them and updated our weights to to predict good estimates but if you think about it even if you accurately estimate state values, if you are in a state you cannot know what the value of next state would be. If you are in a state S0 and if some oracle told you hey your next state will be Sa or Sc and definitely not Sb then we could just compute V(Sa), V(Sc) and take action towards maximum value but we don`t have such magical oracle. So we need to compute some linkage. If you can compute value for taking an action then we can just predict good actions to take. ie hey I am in this state last time I was in this state I took this action and something bad happened so I will avoid it. If I take this other action I will get this predicted value. Just like how we estimated values for states we can estimates values for taking actions in a given state and those values are Q values Q(S,A). Now Q value is very similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*tm3GCx6jLsexDRK1UeocDQ.png" /></figure><p>Here if we were in state S and took an action a, then from that state we could have landed in any of those m states governed by P(s’|s,a) then we can just take the value of those states under policy pi.</p><p>Now in our code just like how we had estimate V(s)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*iP9CcGSWy5FCsSUMdNzRgA.png" /></figure><p>We can estimate Q(S,A)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*K1VwhiS5zZN0Yg63ijHDsw.png" /></figure><p>And as we explore all states over time our estimate value for taking state- action will converge to universe encoded state-action values.</p><p>Now I want to make some changes to policy so we make sure we explore all possible states and also we need to make sure we reach out goal (maximize total returns) we can do these things.</p><ol><li>We will make estimates for all actions in a state and then pick that action which gives maximum values. This is a greedy policy since if we find this action works good we will always stick with it and might not explore different states.</li><li>Initially we take random actions and once in a while we will take action corresponding to maximum q value. as time goes on we will reduce random actions and take more greedy action.</li><li>For predicting q values we can use same function approximator but this time we will output two value since we have two actions, push left/ push right.</li><li>We will keep track of total returns, actions took and current estimates for qvalues and then we can compute errors and update our weights.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*j2_4TzJtEXOapPUXFiwY_w.png" /></figure><p>Here is the code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/452dfbdb9dde7951276f36e84fdf3c72/href">https://medium.com/media/452dfbdb9dde7951276f36e84fdf3c72/href</a></iframe><p>Run this code and you can see around 5000 episodes you will see our policy starts improving. I didn`t experiment with learning parameters but we can see its improving and finally around 7000 episode goal is achieved. But can we make it better? Can we learn with small number of episodes Lets explore that in next blog. The first for loop collects experiences and the second starts learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*UArovPR0tCW3fOBqiFQWKQ.gif" /><figcaption>Around episode 2000</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Ew41SO5DCQfrZQ5DCrozxg.gif" /><figcaption>around 7000 episode</figcaption></figure><p><strong>Disadvantages</strong></p><p>This Monte Carlo method have few disadvantages.</p><ol><li>We need to wait until we run one full episode to compute returns.</li><li>This wont work for non terminal episodes.</li><li>MC method has high variance and low bias.</li><li>low bias because the returns we consider are true values for that state in a episode.</li><li>high variance because this approach doen`t consider environment dynamics much. The values we update are returns computed for entire episodes. Sometime it would be good to consider only immediate reward and may be some portion of future estimate as this would capture environment dynamics better.</li></ol><p>We will explore Temporal difference learning in next blog which addresses these issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79ffd92886a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/reinforcement-learning-79ffd92886a7">Reinforcement learning .</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability Theory for Machine Learning.]]></title>
            <link>https://medium.com/series/probability-theory-for-machine-learning-d5ecb0c0974f?source=rss-63f8d992e93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5ecb0c0974f</guid>
            <dc:creator><![CDATA[Naveen Mysore]]></dc:creator>
            <pubDate>Thu, 22 Jun 2017 03:19:44 GMT</pubDate>
            <atom:updated>2017-06-22T03:19:44.359Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@_naveenmysore/probability-theory-8ae260219253">Introduction</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5ecb0c0974f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probability theory. (Part 1/3)]]></title>
            <link>https://nbitzz.medium.com/probability-theory-8ae260219253?source=rss-63f8d992e93a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ae260219253</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Naveen Mysore]]></dc:creator>
            <pubDate>Thu, 22 Jun 2017 00:50:46 GMT</pubDate>
            <atom:updated>2017-06-22T03:14:23.141Z</atom:updated>
            <content:encoded><![CDATA[<p>I had a very hard time understanding different aspects of probability theories and hence I dig various blogs, articles and complied it into a single blog. Hope it helps you to get a kickstart.</p><h4>What is Probability?</h4><p>Probability is mathematical way of computing uncertainties. In computer science we are used to discrete computations but the real word is very uncertain. For example if you shoot a ball to basket it might or not make it as various environmental factors defines the outcome. Study of probability is very helpful when we need to model complex systems. For example lets consider two statements, “Most birds fly” &amp; “All birds fly, except ostrich, kiwi, baby birds, penguins… ”. The two statements are true but seconds is more complex as it needs to consider every possible non flying birds!!. First statement helps to generalize model into a simple manageable statement. Similarly probability theory helps to generalize complex systems with a simple model.</p><p>Here are the topics I am planning to cover in this blog.</p><ul><li>Random Variable</li><li>How to compute probability ( Box &amp; Tree method )- General/Conditional Probability</li><li>Bayes Theorem</li><li>Probability distribution functions.</li></ul><h4>Random Variable</h4><p>let x be a variable which can take any value. We can call ‘x’ to be a random variable since it can take any random value from a set or a distribution.</p><p>Example: If I have red ball, green ball &amp; blue ball. Then x can take red ball, green ball or blue ball and it cannot take any other color ball since in our set we only have red, green or blue. Similar lets say in an integer range from 0 to 10. x can be 2, 3 or 4 etc. We can represent it like this x=2 when x is 2, or x=red ball when our random variable takes red ball etc.</p><p>As I said probability is chance. Lets say if I have</p><p>5 red balls, 3 green balls, 2 blue balls and I put them all into a box. I shake the box for few seconds and I will ask you to close your eyes and pick a ball. Now what are the chances that you picked a red ball? We know we have a total of 10 balls and out of that 5 balls are red. There are more red balls than green balls &amp; There are more red balls than blue balls. number of(#) red balls is equal to sum of green &amp; blue balls. #Rb = #Gb+#Bb. The proportion of red balls are half the total number of balls so the chances of picking red ball is 5/(3+2) which is 0.5 or we can say 50%.</p><p>In terms of probability we can say P(x=red ball) is 0.5. i.e probability that my random variable x can be red ball is 0.5. It is same as saying probability of picking a red ball is 0.5.</p><h4>How to compute probability?</h4><p>Probability in terms of equation can be represented as</p><p>Total number of favorable outcomes / Total number of outcomes.</p><p>Ie. in case of 5 red balls and 3 green balls, if we were to pick a ball randomly the chances of picking a red ball is 5 / 8 since there is 8 ways of picking a ball randomly and 5 favorable (red)ways of picking red ball.</p><p>Computing probability can be very easy if you can represent your data in structured way. Two methods you we can use is Box method and Tree Method.</p><p>Lets take an example, There are 3 bowls (Bowl A, Bowl B, Bowl C). In Bowl A (Ba) we have 4 Blue marbles (Bm), 2 Red marbles (Rm), In Bowl B (Bb) we have 2 Blue marbles and 1 Red marble and finally in the last Bowl Bowl C (Bc) we have 4 Blue marble &amp; 5 Red marble.</p><p>This might look like lot of data so first lets represent it in a structured way.</p><p>— — — — — — Blue marbles(Bm)— Red marbles(Rm) — — Total marbles</p><p>|Bowl A (Ba)|______4________________2___________________6_______|</p><p>|Bowl B (Bb)|______2________________1___________________3_______|</p><p>|Bowl C (Bc)|______4________________5___________________9_______|</p><p>|Total marbles|____10________________8__________________18_______|</p><p>Now we can see that our data is represented. If I ask how many marbles are there in Bowl A we can quickly see and tell 6. If I ask how many marbles in Bowl C are red we can see and tell 5.</p><p>Now if I ask what is the probability of picking a red marble P(X=Rm) from any bowl we can see total out comes of picking marbles are 10+8 = 18 or 6+3+9 =18. and out of them 8 are favorable (8) so answer will be 8/18.</p><h4>Marginal probability</h4><p>In marginal probability there is no dependence on any other variable. for example I can ask you question what is the probability of picking a red marble. Here we don`t have specific bowl requirement we are free to pick from all bowls. so its 2/6 from bowl A+ 1/ 3 from Bowl b+ 5/9 from bowl C. Marginal probability captures probability for picking a red marble across all possible scenario we can represent it in equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/135/1*3S8PmGa-IoVc6R32boAucA.png" /></figure><p>i indexes from 0 to 3 for three different bowls. This example was for discrete case and for continous case it will be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/78/1*oxbJvL6gfsXx-gY8BvrvFg.png" /></figure><p>We will study continous cases later when we visit probability distributions section.</p><h4>Conditional probability</h4><p>Now lets jump to next level. What is the probability of picking red marble from Bowl A?. It should be straight we have 6 marbles in Ba and 2 are red so answer is 2/6. Lets say our Bowl A , Bowl B and C have different weights. If I put these Bowls on a turn table and rotate them then probabilities of Bowl A, B and C are not say. In an ideal world if every atom of Bowl A, B, C were same then we could say that probabilities are same i.e 1/3 for selecting a bowl. For this example lets say probabilities of picking Ba, Bb &amp; Bc are 0.4, 0.3, 0.3. Now I close your eyes and I pick a red marble randomly from a bowl. I ask you to open your eyes and ask a question what is the probability that this red marble came from bowl A?. If you pay attention to this question if you notice that answer to this question not only depends on picking a marble but also picking a bowl. We have three bowls out of three what are the chances of picking bowl A? and if we “do pick” Bowl A what are the chances of picking red marble from it? Probabilities which depend on some other probabilities are called conditional probabilities. In terms of equation we can represent it as</p><p>P(X=Rm|Ba) ie. Probability that my random variable is red marble given that I pick from Bowl A. That symbol “|” is read as given that. Question like this can be easily solved with drawing a tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*tA4tcQbhkK3IhVFbd9LQ8g.png" /><figcaption>Image1: Probability distribution tree for red, blue marbles and A,B,C bowls</figcaption></figure><p>Each branch of this tree represents a probability. 0.4 represents probability of picking Bowl A. 0.3 represents P(Bb) and P(Bc). P(Rm|Ba) represents picking a red marble from bowl A and it is 0.33 or 2/6 and we solved it earlier. Now for the question asked given that we picked a red marble what is the probability it came from bowl A ?</p><p>(P(Rm|Ba)) is 0.4 * 0.33: i.e upon selecting bowl A with probability 0.4 then only we can pick either red or blue marble and we know probability of picking red from that bowl is 0.4*0.33.</p><p>Note P(Rm|Ba) is not same as P(Ba|Rm)!!. The first one P(Rm|Ba) means probability of picking a red marble given that its Bowl A. Second one means If we have picked a red marble what is the probability that it came from Bowl A?</p><p>We know probability is the ratio of number of favorable outcomes to total number of outcomes. Lets first find the total number of outcomes. We can pick a red marble either form bowl A, B, or C and each of those bowls have there own probability.</p><p>So our answer should be ratio of probability of picking red marble form bowl A (Favorable) to probability of picking a red marble from any bowl P(Rm). Probability of picking a red marble should be probability of picking a red marble from bowl a + probability of picking red marble form bowl b + Probability of picking red marble form bowl C. We have already solved P(Rm|Ba) lets solve other using same logic and add them up.</p><p>P(Rm) = P(Rm|Ba)+P(Rm|Bb)+P(Rm|Bc)</p><p>P(Rm) = 0.4*0.33 + 0.3 * 0.33 + 0.3 * 0.55</p><p>= 0.396</p><p>Now Favorable event P(Rm|Ba) is 0.4*0.33 i.e 0.132</p><p>So P(Ba|Rm) = 0.132/0.396 i.e 0.33</p><p>Similarly try to solve this question. Given that I had picked a Blue marble what is the probability that it came from Bowl c? i.e P(Bm|Bc).</p><p>Now lets represent it in form of neat equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*M6Qi8i3sH0O3CXHUE5j-zw.png" /></figure><p>If we call them as events to generalize P(Ba) is event A and P(Rm) as event B then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*mtuMagTprnltO3B81OlGSg.png" /></figure><p>This is nothing but Bayes rule.</p><h4>Bayes Theorem.</h4><p>Each term in that equation captures a special information about our events. When I asked you that question. Given that I had picked a red marble what is the probability it came from bowl A, I actually gave you some evidence and your task was to hypothesize which bowl it came from. Now the evidence that marble is red means it could have come from bowl a, bowl b, bowl c. So you have hypothesized three different theory. Theory one would say marble could have come from bowl a, theory two would say marble could have come from bowl b and last theory would say marble could have come from bowl c. Among these 3 theories which hypothesis do you believe is true?. We can never be 100% sure that this red marble came from a particular bowl but what we can do is we can be certain to some degree that it came from a particular bowl.</p><p>I.e as we solved in the previous section you can be 0.33 or 33% sure that it might have come from bowl A.</p><p>So, I gave you an evidence that marble is red you proposed a theory with three hypothesis ie P(Ba|Rm), P(Bb|Rm), P(Bc|Rm). So each term in bayes rule captures these interesting</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*NayvGYDQ8MGqHPcx6an1DQ.png" /></figure><p>In P(Ba|Rm) I gave you an evidence its a red marble and you hypothesized that is 33% chance it might have come from Bowl A and this is called <strong>Posterior probability</strong>. If you refer the tree which we draw you can see where all these terms lie on it. P(H) is called <strong>Prior probability</strong>. Lets see what it is called so. If I had not given you any evidence, ie. If I had just said I have picked a marble (I did not say color) and I had asked you to hypothesize what is the probability it came from bowl A then you had three bowls and it could have come from any bowl and each have their own probability so you would have hypothesized that there is 0.4 or 40% chance it came from bowl A (refer tree). So this term P(H) captures what is the probability of your hypothesis if I had not given you any evidence. P(E|H) captures information about <strong>likelihood</strong>. ie, if I had told you that bowl was bowl A what is the probability to pick a red marble. From out tree diagram it is 0.33 P(Rm|Ba).The denominator P(E) ie. probability of picking red marble P(R) and that came from solving this P(Rm|Ba)+P(Rm|Bb)+P(Rm|Bc) and this is called <strong>Marginal probability</strong> ie, under all probable cases what is the probability of picking red marble. Lets summarize this equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*GLkM45IOCRMkDAtbWE5NGQ.png" /></figure><p>From now onwards lets follow the convention of H for hypothesis and e for evidence.</p><h4>Joint probability</h4><p>Lets add two green marbles to our above example and build a new tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Vdzlp69qX7Oy4Hd__X53Kg.png" /><figcaption>Image2: Probability distribution tree for red, blue, green marbles and A,B,C bowls</figcaption></figure><p>Now if I ask you what is the probability that you pick a red and green marble from bowl A all we have to do is P(Rm|Ba)*P(Gm|Ba) and this is how we calculate joint probability.</p><p>Lets solve a problem to concretize our understanding.</p><p>From our bowl and marble setup I have picked one green marble and one red marble. what is the probability that it came from bowl A ?</p><p>As we can see we have two random variables for every possible bowls and its a joint probability problem. Our Evidence is red and green marble unlike the previous case which was only red marble. so P(e) = P(e=Rm,Gm) and my hypothesis P(H) is P(H=Ba). Here P(H) prior doesn&#39;t change as we haven`t changed our bowl arrangement, but our evidence for a given hypothesis changes as we have added two extra marbles(green) to all bowls. Here both out marginal and likelihood needs to consider both red and green marble.</p><p>so P(e|H) = P(X=Rm,Gm|Y=Ba) = P(Ba) * P(Rm|Ba)*P(Gm|Ba)</p><p>our marginal will be sum of all probabilities of having red and green marbles from all three bowls.</p><p>P(e)=P(X=Rm,Gm) = (P(Ba) * P(Rm|Ba)*P(Gm|Ba)) + (P(Bb)*P(Rm|Bb)*P(Gm|Bb)) + (P(Bc)*P(Rm|Bc)*P(Gm|Bc))</p><p>we can write this in a short equation. Before we write lets make sure we point out the difference between these two symbols.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*srxM2MtnE3WLC1iTCyol2w.png" /><figcaption>sigma and integration is for summation in discrete and continous time. Pi symbol here represents product of n terms.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*HTQm7iVaH4wGxLAtXIztqw.png" /></figure><p>Where i indexes from i to n=3 representing bowls and j indexes from 0 to 1 representing marbles red and green. Now that we have our P(H) ie.0.4 since bowls configuration didn&#39;t change. We can answer to the question P(e=Rm,Gm|H=Ba)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*f9H6IIYFIzLgO5mQGTfgXg.png" /></figure><p>In fact you can solve it just by looking at the tree and you don`t need to write in equation format. But writing in equations condenses written material and also it feels fancy to write equations!! :)</p><h4>Expectation.</h4><p>Lets continue with the same example and play a game this time lets take the original tree red &amp; blue case. If you look at the tree each line segment has some probability value. In other words probabilities are distributed across the tree covering various scenario. When we say expectation we say what is the exception for a given distribution.</p><p>In this game if you randomly pick a red marble you will get 2 points and if you pick blue marble you will get 3 points. This action of wining a particular marble to getting points can be mapped with a function f(x). ie. f(red) is 2 and f(blue) is 3. exception is for all possible scenarios of winning (whether its red or blue) what is the total points we can accumulate.</p><p>We know how red &amp; blue marbles probabilities are distributed based on the tree in image 1. and we know f(red) and f(blue)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*KWnp4AxCXygaOg404d93rg.png" /><figcaption>expectation of function value for a random variable distributed over probability P in discrete &amp; continous.</figcaption></figure><p>x~p represents that our expectation values depends on distribution. ie say if mess up with bowls and change bowl probabilities expectation value of red marble changes.</p><h4>Probability distribution</h4><h4>Probability distribution</h4><p>The example which we took was a discrete case ie, the ball can be red, green or blue. Our random variable can take 3 discrete values. We can use probability on continuous cases too. To give you an intuition on continous case where our random various can take any value in a range. Let us consider a dart board</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_ifTjCK3CWCYMlrbbi-oQ.jpeg" /></figure><p>Now imagine this dart board is placed on a <strong>flat ground</strong> and your task is to drop a ball on exact center (bulls eye). Lets say the environment you are doing is task is in vacuum then there is a very high chance your ball might land on center. But lets say you do this task in a windy environment then the chance of your ball landing in bulls eyes is very low. Lets quantize the results of your task by measuring the distance from center to where your ball lands.</p><p>Let the radius of the board be 10cms &amp; let us score your performance by this function. Score = 10- distance from center. If your ball lands exactly on center then you get 10 points since distance from center is 0 and if you ball lands outside the board or on edge then you get 0 points.</p><p>Now if I ask is the probability that your score is 10 (hitting a bulls eye) then its very very low because we need to hit at exact center i.e P(x=10). But if I set a range say around center in range of say +or- 0.5 then P(x=9.8) or P(x=9.9..) etc are in our reach. We can see that our random variable can take continuous values.For every value that our x can take P(x=xi) will give us a probability value. In discrete case i.e colored balls case we will have discrete values . In colored ball example</p><p>P(x=red ball) is 0.5 i.e 5/10</p><p>P(x=blue ball) is 0.3 i.e 3/10</p><p>P(x=green ball) is 0.2 i.e 2 / 10</p><p>where 10 is total number of balls.</p><p><strong>Notice how sum of all probabilities adds up to 1.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*0iJ3rruRIBee9K0leoYxXA.png" /></figure><p>0.5+0.3+0.2 = 1. If you think about it,if you pick all balls at once the probability of picking all balls must be 1.</p><p>Now lets plot a graph for this where my y axis hold probability values and x holds discrete states.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qXCO3KjfvzYPf6qRWlugRg.png" /></figure><p>In case of dart board we can plot a graph. Where x axis holds random variable taking a distance(X=distance from center) or y axis holds probability of getting that score. In a real world lets say we perform the dart board experiment on a medium windy day where wind can blow from any direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*b51Kn_2mZ5xkURP_tZVVaQ.png" /></figure><p>Just like how in discrete case how all probabilities added up to 1 here</p><p><strong>Area under the curve sums to 1.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/91/1*b8MCptNw9K7vJnRsF4CORQ.png" /></figure><p>Where P(x) is probability that our ball lands in that region. Since our dart board is of radius 10 cms. we integrate from -10 to 10. Negative is convention to indicate edge points on left half of board.</p><p>The plot says we you were to keep dropping the ball on dart board 40% of the time it lands in center and rest it will fall around it due to slight wind.</p><p><strong>Probability distribution functions</strong></p><p>Now that we have an intuition of how probability plots looks like we can move on to Probability distribution functions. I.e Can we write a function which can plot those curves. In continuous case we can plot those continuous curve but for discrete case we can plot a scatter plot.</p><p>In continuous case we call these functions which can plot probability distribution as “Probability distribution function” or PDF and in discrete case we call them “Probability Mass function” or PMF.</p><p>P(X=xi) means P is a probability distribution function distributed over X where X can take any value xi. In case of dart board P is a probability distribution function distributed over range -10 to +10 were xi can be any fractional number between -10 to +10.</p><p>Alright, this introduction should be good for us to start. In the next part we will dive deep into probability distributions and its applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ae260219253" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>