<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sri Anumakonda on Medium]]></title>
        <description><![CDATA[Stories by Sri Anumakonda on Medium]]></description>
        <link>https://medium.com/@srianumakonda?source=rss-ff11722a4de3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KKLslvIRpV_1X8wNRozh-g.png</url>
            <title>Stories by Sri Anumakonda on Medium</title>
            <link>https://medium.com/@srianumakonda?source=rss-ff11722a4de3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:18:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srianumakonda/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Everything you need to know about Self-Driving Cars in <30 minutes]]></title>
            <link>https://srianumakonda.medium.com/everything-you-need-to-know-about-self-driving-in-30-minutes-b38d68bd3427?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b38d68bd3427</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cars]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Thu, 13 Jun 2024 20:50:34 GMT</pubDate>
            <atom:updated>2024-06-24T17:03:03.410Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B9rOawvb10VBKyCT.jpg" /><figcaption><a href="https://www.cntraveler.com/story/googles-waymo-partners-with-jaguar-on-fancy-new-self-driving-car">source</a></figcaption></figure><h3>A quick note!</h3><blockquote>I’ve spent the past week learning what it means to create self-driving cars that create value while also reviewing its technical intricacies. This article is a version of the notes that I took while going through <a href="https://www.youtube.com/playlist?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Andreas Geiger’s autonomous vehicle lectures.</a> It also will serve as a reference to beginners who want to learn more about how self-driving cars work!</blockquote><h3>The history of autonomous vehicles, jot-note style</h3><ul><li><strong>1925:</strong> Houdina creates the “American Wonder” which is remotely controlled by another operator in a different vehicle</li><li><strong>1960:</strong> RCA Labs creates a wire-controlled car that allows the car to move on its own through electronic wires in the ground</li><li><strong>1970:</strong> the Citroen DS19 is able to drive up to speeds of 130 km/h via magnetic fields in the ground</li><li><strong>1986:</strong> CMU’s Navlab creates its initial demonstration of vision-based autonomy without any sort of infrastructure/radio communication</li><li><strong>1986:</strong> VaMoRs is demonstrated as the first autonomous vehicle that’s able to drive both autonomously longitudinally and laterally with speeds up to 36 km/h</li><li><strong>1988:</strong> ALVINN is a neural-network-based approach to driving on 30x32 input images</li><li><strong>1995:</strong> “Navlab 5” is able to go all across America (2850 miles) with 98% lateral autonomy</li><li><strong>1995:</strong> AURORA is created as a way to lane-keep by looking at line markings on the ground via a downward camera</li><li><strong>1995: </strong>Adaptive Cruise Control is created and deployed into vehicles, which creates level 1 autonomy</li><li><strong>2004: </strong>the first DARPA Grand Challenge occurs in the Mojave Desert. None of the vehicles completes the race, with CMU travelling 11.78km of the 240km route (furthest distance). GPS is also used.</li><li><strong>2005:</strong> Stanford wins the DARPA Grand Challenge 2 with 5 teams finishing the course</li><li><strong>2006:</strong> High-resolution LiDAR’s are created that have 360ᵒ vision</li><li><strong>2007:</strong> DARPA Urban Challenge occurs where traffic and obstacles are put into place. CMU wins the race, while Stanford comes second.</li><li><strong>2009:</strong> Google’s Self-Driving Car project is started at Google X (which became Waymo in 2016)</li><li><strong>2012:</strong> Deep Learning methods are introduced into autonomous vehicles i.e. depth estimation, pose estimation, etc.</li><li><strong>2015:</strong> Uber starts their self-driving car division (which then gets shut down in 2020)</li><li><strong>2015:</strong> Tesla Model S introduces its Autopilot</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OYUlU52OEvWoTIDS.png" /><figcaption><a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.geospatialworld.net%2Fblogs%2Ffive-levels-of-autonomous-cars%2F&amp;psig=AOvVaw1DCPfwBfP4bwawwBTDGpGk&amp;ust=1716946458882000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=0CBQQjhxqFwoTCMCeoZ-ar4YDFQAAAAAdAAAAABAE">Source</a></figcaption></figure><h3>The main approaches to self-driving</h3><h4>Modular Pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cTaPWz_tg5V3l9ngYfF8Q.png" /><figcaption><a href="https://www.youtube.com/watch?v=twMHsKYtHKA&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=4">Source</a></figcaption></figure><p>The main idea behind the <strong>modular approach </strong>to autonomous vehicles is that you can break the fundamental problem down into <strong>several sub-parts </strong>(problems) that you can solve independently and then put together. The main problems they’re usually broken into are perception, localization, planning, and control.</p><p><strong>Perception + scene parsing</strong> is all about <strong>detecting vehicles</strong> and understanding all the objects that are in your environment. This would include objects such as people and other vehicles, buildings, lane markings, trees, etc. This is mainly done through <strong>computer vision and LiDAR-based methods</strong>. Algorithms such as bounding boxes, semantic segmentation, depth estimation, optical flow, and 3d map reconstruction as a way to parse through our scene and recreate the environment the car is in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ByspIBHgs3vEn-5Y.gif" /><figcaption>Notice how in the gif we’re able to place bounding boxes on objects, create an estimation of distances between the car and other objects, and determine the area we can drive on! <a href="https://github.com/prismformore/Multi-Task-Transformer/blob/main/TaskPrompter/README.md">Source</a></figcaption></figure><p>If you’re using a LiDAR with cameras, then you would generally perform sensor fusion here, where you would combine both your LiDAR representation and camera representation into one big 3D map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/0*bF7m1qIQPP1JWkYD.gif" /><figcaption><a href="https://learnopencv.com/wp-content/uploads/2023/12/3d-lidar-object-detection-inference-3.gif">Source</a></figcaption></figure><p>The second step would be <strong>localization</strong>. The fundamental problem behind localization is the idea that <strong>GPS data isn’t fully accurate;</strong> autonomous vehicles need to be accurate to the centimetre-level while GPS is at the meter-level. Although GPS can be used as a higher-level understanding of the route that needs to be taken, autonomous vehicles need to know exactly where they are in order to make better decisions (ex. overtaking, slowing down, etc.).</p><p>The way localization is approached is <strong>through a series of measurement and update steps:</strong> you “measure” information in the real world (ex. vision and LiDAR data, along with any prior information you’re given), and then update your possible location from a uniform distribution to the “potential” specific location the vehicle is at.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*iCTWv5V1P6iYHRKV.gif" /><figcaption><a href="https://developer.nvidia.com/blog/drive-labs-how-localization-helps-vehicles-find-their-way/">Source</a></figcaption></figure><p>The third step is <strong>planning</strong>, which is all about determining the <strong>optimal trajectory</strong> the vehicle needs to take. The goal here is that given the higher-level route information (GPS), you want to be able to create the trajectory/plan that’ll allow you to get there.</p><p><strong>Control </strong>is then all about <strong>executing the planning stage </strong>i.e. steering, brake, and gas. Based on the trajectory created, the autonomous vehicle should be able to adjust your steering + speed accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*olQGGHgfPH3sNck_.gif" /><figcaption><a href="https://www.mathworks.com/help/examples/autonomous_control/win64/HighwayTrajectoryPlanningUsingFrenetReferencePathExample_01.gif">Source</a></figcaption></figure><p>The biggest <strong>advantage </strong>to modular-based systems is the idea that they’re <strong>interpretable</strong>. Because of the fact that the fundamental problem of autonomous navigation is broken down into immediate steps that humans can understand, it’s <strong>easy to debug during systematic issues</strong> + deployment failures.</p><p>The <strong>problems</strong>, though, lie in the fact that there is no such thing as “joint training” between all modules. This would mean that <strong>different modules are trained on different objectives</strong> and different data; none of the modules think about solving self-driving as a whole but instead solve specific aspects of it (ex. get 100% accuracy on object detection when we might not want to waste compute thinking about parked cars).</p><p>The <strong>primary problem </strong>with the modular approach is its <strong>reliance on pre-mapped environments and HD maps.</strong> Whenever you localize through a modular approach, you need to already have an HD map of the environment you’re driving through: the idea of being able to drop the autonomous vehicle into a new place and have it “magically” work doesn’t exist. Scalability is restricted to the geographical areas that have been pre-mapped with LiDARs and camera sensors.</p><h4>End2End Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qK2ONS5QfAdlGetO5G0p6A.png" /><figcaption><a href="https://www.youtube.com/watch?v=twMHsKYtHKA&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=4">Source</a></figcaption></figure><p>End2end learning (aka imitation learning) is almost the opposite of the approach of the modular approach where there aren’t modules that break the task of driving up into separate sub-tasks. Instead,<strong> sensory inputs go through a neural network </strong>(just like a black box) and <strong>output the direct controls</strong> needed to drive the car (steering, brake, gas). This not only deals with the problem of “joint training” in the modular approach, but <a href="https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping">advances in localization</a> allow for these vehicles to be dropped anywhere in the world and be able to operate.</p><p>The biggest <strong>advantage </strong>to end-to-end learning is how <strong>easy it is to train and deploy</strong> these models; the only things you need are the sensory input (which could be cameras and/or LiDAR) and the output controls. That’s all fed into the neural network, which can learn and train on this information.</p><p>The <strong>problem fundamentally lies in interpretability. </strong>Debugging end2end models and understanding why they made the decisions that they did is extremely difficult as there’s no point of entry to understand the decisions that are made + why they were made. It’s an existing problem that has yet to be solved, but there hasn’t been monumental progress in this area yet.</p><p>The <strong>goal</strong> of end2end learning is to <strong>“imitate” </strong>the actions of what the human is doing i.e. create some policy that best mimics the actions of the data. Instead of hard-coding policies + breaking down the fundamental problem of driving into several sub-tasks, let’s instead use a<strong> data-driven approach</strong> where the only restriction on the policy’s performance is the amount of data that it’s given.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9IjAaoT6q8-WxfbBQtvNQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=XQuLVDEMm1g&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=7">Source</a></figcaption></figure><p>Prof. Geiger defines the <strong>formal definition </strong>of imitation learning as the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hP403zOYMvGeBfLUtHrzKQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=XQuLVDEMm1g&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=7">Source</a></figcaption></figure><p>Imitation learning is all about creating some sort of policy, <strong>π</strong> that’s able to take in information about the current state <strong>s</strong> and figure out the best action <strong>a</strong> given <strong>π</strong>, while minimizing the loss function <strong>𝔏</strong>. This would then be “rolled out” through <strong>n</strong> amount of frames where you iteratively update your states (which is modelled through a probabilistic distribution) and actions (by repeatedly placing <strong>sᵢ </strong>into <strong>π</strong>).</p><p><strong>Conditional Imitation Learning</strong></p><p>Imitation learning isn’t just enough. You need to be able to<strong> incorporate higher-level route information</strong> and not just predict for the next frame (i.e. get general directions so that you can decide whether to plan to move to the left, right, etc.). The goal behind conditional imitation learning is to be able to <strong>take in a directional command</strong> (ex. left, right, straight) along with the current state-environment observation <strong>into the end2end model.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v9rczLenzEEvFMjmX_eNlg.png" /><figcaption><a href="https://youtu.be/U5kZ1uRPx7o?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Most companies that use end2end learning as a way to solve self-driving will generally follow the conditional imitation learning route (<a href="https://wayve.ai/">wayve</a>, <a href="https://comma.ai/">comma</a>). This is mainly because it’s the <strong>easiest way to incorporate navigational information </strong>into the model, allowing the autonomous vehicle to be able to function in the real world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CtaHNkXMR6aMpWbqbjMWmg.png" /><figcaption><a href="https://arxiv.org/pdf/1710.02410">Source</a></figcaption></figure><p>There’s generally <strong>two main approaches </strong>to “command-conditional imitation learning”: you can have an <strong>image encoder and then concatenate </strong>state-environment observations with your command, <strong>or </strong>you can instead create <strong>three possible state-action pairs given a command </strong>and then execute based on the input command (i.e. I can create hypothetical actions that would be taken if I wanted to go left, right, or straight and then choose the hypothetical path based on the command).</p><h4>Direct Perception</h4><p>The “goldilocks zone” between the modular + end2end approach is exactly what Direct Perception is all about; instead of having one big network that goes from perception → control, you instead <strong>have a network that goes from perception → low-dimensional intermediate representations, and then a planning + control module.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJAiVLc-hdzyKjxnkwCH5A.png" /><figcaption><a href="https://www.youtube.com/watch?v=m60wi2-8Mwo&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=8">Source</a></figcaption></figure><p>This <a href="https://deepdriving.cs.princeton.edu/paper.pdf">paper</a> from the Princeton Vision Group goes deeper into how direct perception works, along with a proposal on driving focused on lane-keeping.</p><p><strong>Conditional Affordance Learning</strong></p><p>Conditional Affordance Learning is what allows us to scale Direct Perception techniques to inter-city driving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2NMhg_Jc8KCxk5cw6kuVA.png" /><figcaption><a href="https://youtu.be/tJhcHSmBbmk?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>After you take in video + directional input, you’ll use a <strong>neural network to predict affordances</strong> (which are your centerline + other distances relative to the lane that the ego-vehicle is in) and then use those parameters as an input to the controller. The only difference here is that <strong>directional input is added </strong>as an input to the neural network.</p><h4>Visual Abstractions</h4><p>Sometimes, <strong>intermediate representations </strong>of our driving scene can actually <strong>allow for better end2end performance</strong> of autonomous vehicles. A couple <a href="https://arxiv.org/abs/1905.12887">folks from UT Austin</a> experimented with adding intermediate representations to end2end models and found out that those representations resulted in better performance than pure end2end learning.</p><p>What do we define as a good visual abstraction, though? Prof. Geiger breaks it down into <strong>4 main categories</strong>:</p><ul><li><strong>invariant</strong>: irrelevant noise from multiple images (ex. colors of cars) should be eliminated</li><li><strong>universal</strong>: the abstraction should be able to explain the entire environment/driving scene</li><li><strong>data efficient</strong> with respect to computational power</li><li><strong>label efficient</strong>: minimal effort should be required</li></ul><p>Intermediaries such as <a href="https://en.wikipedia.org/wiki/Image_segmentation#Partial_differential_equation-based_methods">semantic segmentation</a>, <a href="https://en.wikipedia.org/wiki/Depth_perception">depth</a>, and <a href="https://en.wikipedia.org/wiki/Optical_flow">optical flow</a> have been mainly used in applications of conditional imitation learning as a way to help the model better understand the environment that it’s in.</p><h3>Reinforcement Learning</h3><p>The idea of Reinforcement Learning (RL) in autonomous vehicles has been experimented with for decades now. Instead of having to create extensive datasets and labels to train a policy, you can instead have it <strong>directly interact with its environment and allow the policy to become stronger over time </strong>as it interacts more and more with the environment. I’ll go over the fundamental ideas behind RL in this section (MDPs, the Bellman Optimality, and Q-Learning). If you’re curious to learn more how RL works, I wrote an <a href="https://srianumakonda.medium.com/everything-you-need-to-know-about-reinforcement-learning-in-80minutes-4cd5a365e340">80-minute article</a> going deep into the science of RL.</p><p>This is what RL generally looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*8ozFpiY5W76_i0v-hhmZUA.png" /><figcaption><a href="https://www.youtube.com/watch?v=GYnlqiSqZiU&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=13">Source</a></figcaption></figure><p><strong>Markov Decision Process (MDPs)</strong></p><p>The fundamental goal of all RL agents is to <strong>maximize total future reward, </strong>where the agent might need to make short-term sacrifices in order to maximize reward in the long-term. We formalize the Reinforcement Learning problem through an MDP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3a3HGNGFnlpC6LKvIm3z4A.png" /><figcaption><a href="https://www.youtube.com/watch?v=GYnlqiSqZiU&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=13">Source</a></figcaption></figure><p>All MDPs must obey the Markov property, which is the idea that the current state must be only dependent on the previous state (which implies that all the previous states are irrelevant except for the immediate previous state). This then brings us to the RL loop:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vLBzQw8Cl4dWld7LBuUAPA.png" /><figcaption><a href="https://www.youtube.com/watch?v=GYnlqiSqZiU&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=13">Source</a></figcaption></figure><p>The way the actions are chosen are through our policy, <strong>π</strong>. There’s <strong>two types </strong>of policies: <strong>deterministic policies </strong>(direct mapping), or <strong>stochastic policies </strong>that try to model the uncertainty of the environment into the policy. The way good actions are discovered is through exploration (a big reason why RL isn’t deployed in the real world and mainly in simulation). The <strong>balance of exploration</strong> (trying a new action) <strong>and exploitation</strong> (following through with previous actions) is what allows for the <strong>optimal policy</strong> to be created.</p><p>The idea of balance is mainly done through an <strong>ϵ-greedy algorithm</strong>, where <strong>probability ϵ is your action chosen at random </strong>(and probability <strong>1-ϵ being to choose the best action</strong>). ϵ starts off as a large value and becomes smaller over time.</p><h4>Value Functions</h4><p>The idea behind value functions is that you want to be able to calculate the cumulative (total) reward from your current state and the policy that’s chosen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJsXPmwOT3_49Jw9M4I-0g.png" /><figcaption><a href="https://www.youtube.com/watch?v=ppvxzV_iHyY&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=14">Source</a></figcaption></figure><h4><strong>Action-Value Functions</strong></h4><p>Action-value functions are slightly different; instead of calculating the return just based on the current state, you’re <strong>calculating the cumulative reward </strong>based on the action that’s taken.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sWEMQfHZKnpimA3aLBQ4UQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=ppvxzV_iHyY&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=14">Source</a></figcaption></figure><h4>Bellman Optimality</h4><p>Finding the optimal action for both the state-value function and the action-value function is hard because of the<strong> infinite amount of states and actions</strong> that you can be in. What we can do is use a B<strong>ellman Optimality Equation </strong>for both functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vq0TiPZJIA9CxF1xrNHfxA.png" /><figcaption><a href="https://www.youtube.com/watch?v=ppvxzV_iHyY&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=14">Source</a></figcaption></figure><p>Instead of considering all the possible timesteps into the future, we <strong>look directly at the next timestep and recursively iterate</strong> through that over time. How do we find Q*, though? This is where other methods, such as Q-learning, come into play:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpjKCxw0GasqUWqHUuu1uA.png" /><figcaption><a href="https://www.youtube.com/watch?v=ppvxzV_iHyY&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=14">Source</a></figcaption></figure><h4>Deep Q-Learning</h4><p>Q-Learning can’t scale to high-dimensional spaces because of the amount of possible states that exist. Instead of creating a computationally inefficient Q-table, we can <strong>create a function approximation</strong> (neural network) to estimate what Q would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*6mE0a1PXmC0zvsiOlQU61Q.png" /><figcaption><a href="https://youtu.be/Q8qgjUqj5ss?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Our <strong>loss function</strong> would just be a straightforward <strong>MSE </strong>in our Q-values, while our <strong>gradient update</strong> would be with respect to the <strong>Q-function parameters:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ZCLOsV2MxluSZQUAIM8UQ.png" /><figcaption><a href="https://youtu.be/Q8qgjUqj5ss?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>The problem is that this doesn’t converge in practice. To solve this problem, we need to do two things: experience replay and add <strong>delays to our Q-network updates</strong>.</p><p>We can <strong>create a replay memory </strong>that stores our new “experiences” (state, action, read, and next state) and then train on those samples. This allows us to train on more recent states and not let older states influence the neural network.</p><p>We also <strong>change the set of weights that we use </strong>in order to reduce oscillations in our target data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xcjxjc932jYV7NWihwFmCQ.png" /><figcaption><a href="https://youtu.be/Q8qgjUqj5ss?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>This is what the <strong>final procedure</strong> would look like:</p><ol><li>Take an <strong>action </strong>based on our ϵ-greedy policy</li><li><strong>Store </strong>this information (state, action, reward, next state) into our replay memory</li><li><strong>Sample </strong>a couple of transitions from our replay memory</li><li><strong>Calculate </strong>the Q-targets using our older parameters (θ⁻)</li><li><strong>Optimize </strong>the loss function (MSE) via stochastic gradient descent (SGD).</li></ol><h4>Deep Deterministic Policy Gradients (DDPG)</h4><p>Deep Q-Learning suffers from the fact that the set of possible actions are discrete and from uniform sampling of our replay buffer. DDPG instead focuses on creating a policy that allows for continuous action spaces. What we can do is create<strong> 2 networks</strong>: a <strong>deterministic policy that maps from a state → action</strong> and a <strong>critic </strong>that helps us <strong>model the Q-function.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*X1_B2H_rsxAfwl6i-3ktzw.png" /><figcaption><a href="https://youtu.be/Q8qgjUqj5ss?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RCNpxoSTt0Nh16GhtFZg5g.png" /><figcaption><a href="https://youtu.be/Q8qgjUqj5ss?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Experience replay and target networks are also used to make sure that oscillations are minimized during training.</p><h3>Visual Odometry</h3><p>Visual Odometry is all about understanding the <strong>motion </strong>of our autonomous vehicle with respect to images. There’s <strong>2 approaches</strong> to tracking pose (position + orientation) of our autonomous vehicle: <strong>indirect </strong>and <strong>direct methods.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oYBntPUT4grP32rfbTfreg.png" /><figcaption><a href="https://www.youtube.com/watch?v=qdjxd7LSjF0&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=25">Source</a></figcaption></figure><h4>Indirect visual odometry</h4><p>The first step with indirect visual odometry is to <strong>find key points</strong> (salient points) and <strong>extract features </strong>from them. The goal is to ensure that our feature extraction algorithm is invariant to changes in lighting and perspective to maximize performance. Using algorithms like <a href="https://en.wikipedia.org/wiki/Oriented_FAST_and_rotated_BRIEF">ORB</a>, <a href="https://en.wikipedia.org/wiki/Features_from_accelerated_segment_test">FAST</a>, and <a href="https://en.wikipedia.org/wiki/Scale-invariant_feature_transform#:~:text=SIFT%20feature%20matching%20can%20be,nearest%2Dneighbors%20for%20each%20feature.">SIFT</a>, we can then perform feature matching between two images to find similarities.</p><p><strong>2D → 3D</strong></p><p>We need to be able to convert from 2D points to 3D points in order to perform accurate projections. We do this by <strong>converting an inhomogeneous vector x = [x y]</strong> into a <strong>homogenous vector x~ = [x y w~]. </strong>By converting a 2d point into a 3d point, we’re able to perform all of our operations (rotations, shifts, scaling, translations) <strong>through matrix multiplication</strong>, therefore simplifying operations and making the overall process easier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VXtsuNc7KJqzFsp2Uolaw.png" /><figcaption><a href="https://www.youtube.com/watch?v=qdjxd7LSjF0&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=25">Source</a></figcaption></figure><p>The way we’re able to convert from 3D → 2D points is through the calibration matrix <strong>K</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WX4CuTLa3lhAZsFjRbL0aQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=qdjxd7LSjF0&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=25">Source</a></figcaption></figure><p><strong>Epipolar Geometry</strong></p><p>The goal behind epipolar geometry is to understand the motion in a scene between 2 images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cedtwM0yuQx4J5RqicsHzA.png" /><figcaption><a href="https://www.youtube.com/watch?v=qdjxd7LSjF0&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=25">Source</a></figcaption></figure><p>The idea behind epipolar geometry is that we have some sort of <strong>essential matrix </strong>that captures the projective geometry on 2 images (i.e. where would x₁ on the second image). If we multiplied <strong>x₁ᵗ by F and x₂</strong> (<strong>x₁ᵗFx₂ = 0</strong>), we’d get 0 as this would mean that <strong>x₁ᵗ lies on the same plane as x₂</strong> (note that <strong>ᵗ denotes transpose </strong>in this instance). We can use non-linear optimization to optimize our <strong>R</strong> (rotation matrix i.e. orientation of the camera in the world coordinate system) and <strong>t </strong>(the position of the camera’s center in the world coordinate system)<strong>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkOit4rT_OjI8B559YyIrQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=qdjxd7LSjF0&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=25">Source</a></figcaption></figure><p>What we’re doing here is essentially taking the <strong>absolute magnitude in the difference of the predicted points </strong>x₁ and x₂ in order to minimize reprojection errors. Therefore, we need only 4 images (2 from the left and two from the right).</p><p><strong>Note:</strong> one of the biggest problems with monocular visual odometry is that we <strong>aren’t able to determine global scale</strong> (the relative distances from reconstruction into absolute, real-world measurements) because depth information cannot be extracted in a monocular camera.</p><h4>Direct visual odometry</h4><p>In direct visual odometry, we completely <strong>skip the step of keypoint detection </strong>and instead <strong>use depth from sensors</strong> (RGB-D, LiDAR) to create a 3D map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UNF_y5DftDU3RIrq7fLJxw.png" /><figcaption><a href="https://www.youtube.com/watch?v=qdjxd7LSjF0&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=25">Source</a></figcaption></figure><p>If we know the per-pixel depth given the image intensities <strong>I₁</strong> and depth map <strong>D₁, </strong>the image can be simulated from multiple viewpoints.</p><h4><strong>Simultaneous Localization and Mapping (SLAM)</strong></h4><p><strong>Note:</strong> for this section, I assume you have a basic understanding of what localization is. If you don’t, check out <a href="https://srianumakonda.medium.com/who-and-where-are-we-in-this-world-d5636f58455c">this article</a> that I wrote explaining how localization + particle filters work!</p><p>The idea behind feature-based SLAM is to <strong>optimize reprojection errors.</strong> We do this through an approach known as <strong>Bundle Adjustment</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l1tch9Pd34Gjc0rhM_LCew.png" /><figcaption><a href="https://youtu.be/8nZy1cZFoZM?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Note that N denotes our camera views and P landmarks that we’re observing. We minimize Π and Xᵥᵥ with respect to all the cameras.</p><p>We can also perform <strong>loop closure detection</strong>, where our camera ends up back at the same location and determines that based on the landmarks in its environment.</p><h3>Image Reconstruction</h3><p>Given two stereo images, we can <strong>calculate </strong>the <strong>change in pixel movements </strong>(disparity) to create a disparity map, allowing us to study the motion of our vehicle as it moves in its environment. We then would use this disparity map (which can also be thought of as a depth map) to create a 3D structure of our environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hSDXxr4akXI-EsQAUP8xZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnzVUL_jmplMPgNKc48Dbw.png" /><figcaption><a href="https://youtu.be/SOERX7QUlJ8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a> (note that the 160 means the amount of pixels shifted)</figcaption></figure><p>For binocular stereo matching, our goal is to create a 2.5D disparity map from 2 images of our scene (static). The <strong>overall process </strong>would look like the following:</p><ol><li><strong>Calibrate </strong>both of our cameras and apply this to our images</li><li>Create the <strong>disparity map</strong></li><li><strong>Remove </strong>any sort of <strong>outliers </strong>in our disparity map</li><li>Using triangulation, <strong>calculate depth</strong></li><li>Construct a <strong>3D model</strong></li></ol><p>Since our cameras are already calibrated, this would mean that we already know our rotation and translation matrices. Therefore, what we’re <strong>solving for are the corresponding points between our left and right cameras</strong>. The benefit by knowing our <strong>R</strong> and <strong>t</strong> matrices is from the fact that we <strong>know the essential matrix,</strong> which would mean that we’re able to <strong>leverage the epipolar constraint </strong>to make <strong>correspondence a 1D problem:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbVB0RS_liBEajqH-CS5iQ.png" /><figcaption><a href="https://youtu.be/SOERX7QUlJ8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>When we look at the point x₁ on the left image, the line that x₁ will project onto the right camera (as denoted by l₂). Therefore, finding a correspondence for x₁ would mean that we have to <strong>search only on the epipolar line l₂</strong>. We do this through a process known as Image Rectification.</p><p>The idea behind <strong>image rectification </strong>is to rewarp our cameras to be perfectly parallel to each other given our calibration matrix <strong>K</strong> and the essential matrix <strong>E</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-AqfIFgFogejQYXBup7Yg.png" /><figcaption><a href="https://youtu.be/SOERX7QUlJ8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>We can then use a <strong>sliding window implementation </strong>(block matching)<strong> </strong>to go through the entire line and select the window with the highest correspondence. We can normalize the feature vectors to account for lighting/illumination changes and then calculate the zero normalized cross-correlation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJ1Pgd-Ck0XDn9RIfwoMTg.png" /><figcaption><a href="https://youtu.be/SOERX7QUlJ8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>To deal with outliers, we’d use a <strong>Left-Right Consistency Test</strong> where we calculate a disparity map for both the left and right cameras and then verify to see if the disparity pixel value matches each other. For all the pixels that don’t have the same disparity value, you can just delete them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GC0SjMDhQgNQZ4PNt6rwdg.png" /><figcaption><a href="https://youtu.be/SOERX7QUlJ8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>You can then use <strong>basic geometry to determine depth</strong> from disparity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7sBqx-ZUuSc8mFQ_qz-Qlg.png" /><figcaption><a href="https://youtu.be/SOERX7QUlJ8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h4>Optical Flow</h4><p>Optical flow is the idea behind understanding the <strong>motion of objects in your scene over time</strong>. The reason why we’d want to study optical flow is that understanding the motion of objects (other cars) over time can allow us to predict locations + future trajectories that the cars are going on. Let’s use the <strong>Horn-Schunck Optical Flow </strong>model to better understand how flow works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*syYryOfCY8JuMn6HfexMlw.png" /><figcaption><a href="https://youtu.be/CrazEFoJdj8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Note that <strong>u(x,y) and v(x,y) represent the flow in the x and y direction</strong>, respectively. The main idea behind the energy functional is that we <strong>displace the current pixel </strong>(x,y) <strong>by their optical flows </strong>(u and v) in the <strong>next timestep </strong>(think of it as us predicting the motion at timestep <strong>t+1</strong>) and then subtract that from the intensity at the current timestep. If our optical flow is estimated correctly, then we would be getting an extremely small value (since our intensity i.e. pixel brightness, would stay the same).</p><p>We also want to make sure that the flow field to remain smooth without any sort of sudden shifts in the image. By taking the <strong>magnitude of the gradient</strong>, we’re able to <strong>penalize for large variations in the flow</strong>, which is then <strong>multiplied by regularization parameter λ</strong>.</p><p>The main <strong>problem </strong>with the energy functional is that minimizing it is extremely complicated as a result of the <strong>non-convex nature</strong> of the function (which would mean that we have several local optima). What we can do is instead<strong> linearize the brightness constancy assumption (</strong>intensity remains the same over time)<strong>.</strong></p><p>We can do this by taking a first-order Taylor series of the intensity function, as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULMLY6AuC0AKVjPRJXozgw.png" /><figcaption><a href="https://youtu.be/CrazEFoJdj8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Plugging that back in gives us this linearized energy functional:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Jg7YrEos2DD5IF5E8_l4A.png" /><figcaption><a href="https://youtu.be/CrazEFoJdj8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p><strong>We can now discretize </strong>(which essentially means that now we have numerical values for each x,y index of the flow and isn’t a function anymore) the flow maps (u and v → <strong>U</strong> and <strong>V</strong>), <strong>differentiate wrt. U and V</strong> and <strong>set the gradient to zero</strong>, yielding us a linear system that can be solved:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fzmCIgHxtVj9Rv2l73vufw.png" /><figcaption><a href="https://youtu.be/CrazEFoJdj8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Deep Learning has made a huge emergence in its applications to Optical Flow. <strong>U-Net-like architectures</strong> (encoder-decoder) have allowed for accurate predictions of flow in several applications:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLbcOt773fu1mzKUI2R2fA.png" /><figcaption><a href="https://youtu.be/CrazEFoJdj8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h3>Vehicle Dynamics</h3><p>To control a car, you need to understand how a car moves. That’s exactly what vehicle dynamics is all about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrlEX79LT2gFI8fie9JxWw.png" /><figcaption><a href="https://youtu.be/wuUUN_DvYP4?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Since cars can’t move sideways, our <strong>velocity will always be in a nonholonomic system</strong>, but the cars are <strong>grounded by 3 holonomic constraints.</strong></p><h4>Coordinate Systems</h4><p>Coordinate systems define vehicle motion. There are generally <strong>3 main coordinate systems </strong>used: <strong>inertial</strong>, <strong>vehicle</strong>, and <strong>horizontal frame motion.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Es1RENRFBrVyOCWpPvUnQ.png" /><figcaption><a href="https://youtu.be/wuUUN_DvYP4?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h4>Kinematics of Rigid Bodies</h4><p>A rigid body is essentially some sort of mass (like a car). The motion of a rigid body is mainly described through a reference point (which could be its center of mass, for example) and the relative motion of all the other points in the rigid body:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovPQb-tE6gV_Wnud0IrJ_w.png" /><figcaption><a href="https://youtu.be/wuUUN_DvYP4?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>There’s always a linear + rotational component when we describe the motion of a rigid body (which is where ω comes from).</p><h4>Kinematic Bicycle Model</h4><p>This is one of the simplest models of motion, where we assume that there is no slipping but the wheels are always in the direction of our velocity vector. The kinematic bicycle model assumes that cars work on 2 wheels (like a bicycle) and not 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9aySs_ZMjGRMzSsnzcfP9g.png" /><figcaption><a href="https://youtu.be/wuUUN_DvYP4?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>We can then use the vehicle velocity vector as our way to determine our x, y, ψ, and β:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VFlBFFfvlsSlcZiuzqAByw.png" /><figcaption><a href="https://youtu.be/wuUUN_DvYP4?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>The problem with the bicycle model is that if the steering angle for both front wheels is the same, then you’d be slipping (which is not what we want, and also why those two angles are different). We can use Ackermann Steering Geometry to help with this in smaller angles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O-U2eR4_-3lr7HVC_SY15g.png" /><figcaption><a href="https://youtu.be/wuUUN_DvYP4?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h4>Dynamic Bicycle Model</h4><p>Since the mass of a rigid body isn’t concentrated in a singular point (center of mass), you need to integrate a density function ρ over the space of the object to get its mass:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZpih1yTzukQ4xNR7wqmig.png" /><figcaption><a href="https://youtu.be/6fyUnoRxPvs?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Your inertia tensor changes if you assume that the origin is in the rigid body:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kPB4YITKO4ovZZ9Yj6nwng.png" /><figcaption><a href="https://youtu.be/6fyUnoRxPvs?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Putting this all together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvC69gCEhQnnpeMgTtFyow.png" /><figcaption><a href="https://youtu.be/6fyUnoRxPvs?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h3>Path Planning</h3><p>Once you understand the scene + all the objects around your vehicle, you need to create a trajectory that you can execute on. Path Planning is all about decision-making and <strong>creating a plan that you can execute on</strong> that helps you get from your current location → destination. We can break this down into a series of simpler steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9-a0xmb_rXopx0KBFdcfg.png" /><figcaption><a href="https://youtu.be/VAhBWF3PRM8?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>We can create route planning as a graph-based optimization problem where we’re trying to go from point A → B in the shortest distance possible.</p><h4>Breadth-First Search</h4><p>Road networks can be represented as directed graphs with vertices and edges, with each edge corresponding to a road/lane and a node representing a landmark/location. Breadth-first search is all about <strong>creating the shortest route</strong> possible to minimize distance + maximize efficiency when driving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*bf5XdS0_cl5zGWtdqG4CFw.png" /><figcaption><a href="https://youtu.be/1T0trI6yDQc?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>The overarching idea behind breadth-first is that we can use<strong> </strong><a href="https://srianumakonda.medium.com/understanding-data-structures-with-grokking-algorithms-57005ce03a0#0f96"><strong>queues</strong></a><strong> to store paths,</strong> a set of all nodes that we processed, and a dictionary to store elements that we can backtrack on once we find a path. The <strong>problem </strong>with breadth-first search is that that we <strong>aren’t able to accurately represent </strong>road lengths, traffic, and other <strong>exogenous variables</strong>. We instead <strong>need a weighted graph </strong>that can account for this.</p><h4>Dijkstra’s Shortest Path Algorithm</h4><p>The main difference between Dijkstra’s algorithm and something like a breadth-first search is the fact that we’re able to <strong>store information about the costs </strong>of going down a certain path (i.e. weights). We use a <strong>min heap</strong>-like structure where we go down a certain path until the cost of going down that path exceeds the cost of its adjacent node. We then calculate the trajectory to get to the tail via the adjacent node and then repeat the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*hbWRjj5KyZ1H_TxHsxCJVA.png" /><figcaption><a href="https://youtu.be/1T0trI6yDQc?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Dijkstra’s algorithm is <strong>guaranteed to find the shortest path.</strong> Here’s a visual of what that would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/0*aditk6M48R01MpTZ.gif" /><figcaption><a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FFile%3ADijkstra_Animation.gif&amp;psig=AOvVaw3j4JN-jSvsda3XQYqh4oVb&amp;ust=1718388475145000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=0CBIQjhxqFwoTCKCu65iW2YYDFQAAAAAdAAAAABAE">Source</a></figcaption></figure><p>The only <strong>problem </strong>with Dijkstra’s algorithm is the fact that it’s <strong>not computationally scalable </strong>to large graphs (cities). What we can do instead is use <strong>planning heuristics </strong>via eucledian estimates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j76aqSdVGVlzyRUyPN837A.png" /><figcaption><a href="https://youtu.be/1T0trI6yDQc?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h4>A* Algorithm</h4><p>Exploiting planning heuristics is exactly what A* is all about. We <strong>combine Dijakstra’s algorithm and eucledian distances</strong> (add them together) to determine the <strong>optimal trajectory</strong> that we want to follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*YSDlocIFDuN_mtLXNnDubA.png" /><figcaption><a href="https://youtu.be/1T0trI6yDQc?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>Navigation planners like <strong>Google Maps exploit the A* algorithm </strong>to find the optimal path that you need to go when travelling. Now that we’ve created a higher-level route as to how our autonomous vehicle can get to its destination, we now need to follow the route:</p><h4>Behaviour Planning</h4><p>We want to <strong>discretize possible behaviours </strong>that our autonomous vehicle would need to follow when driving (ex. red light, overtaking a vehicle, slowing down at a stop sign, etc.). Behaviour Planning is all about understanding how traffic rules and interactions with the ego-vehicle’s environment determine the actions that it must take.</p><p>We can discretize by using <strong>Finite State Machines </strong>(FSM) to account for all possible situations in driving. Think of it as a series of if-statements and determining the best course of actions based on whether those thresholds are met:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EIieo6RgPbAd-nqlGDgMKg.png" /><figcaption><a href="https://youtu.be/FOww4GONkwQ?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h4>Motion Planning</h4><p>Now that we have some sort of understanding as to the behaviour that needs to be executed from our FSM, we can now calculate a safe trajectory that can be executed. There’s <strong>2 general approaches </strong>to the motion planning problem:</p><ul><li>Variational methods</li><li>Graph-search methods</li></ul><p>The idea behind <strong>variational methods </strong>is to <strong>minimize a functional J(π) over our trajectory π</strong>, where π describes the path of our vehicle over time. Variational methods are often a non-linear problem, which you’ll have to solve via numerical optimization. <a href="https://srianumakonda.medium.com/learning-to-drive-from-a-self-driving-car-a4dda9eb2307"><strong>Here’s a writeup</strong></a><strong> </strong>I did on the <strong>Jerk Minimization Trajectory Algorithm </strong>if you’re interesting on what an application of variational methods looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PaPIZU3GIg0WlRs5SewnFg.png" /><figcaption><a href="https://youtu.be/FOww4GONkwQ?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p><strong>Graph-search methods</strong>, just like in route planning, <strong>discretize </strong>the configuration space into a set of nodes + edges, which you’d then <strong>apply Dijkstra or A* </strong>in order to determine the optimal path that needs to be followed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nt-LCOIh0xt_NZubElAZ5A.png" /><figcaption><a href="https://youtu.be/FOww4GONkwQ?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h3>Vehicle Control</h3><p>Control is the way you’re able to <strong>bridge the gap between software and hardware</strong>. Even if you create a trajectory for a vehicle to follow, you need to be able to break it down into intermediate steps (brake, throttle, steer) and directly interface it with the car. There are <strong>2 approaches</strong> to the control problem: <strong>open-loop</strong> and <strong>closed-loop control.</strong></p><p><strong>Open-loop control</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TP2UXw9IzDHSi8Ix5g-47w.png" /><figcaption><a href="https://youtu.be/O8RENvOxbTY?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>At every point in time, there might be a certain speed that we want to go at (<strong>defined by r(t)</strong>). The controller should take that parameter as input, and then <strong>output the controlled variable y(t)</strong> (ideally approaches r(t) over time) while <strong>taking into account environmental noise</strong> and other factors that need to be accounted for (z(t)).</p><p>Open-loop control isn’t scalable mainly because of the fact that predicting all the noise (disturbances) is impossible in the real world. Any sort of unknown disturbance that occurs in the environment will cause the car to go into chaos.</p><p><strong>Closed-loop control</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ryc1W2P_Um-QKaOLwk5u5A.png" /><figcaption><a href="https://youtu.be/O8RENvOxbTY?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>The difference is that we now have <strong>sensor measurements of the Process</strong> that can then go back into the controller (so that we can compare y(t) and r(t)). Subtracting those two would give us the error which is exactly what the controller is optimizing for.</p><h4>PID controllers</h4><p>There are <strong>3 main values </strong>that come out of a PID controller:</p><ul><li><strong>proportional</strong>: a correcting value that’s proportional to the error, multiplied by the proportional gain Kₚ</li><li><strong>integral</strong>: which integrates the total error over all time steps, multiplied by the integral gain Kᵢ</li><li><strong>differential</strong>: calculates the derivative of the error with respect to time, multiplied by the differential gain K_d</li></ul><p><strong>u(t) is the sum of all these 3 elements together.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2jJAxfcvbalZ5CmT0WhLYQ.png" /><figcaption><a href="https://youtu.be/Gry1bbR1atc?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>The proportional element is the most important element, mainly because of the fact that it’s <strong>proportional to the error</strong>. The problem with using P alone is that it can lead to <strong>overshooting</strong> as we’d be crossing e(t) = 0, making e(t) negative and then overshooting in the opposite direction repeatedly.</p><p>This is where the idea of adding a <strong>differential element </strong>comes in. By calculating the rate of error growth with respect to time, we’re able to introduce a <strong>damping behaviour</strong>, allowing for a smoother acceleration/deceleration.</p><p>The <strong>integral element </strong>corrects for residual errors by looking at changes in our total error, where the desired goal is to keep the error the exact same over time (indicating that we have zero error). This will allow for smoother long-term driving, keeping the error as low as possible.</p><p>How do you select the optimal parameters Kₚ, Kᵢ, K_d? The most common method of doing this is the<strong> Ziegler-Nichols method</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xeyK3XijphpGDcu3282o3Q.png" /><figcaption><a href="https://youtu.be/Gry1bbR1atc?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><h4>Model Predictive Control (MPC)</h4><p>When traveling at high speeds, it’s important to understand the long term so that we can <strong>prepare to adapt to our environment </strong>(ex. don’t maintain high speeds for too long). This is where Model Predictive Control comes into play.</p><p>MPC is a <strong>non-linear optimization system</strong> that allows us to look into the future and determine how we best want the autonomous vehicle to be controlled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e7oH9z50dJ7DB-litq-eiQ.png" /><figcaption><a href="https://youtu.be/r7Gmw3rBoEA?list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr">Source</a></figcaption></figure><p>The goal is to optimize the cost function given parameters δ. <a href="https://www.mathworks.com/help/mpc/gs/what-is-mpc.html">Here’s an article</a> I’d recommend to get an in-depth explanation about how MPC works.</p><h3>Evaluating self-driving cars</h3><p>It’s important to understand the performance of your autonomous vehicle as it scales with data. There are 2 main ways of evaluation: <strong>offline </strong>and <strong>online systems</strong>.</p><p>The idea behind online evaluation is that the model interacts with its environment in real-time, where its actions directly impact the state of <br>the environment that it’s in. In offline learning, you already have a fixed dataset that you’re directly comparing to (where you also know what the ground truths are).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJbGGejmNlzYgCEzA5HPVw.png" /><figcaption><a href="https://www.youtube.com/watch?v=rl5yPBsfc2k&amp;list=PL05umP7R6ij321zzKXK6XCQXAaaYjQbzr&amp;index=12">Source</a></figcaption></figure><h3>Further reading</h3><p>Every single milestone made in the autonomous vehicle industry feels like we’re 10x closer towards owning one; in reality, we’re probably 100x further away from achieving that goal. Solving autonomous vehicles isn’t just about going from point A → B anymore. It’s about exploring the stochastic nature of our world as we continue to navigate and understanding how + why algorithms make the decisions that they do.</p><ul><li><a href="https://arxiv.org/pdf/2001.02522">On Interpretability of Artificial Neural Networks: A Survey</a></li><li><a href="https://arxiv.org/pdf/2309.15268">ObVi-SLAM: Long-Term Object-Visual SLAM</a></li><li><a href="https://arxiv.org/pdf/2311.12320">A Survey on Multimodal Large Language Models for Autonomous Driving</a></li><li><a href="https://arxiv.org/pdf/1710.02410">End-to-end Driving via Conditional Imitation Learning</a></li><li><a href="https://docs.ufpr.br/~danielsantos/ProbabilisticRobotics.pdf">Probabilistic Robotics</a></li><li><a href="https://arxiv.org/pdf/2112.02133">Optimal Trajectory Generation for Autonomous Vehicles Under Centripetal Acceleration Constraints for In-lane Driving Scenarios</a></li><li><a href="https://www.youtube.com/playlist?list=PLpJxwrRy4QbvkeWEkSSBhny4C2FhvQ08i">Tartan SLAM Series</a></li><li><a href="https://arxiv.org/pdf/2111.12137">Learning Interactive Driving Policies via Data-driven Simulation</a></li><li><a href="https://www.youtube.com/watch?v=6x-Xb_uT7ts">[CVPR’23 WAD] Keynote — Ashok Elluswamy, Tesla</a></li><li><a href="https://arxiv.org/pdf/1606.05830">Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age</a></li><li><a href="https://wayve.ai/thinking/lingo-2-driving-with-language/">LINGO-2: Driving with Natural Language</a></li><li><a href="https://arxiv.org/pdf/2401.03629">DDM-Lag : A Diffusion-based Decision-making Model for Autonomous Vehicles with Lagrangian Safety Enhancement</a></li><li><a href="https://szeliski.org/Book/">Computer Vision: Algorithms and Applications, 2nd ed.</a></li><li><a href="https://dspace.mit.edu/bitstream/handle/1721.1/36832/16-412JSpring2004/NR/rdonlyres/Aeronautics-and-Astronautics/16-412JSpring2004/A3C5517F-C092-4554-AA43-232DC74609B3/0/1Aslam_blas_report.pdf">SLAM for Dummies: A Tutorial Approach to Simultaneous Localization and Mapping</a></li><li><a href="https://www.ri.cmu.edu/pub_files/2012/5/ICRA12_xuwd_Final.pdf">A Real-Time Motion Planner with Trajectory Optimization for Autonomous Vehicles</a></li><li><a href="https://arxiv.org/pdf/2310.08710">Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research</a></li><li><a href="https://arxiv.org/pdf/2306.16927">End-to-end Autonomous Driving: Challenges and Frontiers</a></li><li><a href="https://arxiv.org/pdf/2402.01575">Efficient and Interaction-Aware Trajectory Planning for Autonomous Vehicles with Particle Swarm Optimization</a></li></ul><h4>Companies to check out</h4><ul><li><a href="https://srianumakonda.medium.com/the-state-of-comma-ai-2140aabc6f52">comma.ai</a></li><li><a href="http://wayve.ai/">wayve</a></li><li><a href="https://www.tesla.com/">tesla</a></li><li><a href="http://waabi.ai/">waabi</a></li><li><a href="https://waymo.com/">waymo</a></li><li><a href="https://www.getcruise.com/">cruise</a></li><li><a href="https://aurora.tech/">aurora</a></li><li><a href="https://torc.ai/">torc robotics</a></li><li><a href="https://kodiak.ai/">kodiak robotics</a></li><li><a href="https://www.serverobotics.com/">serve robotics</a></li></ul><p><em>Thanks for reading this article! My goal with this article is to help teach more people about how self-driving cars work from scratch and provide entry points into finding a research area that you might be passionate about. It would mean a lot if you shared this resource wherever you can! I spent a lot of time curating notes + creating a good understanding for people to learn from, and my goal is to help as many people as I can!</em></p><h4>Contact me</h4><ul><li>Check out my <a href="http://srianumakonda.com/">website</a></li><li>Connect with me on <a href="https://www.linkedin.com/in/srianumakonda/">LinkedIn</a></li><li>Shoot me an <a href="http://srianumakonda@cmu.edu/">email</a></li><li>Stay on the loop with what I’m building on <a href="https://srianumakonda.substack.com/">Substack</a>!</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b38d68bd3427" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The state of comma.ai]]></title>
            <link>https://srianumakonda.medium.com/the-state-of-comma-ai-2140aabc6f52?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2140aabc6f52</guid>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Wed, 13 Sep 2023 03:55:23 GMT</pubDate>
            <atom:updated>2023-09-13T11:13:38.367Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5iv9LKppdt8HGojq.jpg" /><figcaption>comma.ai <a href="https://www.theverge.com/2016/6/6/11866868/comma-ai-george-hotz-interview-self-driving-cars">7 years ago</a>!</figcaption></figure><blockquote><strong>“Give me $1,250 and I can make your car drive on its own.”</strong></blockquote><p>It’s almost crazy to think that in a world where we’ve spent more than $100 billion in the self-driving industry, we haven’t even truly solved the problem of self-driving cars. <em>When will we get self-driving cars? How far have we even gotten from the DARPA days?</em></p><p><a href="http://comma.ai/">comma</a> has been a company that’s stood out to me because of the amount of progress that they’ve made with some of the most limited resources compared to the overall industry. <strong>I’ve spent the past little bit understanding what the fascination with end2end learning + solving self-driving with cameras is and why this will be the way we solve self-driving.</strong></p><p>My intention with this blog is to dive into the software of <a href="http://comma.ai/">comma.ai</a> through this year’s COMMA_CON talks (end2end learning) since I am a <a href="http://srianumakonda.com/">computer vision geek</a>, and to figure out where comma really is today + <strong>how far they are from “solving” self-driving</strong>. I’m going to focus on the two main computer vision talks focusing on the primary ML stack (Harald’s and Yassine’s) with my commentary and then provide <strong>my own thoughts with respect to how far comma is from solving the “self-driving problem.”</strong></p><h3>The openpilot tldr;</h3><p>openpilot is an <a href="https://www.comma.ai/openpilot">advanced driver assistance system</a> (ADAS), which essentially is a level 2 driving system (out of <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwik06mXpKaBAxUCmGoFHcByArcQFnoECCkQAQ&amp;url=https%3A%2F%2Fwww.carmagazine.co.uk%2Fcar-news%2Ftech%2Fautonomous-car-levels-different-driverless-technology-levels-explained%2F&amp;usg=AOvVaw3iBRfmPZ1pLETSMY4fBY_4&amp;opi=89978449">the possible 5</a>). It’s essentially the software that makes up the comma3 (and the 3x), which turns your car into a <strong>self-driving</strong> car.</p><p>But if you wanted to actually control the car, how would that work? You can use a CAN BUS (Controller Area Network). The CAN BUS has two parts to it: an identifier and a data portion, which helps provide all the data you need to collect data (lateral + longitudinal data) + be able to hack into your car and make it drive on its own!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Vd-GrvB7mYb-5Yax8W_Ag.png" /><figcaption>an example of a message sent in the bus! <a href="https://www.youtube.com/watch?v=nNU6ipme878">source</a>.</figcaption></figure><p>The CAN-TO-USB interface (a.k.a, the panda) is what allows openpilot to send all the driving information into the car (which connects to the car via the OBD-II PORT).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*extCmZ4oTmJMAait8N8a3Q.png" /><figcaption><a href="https://www.youtube.com/watch?v=nNU6ipme878">source</a>.</figcaption></figure><p>The raw car data is then processed with the <a href="https://github.com/commaai/openpilot/tree/master/tools/cabana#readme">comma cabana</a> in order to figure out what parts of the raw data are relevant to driving + which ones are not. From there, it’s simply a matter of figuring out what are the data pieces that need to be sent to the car + sending the control messages. <a href="https://github.com/commaai/panda">panda</a> ensures that the data being sent in the car is safe (and won’t make any accidental sharp turns/acceleration).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SeSG0XjQsOGq0WfhZULQwQ.png" /><figcaption>the higher level openpilot architecture. <a href="https://www.youtube.com/watch?v=nNU6ipme878">source</a>.</figcaption></figure><p>For those curious to learn more about the technical specifics behind porting a car, <a href="https://www.youtube.com/watch?v=KcfzEHB6ms4&amp;list=PLzFUMGbVxlQutNi3HJsEn7lwrdb-kymMV&amp;index=27">Jason’s talk</a> at COMMA_CON does a great job explaining it.</p><h3>The meat of driving: end2end learning</h3><p>Great! Now you know about how comma works from an overarching level, but how does it learn to drive? The answer is through end2end learning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*I5rFxHHZubDcWKnmIVHjTg.png" /><figcaption>old blog but still relevant! <a href="https://blog.comma.ai/end-to-end-lateral-planning/">source</a>.</figcaption></figure><p>The main idea behind end2end learning is that you can <strong>take in a raw image</strong> of the driving scene and be able to <strong>achieve lateral + longitudinal driving with the use of Convolutional Neural Networks</strong>. The question end2end learning essentially tries to answer is, <strong>“what if i didn’t create human intermediaries (semantic segmentation, detections, etc.) and allowed a model to learn driving on its own?”</strong></p><p>Why would we want to do that? Because of the fact that <strong>creating intermediaries restricts the model’s ability to learn mappings of its own. If you want to create a self-driving car that’s robust to any scenario, you need to allow it to learn its own mappings + patterns </strong>(that humans might not have even thought of!).</p><p>You can get results so good that you don’t even need to tell your model to detect traffic lights (it just knows!):</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSUIZYzxtMQs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSUIZYzxtMQs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSUIZYzxtMQs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1450c896d7c9b3f9f29e16e001059ac2/href">https://medium.com/media/1450c896d7c9b3f9f29e16e001059ac2/href</a></iframe><p>One of the questions that popped into my mind was, “okay, great. we have a car that’s able to drive given an input image. How do we actually use that information to go from point A → point B?”</p><p>The model knows where to drive by leveraging the built-in navigation systems via <a href="https://www.mapbox.com/">MapBox GL</a>. It’s then processed in <a href="https://www.youtube.com/watch?v=ZEMPQjK1ZPw&amp;list=PLzFUMGbVxlQutNi3HJsEn7lwrdb-kymMV&amp;index=10">60-second videos</a> to train this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHrPv_zhOo7JIGKlctMIOA.png" /><figcaption><a href="https://www.youtube.com/watch?v=ZEMPQjK1ZPw&amp;list=PLzFUMGbVxlQutNi3HJsEn7lwrdb-kymMV&amp;index=10">source</a>.</figcaption></figure><p>To render the maps, you need latitude + longitude for each map (done with the comma e2e model)<strong> and the route </strong>(the trickier part — why? because <strong>GPS is noisy, </strong>which they solve with <a href="https://github.com/valhalla/valhalla">Valhalla</a>).</p><p>An autoencoder is then trained to ensure that the bottleneck layer (the feature vector) can be used as a way to compress the map data (since 256x256 is A LOT of pixels). Boom! You now have a car that can drive from Point A → Point B:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sRH3tlU5KO2pD4NYoM7BSQ.png" /><figcaption>orange = predicted path, blue = ground truth. tested in simulation (which we’ll go over soon!). <a href="https://www.youtube.com/watch?v=ZEMPQjK1ZPw&amp;list=PLzFUMGbVxlQutNi3HJsEn7lwrdb-kymMV&amp;index=10">source</a>.</figcaption></figure><p>comma also focuses on driving monitoring + ensuring that the driver is paying attention at all times. This is done through the use of more end2end models. You can also use this information to <a href="https://youtu.be/3-pCMvkOAms?si=5UgE5LsaHnyb7GtZ&amp;t=1105">predict uncertainty</a> + figure out at what points in time the model is not confident in its control of the car (which can help drivers better understand when to disengage from openpilot and take over).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVR9MsnOw7oE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVR9MsnOw7oE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVR9MsnOw7oE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad850b09e1d2c0a8bb23fb9d2f314ccb/href">https://medium.com/media/ad850b09e1d2c0a8bb23fb9d2f314ccb/href</a></iframe><p><strong>p.s.</strong> if you’re curious about what car customization looks like + ensuring that openpilot works on your car, look no further than <a href="https://www.youtube.com/watch?v=sU2mLUn_pi8&amp;list=PLzFUMGbVxlQutNi3HJsEn7lwrdb-kymMV&amp;index=22">vivek’s talk</a>!</p><h3>Driving to Taco Bell with end-to-end ML</h3><p>What else would be a better way to start COMMA_CON by listening to Harald and the end2end research that he’s doing?</p><p>The <strong>constraints </strong>that they’re using to deploy self-driving systems:</p><ol><li><strong>fully end2end models.</strong> why? because it “gives more flexibility to the ML models to learn behaviors and patterns itself.” The key idea with end2end learning is that there is no explicit human control as to what underlying semantic information you want the car to see (ex. lane lines, pedestrians, semantic segmentation, etc.). Instead, end2end models implicitly allow the neural networks to learn all this spatial information without being “constrained” (perception layer).</li><li><strong>scaling with compute.</strong> this one i believe is one of the more important factors when it comes to ML deployment. it’s important that you can a) create a policy that has good performance when it comes to driving and b) it’s easy to scale across multiple mediums (cities) with minimal difficulties</li><li><strong>good loss functions</strong> to ensure that you have a model that understands what to optimize for + ensure that loss function “reflects good driving.”</li><li><strong>diversity of data</strong>, which is something that you get from users and their driving logs. one thing that i would add here though is that <strong>there is a difference between gathering data vs. gathering HIGH-QUALITY data. </strong>If you want to create a “superhuman driving agent” then you need to either at least a) create/capture data that your car has never seen before that it will most likely see in the real world or b) create a policy so good that it doesn’t need any prior information on how to deal with out-of-distribution (OOD) events.</li></ol><p>He breaks <a href="http://comma.ai/">comma.ai</a>’s end2end approach into <strong>3 main pipelines:</strong></p><ol><li><strong>end2end lateral</strong> (steering/yaw rates) control</li><li><strong>end2end longitudinal </strong>(position/speed/acceleration) control</li><li><strong>end2end navigation</strong> (so that the car knows where it needs to go!)</li></ol><p>how do they prove this? by showing a driving log to tacobell 🌮</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSUIZYzxtMQs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSUIZYzxtMQs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSUIZYzxtMQs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1450c896d7c9b3f9f29e16e001059ac2/href">https://medium.com/media/1450c896d7c9b3f9f29e16e001059ac2/href</a></iframe><p>One of the initial problems Harald brings up with the warped simulation that comma’s been using is that the <strong>cars + objects were warped sideways.</strong> how do they solve this? <strong>use a segnet to make depth assumptions + basic objects!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ON_pfxDYUWrMAy0wTDrU7Q.png" /><figcaption><a href="https://www.youtube.com/watch?v=zu5OFv3pzRU&amp;t=4s">source</a>.</figcaption></figure><p>Using sem-seg information, we can <strong>create a basic depth map</strong> and be able to pair that information to create a better simulation.</p><p><a href="http://comma.ai/">comma.ai</a> wrote a <a href="https://blog.comma.ai/end-to-end-lateral-planning/">blog</a> a while back on their end2end lateral control model. 2 years later, the<strong> ideas have stayed the same but there’s been major updates</strong> (such as getting rid of the “student-teacher approaches” + the multiple architectures used) <strong>to ensure max efficiency and performance.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*skQGvuVyqHIJXjz9ysQbxQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=zu5OFv3pzRU&amp;t=4s">source</a>.</figcaption></figure><p>A fascinating idea that they rejected this year is the <strong>usage of stoplines to provide information </strong>to the e2e model as to <strong>when + where to stop</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HWe0e4OZltJzqzaipDPD0A.png" /><figcaption><a href="https://www.youtube.com/watch?v=zu5OFv3pzRU&amp;t=4s">source</a>.</figcaption></figure><p>Because of the <strong>sheer amount of variation</strong> that comes from user data (ex. sudden breaking/breaking too early), <strong>it’s hard to create these models that are 100% dependent on your data</strong>. I think, potentially, there is room for an algorithmic implementation of these types of stop lines (ex. gather scene information such as traffic lines + stop signs and create a model using those parameters + the image to create the stopline), but I definitely agree with the fact that leveraging users’ data for this task was not going to give good results.</p><p>Idea #2 that they rejected was the use of Depthnets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9Jpl6820ashxquGXIQGsQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=zu5OFv3pzRU&amp;t=4s">source</a>.</figcaption></figure><p>Depthnets are <strong>noisy + hard to predict + has lots of failure points </strong>(shadows, traffic lights, etc.). Performing these types of tasks without multiple camera systems is extremely difficult as you’re dependent on a singular camera to be able to solve this problem.</p><p><strong>What they’re currently working on:</strong></p><ol><li><strong>switch from physics-based-simulation to ML/end2end based</strong> (have a policy learn full simulation and get rid of any “human”/classical intervention)</li><li>using <strong>Reinforcement Learning (RL) based control methods</strong>. <a href="http://comma.ai/">comma.ai</a>’s <a href="https://blog.comma.ai/end-to-end-lateral-planning/">2021 blog</a> mentions that they do use a Model Predictive Control (MPC) solver to ensure smooth execution of driving. instead of following an algorithmic approach, use RL as a way to improve the model with respect to data + time.</li></ol><h3>Learning a Driving Simulator</h3><p>One of the first simulators built back in 2016 was a “<a href="https://arxiv.org/pdf/1608.01230.pdf">small offset simulator.</a>” The idea here is essentially to<strong> “offset” the camera slightly to the left + right to see if the end2end policy is able to recover from the shifts.</strong></p><p>Yassine breaks down the ML Simulator architecture into 3 main components: <strong>the pose tokenizer, the image tokenizer, and the dynamics transformer.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nTTzR26lZy_iD78b1EWzfg.png" /><figcaption><a href="https://www.youtube.com/watch?v=hpRzNxQvZDI">source</a>.</figcaption></figure><p>Let’s dive into this together:</p><h4><strong>Image tokenizer</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4SIbl2qHZs7PHzh3ky7nOQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=hpRzNxQvZDI">source</a>.</figcaption></figure><p>The idea is straightforward:<strong> take the image, compress it</strong> into a tokenizer (encoded vector). Using that, we can then <strong>upsample with the decoder</strong> and validate the “realness” of the image via the discriminator.</p><p>The tokenizer essentially follows a VQ-GAN structure. It’s similar to a GAN (read more <a href="https://srianumakonda.medium.com/datagan-leveraging-synthetic-data-for-self-driving-vehicles-6e629968a567">here</a>) . I won’t dive deep into the VQ-GAN architecture and its inner workings since this <a href="https://ljvmiranda921.github.io/notebook/2021/08/08/clip-vqgan/">blog</a> perfectly explains it.</p><p>Their results?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vLy-gRNxe4iUTexHHxsEgQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=hpRzNxQvZDI">source</a>.</figcaption></figure><p>As you can tell, the<strong> bit resolution is directly correlated with the quality</strong> of the VQ-GAN reconstruction (the lower the bits, the worse it gets).</p><h4>Pose tokenizer</h4><p>The main idea behind a pose tokenizer is to <strong>take in the pose data</strong> (x, y, z, yaw, pitch, roll) and be able to <strong>tokenize that</strong> data in such a way that you can create a set of tokens (continuous data → discrete).</p><p>The comma team does it in a quite straightforward manner — they digitalize this information with uniform binning of the continuous data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQPA37EYhQ30G2eRyQiJyQ.png" /><figcaption><a href="https://www.youtube.com/watch?v=hpRzNxQvZDI">source</a>.</figcaption></figure><h4>Dynamics Transformer</h4><p>They then use a <strong>transformer to predict the next token</strong> (of the information that we were encoding above) which is then <strong>rolled for the next <em>n </em>amount of frames</strong> (which is known as <strong>autoregressive sampling</strong>). These are the results Yassine shows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dzDsujK2iteyBOc68MZJOA.png" /><figcaption><a href="https://youtu.be/hpRzNxQvZDI?si=abspif3pscqiDD0I&amp;t=845">demo</a>. the coolest thing? everything is <a href="https://github.com/commaai/commavq">open-sourced</a>! <a href="https://www.youtube.com/watch?v=hpRzNxQvZDI">source</a>.</figcaption></figure><p>One of the initial thoughts that I had when Yassine showed how the dynamics transformer is the fact that <strong>spatiotemporal information can be leveraged here</strong> in order to create a better, higher-quality model (and move towards video data v.s. feeding in outputs → inputs for a certain amount of time). What he brings up, though, is this interesting idea where you essentially <strong>have a recurrent layer placed in the decoder</strong> (with everything else frozen) where the decoder can <strong>learn the disparities from multiple input images and be able to improve the quality of the simulated prediction</strong>. Here’s the <a href="https://youtu.be/hpRzNxQvZDI?si=CPT3sr-hJSTXk3C3&amp;t=1118">demo</a> he shows in his talk (it’s a really interesting idea!).</p><p>With respect to moving, my assumption (and Yassine confirms this in the Q&amp;A) is that they most likely are able to <strong>control what type of simulated output they want</strong> (ex. brake, acceleration, moving right/left, etc.) by <strong>controlling the pose-estimation tokens</strong> (in such a way that your values make the dynamics tokenizer to predict the next frame as going in <em>x </em>direction).</p><p>The biggest room for potential would potentially be <strong>creating some optimized pose tokens</strong> where you can essentially <strong>figure out where your model lacks</strong> (ex. could be with lane changing to the right) and gear simulation towards there. Then you can a) <strong>finetune your simulation</strong> and automate the pose tokenizer process but also b) <strong>create high-quality data</strong> that can dramatically improve model performance.</p><p>I think the simulation stack here is quite interesting. The majority of the problem will most likely be centered around ensuring that a) <strong>you have a proper dynamics transformer architecture</strong> where you can create videos (and leverage temporal information) while also ensuring that b) y<strong>ou have high quality simulation.</strong></p><h3>What does it take to solve self-driving?</h3><p>George says that self-driving is already <a href="https://youtu.be/dNrTrx42DGQ?si=hf4oMiGZTFHLMBk4&amp;t=4127">solved</a> and that we are a couple “bugs” away from solving the overall driving problem. Is that true?</p><p>The way George defines the self-driving problem is: “<strong>how do you build a model that outputs a human policy for driving?”</strong></p><p>He brings up a really good point on Lex Fridman’s podcast about how, fundamentally, you can<strong> create a basic policy that does the job</strong> (drives like a human). From there, it’s just a matter of <strong>fixing the “bugs” + scaling</strong>; scaling in compute + scaling in data. I’ll focus more on the ladder since the former is quite straightforward to understand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*o9sfUSuQQQ-sGfktgJg5Fg.png" /><figcaption>Udacity’s Self-Driving Car Dataset and their corresponding steering ground truth’s between [-1, 1] radians. <a href="https://www.researchgate.net/publication/333323920/figure/fig2/AS:761790254301185@1558636358715/The-angle-distribution-within-the-entire-Udacity-dataset-angle-in-radians-vs-total.png">source</a>.</figcaption></figure><p>The <strong>majority of data</strong> that you get will likely be <strong>straightforward, basic driving</strong> (i.e. staying in your lane + following basic traffic rules). The fundamental definition of a generalizable self-driving agent is to be able to <strong>navigate under all sorts of conditions and react to every scenario you face. </strong>When the majority of the data you collect isn’t diverse (ex. the steering values of the Udacity dataset above), then your model will not be able to react accurately in situations. <strong>There is no second chance when it comes to self-driving cars: a single mistake is life or death, and failure to prepare for every scenario is a recipe for disaster.</strong></p><p>My two cents when it comes to creating generalizable self-driving is that you need to have the right combination of the following things in order to create a framework to “solve self-driving”:</p><ul><li>compute (which George talks about)</li><li>a strong enough self-driving policy (which is bounded by the quality of data + model architecture. my contrarian take is that <strong>we already have good enough models but not good enough data</strong>)</li><li>going from simulation (which what comma is building) to <strong>high-quality simulation </strong>where you can improve your model + sample high-quality data that can exponentially improve performance</li><li>interpretability of these models (from both a regulatory perspective + potential avenues in debugging + training these models)</li></ul><p>Let’s dive a little bit deeper into the ladder 2 buckets:</p><h4>What ideal simulation would look like</h4><p>Simulation is what enables self-driving to scale. It allows you to not only scale faster but also be able to improve model performance if used correctly. Some of the characteristics that a high-quality, robust simulation should have include the following:</p><ol><li>Being able to <strong>create simulations of all sorts in video</strong> to allow for training + deployment across multiple environments (snow, rain, fog, etc.)</li><li>Ability to <strong>create challenging scenarios that the existing model might not have exposure to</strong> in certain geographies (ex. snow in Texas). The second part to this would be to use this data and improve the existing model.</li><li>Building on from the existing point, be able to s<strong>imulate out-of-distribution/extreme edge case scenarios</strong> where there is high likelihood of failure and ensure that your model can deal with these types of situations.</li></ol><p>I think Wayve’s research with <a href="https://wayve.ai/thinking/introducing-gaia1/">GAIA-1</a> is a really good example of what a <strong>powerful simulation with pure computer vision looks like</strong>. Leveraging this in such a way that allows you to be able to improve + validate existing models can result in really big breakthroughs in the end2end learning space.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5Jx2QgEUZUI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5Jx2QgEUZUI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5Jx2QgEUZUI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e983c07b7519c60d99ad9cce4dfe9d4/href">https://medium.com/media/1e983c07b7519c60d99ad9cce4dfe9d4/href</a></iframe><h4>Interpretable end2end learning</h4><p>I expand a little more about this in my talk @<a href="https://youtu.be/8lLRZzFft5Y?si=m-6uJgd3C7dnjoLd&amp;t=1416">the Austin Computer Vision Meetup</a>, but the rundown essentially is that understanding the <strong>how + why your model makes the decisions it does can play a really big role in</strong>:</p><ol><li>Debugging failure cases/crashes and understanding why the model made the decisions that it did</li><li>Leveraging interpretability in simulation + improving training</li><li>Potentially finding outliers in data that have a big impact on model performance</li></ol><p>From a regulatory standpoint, it’s important [to some degree] that you have some ability to <strong>peek into this black box </strong>and <strong>understand why your model made the decisions it did</strong>. But I do think that there is a good amount of information that can be learned through the ability to peek into this black box.</p><p><strong>Understanding potential failure cases of your model allows you to iterate and leverage simulation to improve these models.</strong> For example, if my comma 3 struggled in snow, then understanding what the model can see via <a href="https://arxiv.org/pdf/2301.07002.pdf">saliency maps</a> (as an example) allows us to understand if there are possibly any characteristics of snow that it struggles with. Using that information, we can then restimulate these environments to improve these models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*We-Ho7tbgzUGw_Jex_et8Q.png" /><figcaption><a href="https://people.eecs.berkeley.edu/~jfc/papers/17/iccv-final-interpretable_learning_for_self_driving_cars.pdf">source</a>.</figcaption></figure><h3>So, have we solved self-driving?</h3><p>The short answer is <strong>yes and no.</strong></p><p>The way we get to solve self-driving cars is clear (and has been clear for the past couple of years), but <strong>it will take time until the problem truly gets solved</strong>. One of the questions I like to ask myself is, <strong>“what does it even mean to solve self-driving?”</strong> The world, in nature, is not stochastic, which means that there will always be situations encountered that your model has not been exposed to in its dataset.</p><p>How do you create a model that can adapt + deal with any situation that it’s put in? Will simulation allow you to skip the “real-world testing” and be able to deploy these models straight from simulation? <strong>How powerful of a simulator do we need in order to solve self-driving cars?</strong> To what degree do we need to worry about understanding these models? <strong>Does it even matter if we can understand these models as long as the model can</strong> drive with good performance? How good of a model is needed? <strong>Are we trying to replicate human driving</strong> (and, to be fair, humans are excellent drivers), <strong>or are we trying to create a self-driving car where we want superhuman performance?</strong> How do you even regulate self-driving cars?</p><p>These are some of the questions that we should be thinking about when it comes to creating self-driving cars. For me, <a href="http://comma.ai/">comma.ai</a> is one of the most inspiring companies right now in the self-driving space because of how fascinating the research that they’re doing is + how much progress they’ve made in the space with relatively limited resources [compared to most companies]. I really do believe that the “comma approach” has the potential to really disrupt the self-driving/robotics industry as a whole and that this will be the way we <strong>solve </strong>self-driving cars.</p><h3>Sri Anumakonda on Twitter: &quot;rewatching @realGeorgeHotz&#39;s talk at @comma_ai COMMA_CON and every single time i come across this stat it continues to blow my mind...wow. pic.twitter.com/tg9yRgd39M / Twitter&quot;</h3><p>rewatching @realGeorgeHotz&#39;s talk at @comma_ai COMMA_CON and every single time i come across this stat it continues to blow my mind...wow. pic.twitter.com/tg9yRgd39M</p><p><em>Thanks for reading this article! My goal with this article is to teach more people about how cool self-driving is through one of the self-driving companies that I truly believe will solve this problem + push the frontiers of humanity. It would mean a lot if you shared this resource wherever you can! I spent a lot of time curating notes + creating a good understanding for people to learn RL from, and my goal is to help as many people as I can!</em></p><p><em>To learn more about me and the research that I’m doing in self-driving cars, here’s my </em><a href="https://twitter.com/srianumakonda"><strong><em>twitter</em></strong></a><strong><em> + </em></strong><a href="https://www.linkedin.com/in/srianumakonda/"><strong><em>linkedin</em></strong></a><strong><em> + </em></strong><a href="http://srianumakonda.com/"><strong><em>website</em></strong></a><strong><em>. </em></strong><em>My research primarily focuses on leveraging computer vision to train better autonomous vehicle policies while ensuring that we create self-driving systems that are able to scale. Right now, I’m currently doing research at UIUC where I’m looking into using generative modelling techniques to create better simulation + training for self-driving cars.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2140aabc6f52" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything you need to know about Reinforcement Learning in <80minutes]]></title>
            <link>https://srianumakonda.medium.com/everything-you-need-to-know-about-reinforcement-learning-in-80minutes-4cd5a365e340?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cd5a365e340</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Tue, 25 Jan 2022 15:49:06 GMT</pubDate>
            <atom:updated>2022-01-25T15:54:37.338Z</atom:updated>
            <content:encoded><![CDATA[<h3>Everything you need to know about Reinforcement Learning in 80 minutes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U1dnPMkRtv9y5Pk4.png" /></figure><h3>A quick note!</h3><blockquote>Hey! This article will contain a lot of my notes that I’ve been taking from David Silver’s Reinforcement Learning Course with some of my notes. This article can be used as a self-study guide or as a read-a-long with his course! This article assumes that you have a good understanding of high school math + basic differentiation. You can find the original lectures <a href="https://www.youtube.com/playlist?list=PLqYmG7hTraZBiG_XpjnPrSNw-1XQaM_gB">here</a>. To learn more about me, here’s my <a href="https://twitter.com/srianumakonda"><strong>twitter</strong></a><strong> + </strong><a href="https://www.linkedin.com/in/srianumakonda/"><strong>linkedin</strong></a><strong> + </strong><a href="http://srianumakonda.com/"><strong>website</strong></a><strong>.</strong></blockquote><blockquote>It would mean a lot if you could help share this resource wherever you can! I spent a lot of time curating notes + creating a good understanding for people to learn RL from, and my goal is to help as many people as I can!</blockquote><p>Reinforcement Learning sits at the intersection of many fields with the same fundamental idea: <strong>the science of decision making.</strong> In Computer Science, it’s machine learning but in Neuroscience, it’s reward systems. Engineering, optimal control.</p><p>Why is RL different from other machine learning paradigms + fields?</p><ul><li>There aren’t supervisors, only rewards</li><li>Feedback in RL is generally delayed, not spontaneous</li><li>Time plays a big role for performing in dynamic systems such as RL</li><li>The agent’s actions affect the subsequent data it receives (closed feedback loop)</li></ul><p>General applications right now?</p><ul><li>Fly stunt maneuvers in a helicopter</li><li>Defeating world champion in games i.e. GO, Atari</li><li>Investment portfolios</li><li>Walking robots</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/0*TUJ6DYBVShHa2hDl.gif" /></figure><h4><strong>Defining the Reinforcement Learning problem</strong></h4><p>A reward Rₜ is this general scalar feedback signal to help the agent understand how well it’s doing at step t. Generally, the job of the agent is to <strong>maximize the cumulative reward</strong>. RL is based on this “reward hypothesis”; all goals can be described through the maximization of expected cumulative reward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*_56p7qqAEK7A8DF1" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><p>What’s the goal here? <strong>select certain actions that maximize the total future reward.</strong> Sometimes, it might be better for you to endure a short-term loss for long-term gains in an environment (ex. chess.)</p><h4><strong>Understanding the agent + environment</strong></h4><p>The key principles of RL are based on this interaction of agent and environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/0*eU4-M3HQ5VkB_Jcr" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><p>At each step t, the agent:</p><ul><li>executes action Aₜ</li><li>receives observation Oₜ</li><li>receives scalar reward Rₜ</li></ul><p>the environment:</p><ul><li>receives action Aₜ</li><li>emits observation Oₜ</li><li>emits scalar reward Rₜ</li></ul><h4><strong>History + state</strong></h4><p>The history is generally the sequence of observations, actions, and rewards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*TQ0yBbLHGukJVYvp" /></figure><p>We want to measure all observable variables up until time t. Generally, what happens next depends on that history.</p><ol><li>The agent selections actions</li><li>The environment selects observations/rewards</li></ol><p>The state is the information that we can use to determine what happens next. It’s formally defined as a <strong>function of history</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*VDUK_HsbJW5zoM0O" /></figure><p>For example, the environment state, Sᵉₜ is the environment’s private representation. This is exactly whatever data the environment generally uses to pick the next observation/reward to give to the agent [note: the environment state is not visible to the agent. Even if it is, it may contain irrelevant info.]</p><p>The agent state Sᵃₜ is the agent’s internal representation of those observations. The state generally contains whatever information the agent needs to use to pick the next action. It can be any function of history.</p><h4><strong>Markov state</strong></h4><p>Also known as the information state. Think of it like an array that contains all the useful information through history.</p><p>A state Sₜ is Markov if <strong>and only if…</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*y3DCnYF7kYY4Ww4i" /></figure><p>The probability of the next state [given the state that you are currently in] will be the same even if you showed all the previous states of the system.</p><p>What does this mean? You can throw all the previous states away and just use the current one and you’ll still get the same probability. <strong>The future is independent of the past given the present.</strong></p><p>Therefore, the environment state Sᵉₜ is Markov because it fully characterizes what’ll happen next in the environment because that’s exactly what the environment is using to predict its next environment. The same logic also applies with the history Hₜ.</p><h4><strong>Fully observable environments</strong></h4><p>This is exactly when the agent can directly see + observe the environment state (this is generally ideal). The agent state = environment state = information state. Oₜ = Sᵃₜ = Sᵉₜ. This is formally defined as the <strong>Markov Decision Process (MDP).</strong></p><h4><strong>Partially observable environments</strong></h4><p>The agent indirectly observes the environment. Think of it like a robot with camera vision; it isn’t told where it actually is [location].</p><p>Agent state ≠environment state. This is formally defined as the <strong>partially observable Markov Decision process (POMDP).</strong></p><p>In this environment, the agent has to construct its own state representation Sᵃₜ whether this might be…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*XeA5vvQgVk1nB6DV" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><p>A proper RL agent generally includes ≥1 of these components:</p><ul><li>Policy: agent’s behaviour function</li><li>Value function: how good each state/action is</li><li>Model: how the agent represents the environment</li></ul><h4><strong>Policy</strong></h4><p>The policy is exactly what the agent’s behaviour would be that focuses on mapping from <strong>state → action.</strong> In a deterministic policy, we generally write that down as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/0*cL7O4VN6hR1dQcYb" /><figcaption>Deterministc policy</figcaption></figure><p>or</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/0*x4873LycniEG1AGT" /><figcaption>Stochastic policy</figcaption></figure><h4><strong>Value function</strong></h4><p>The value function is generally the <strong>prediction of future rewards</strong>. We can use this to evaluate the “goodness/badness” of states which we then can use to select between actions.</p><p>Think of it as money. Let’s say that the more money you have, the happier you become. The value function figures out what stage you’re at in becoming happy. If you don’t have enough money, then you’ll be unhappy.</p><p>That’s exactly the same with reward. The more reward we have, the happier we become. Our goal is to get as much reward as we can, which we can model through this equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/0*2QFpwyO3YsTgdFVU" /></figure><p>Where γ is something known as the <strong>discount factor.</strong> Our reward has a timescale to how far ahead we look into the future. The way γ works is that it’s a value between 0 and 1 i.e. γ ∈ [0, 1]. Once that value is selected, we exponentiate γ more and more until it essentially reaches 0. <strong>That’s exactly the point in time</strong> [at our current state] <strong>until we don’t care about it.</strong></p><h4><strong>Model</strong></h4><p>The model is what predicts what the environment will do next. Why this is important? Modelling the environment and what the future might look like might allow us to make a plan to help you figure out what you want to do next.</p><p>This is broken down into 2 main parts:</p><ol><li>Transitions: this is exactly what predicts the next state in the environment ex. if i predicted that i should steer 5 degrees to the right, what will the camera input look like for the next frame?</li><li>Rewards: now given that transition, what would our next [immediate] reward look like? What would be the reward be if I steered 5 degrees to the right?</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/0*wisMbx4I78peZOsG" /></figure><p>We predict the next state and reward in the environment by using information from the current state + action. note: P is our transition model and R is the reward prediction.</p><h4><strong>Categorizing RL agents</strong></h4><p>We can mainly break it down in this hierarchy system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*EYM3kfzQgeVONMDA" /></figure><p>The main difference between policy-based and value-based is that policy-based methods are all about creating that mapping i.e. a= π(s) and use that during learning. You’re building the ideal policy which changes over time given more info to optimize. In the value-based approach though, you just store a value function. The policy itself is derived from the value function i.e. pick the action with the best value.</p><p>Let’s say that this was our maze:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/0*6aV82e93g6hKpFKB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><p>Value-based approaches would simply tell you the value for a given action. The RL agent is <strong>greedy. It wants to increase its value as fast as it can so it’ll just follow the numbers.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*spfFw-wGvJ4bcY9J" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><p>But with policy, instead of representing numbers based on each state, we can explicitly represent the direction/map that the agent should follow and just simply work in that space. Notice that the value function never explicitly stores the value function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/0*oA1r2ZnBKf4yDibZ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><p>The actor-critic algorithm simply stores the information of these 2 (policy + value functions) methods together.</p><p>We also have something unique known as <strong>model-free agents.</strong> Model-free means that we not try to explicitly understand the environment around us. Instead, we directly try to implement a policy function. We just get experience and figure out the policy of how to behave to figure out the best way to get a reward. We don’t overcomplicate the problem as it is.</p><p>Alternatively, we have <strong>model-based agents.</strong> For this, we need to build a model of how the environment works. And then from there do we decide where we want to go ex. model-based would model a helicopter and its environment in the air and then decide what direction should i be moving in.</p><p>We can group this all in a venn diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*68omXFRbxAJNVe86" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/intro_RL.pdf">Source</a></figcaption></figure><h4><strong>Problems within Reinforcement Learning</strong></h4><p>There are generally 2 fundamental problems in sequential decision making that we try to solve:</p><ol><li>the environment is initially unknown</li><li>the agent interacts with the environment</li><li>the agent improves this policy</li></ol><p>We define this to be the <strong>Reinforcement Learning problem. t</strong>his is all about trial-and-error making. I’m sure you’ve heard of the walking analogy; a baby tries to walk on its two feet. But it falls at first. Over time, it then starts to better understand how to walk and eventually, just walks!</p><p>The other problem we like to define is known as the <strong>planning problem.</strong> This is completely different:</p><ol><li>the model of the environment is fully known (ex. we know the weather outside, wind blowing against the self-driving car)</li><li>agent performs computations with this model</li><li>agent then improves this policy</li></ol><p>Notice the difference here. The latter is given “privileged” information about the environment while the former does not. If I was playing Forza, I would be given information such as how to drive if I was using a planning approach. I would not if I were a normal RL agent without any info.</p><p>RL is generally a trial-and-error problem. The goal is to discover a good policy from its experiences of the environment without losing too much reward. But we need to find this balance between <strong>exploration to see if there’s more reward or exploration and using already-known information.</strong></p><p>What does this look like? If I wanted to eat dinner, would I try to <strong>explore</strong> and find new restaurants that I might love or <strong>exploit</strong> and stick with my current favourite restaurant?</p><h4><strong>Prediction and control</strong></h4><p>Prediction is all about evaluating the future when you’re given a policy while <strong>control is all about optimizing the future</strong> while trying to find the policy that does so.</p><p>For example, prediction is asking “if i continued driving straight, how much reward would i get?” vs if i was talking in control terms, “which direction should i walk to then figure out the best reward i can get?”</p><h3><strong>Markov Decision Process</strong></h3><p>This is what we use to formally describe an environment in RL where <strong>we can see everything in the environment</strong> [is fully observable]. All the information that there is presented to our agent. What this means is that the current state of the agent completely characterizes the process in the way in which the environment unfolds depending on that state which we already know [because the environment is fully observed].</p><p>Generally, almost ALL RL problems can be <strong>formalized as MDPs ex.</strong></p><ul><li>optimal control ex. steer a self-driving car usually deals with something known as continuous MDPs (regression)</li><li>partially observable problems can be converted into MDPs</li><li>bandits (expansion of exploration vs exploitation) are generally MDPs with one state</li></ul><p>recap: the markov property is when “the future is independent of the past given the present”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*5P7gHAJeLcpNm5dm" /></figure><p>where S is our current state. The state completely characterizes all the information that we know. Because all previous state information is irrelevant, we can discard it.</p><p>For any Markov property, if you start in some state s and you have some successive state s’, then it’s <strong>possible to find a probability of transitioning from the current state to the successive state.</strong></p><p>Remember that Sₜ has all the information in the current state. There could be a well-defined transition probability that I’m currently in a state that I already was in before (the past), there’s some probability that I will transition to some next state. ex. if an agent is at a yellow light, then there’s a set probability that i speed through it or slow down. All these outcomes are <strong>characterized in the state that it was in before</strong> (the state when you see that yellow light). This is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/0*j3F12NTlWODEBNra" /></figure><p>So, it’s the probability of transitioning from state s → s’ that gives is the probability of a random next state (speeding through the light) given the previous state that we were in (seeing that yellow light). We can put this in matrix form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/0*3-8p72Gb5eR0fjxB" /></figure><p><strong>where each row of the matrix sums to 1</strong> and n is the number of possible outcomes for each state that i was in (ex. one outcome is speeding through, another is slowing down the yellow light).</p><p>Formally, the Markov process is a sequence of states i.e. S₁, S₂, … with the Markov property where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U7W0PP2ZfM-YbDiF" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>This is an example of a “student markov chain”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/0*a-vTfuk7inMxGDx0" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>What’s even going on here? Let’s say that you’re in “Class 1” right now. There’s a 50% probability that you go on Facebook or you end up staying through the whole class and then go to class 2. If you go onto facebook though, there’s a 90% chance you end up staying on (because you’re now in “that cycle”) or there’s another 10% chance that you shift your focus back to “Class 1”.</p><p>Let’s say you make it to “Class 2”. At this point, there’s an 80% class that you make it to “Class 3” or just end up getting way too bored and fall asleep with a probability of 20%.</p><p>If you make it past “Class 2,” congrats! You’re now in “Class 3” where you have a 60% of passing or you end up going to the pub for the drink. If you end up going to the pub and have a few shots, there are probabilities that you’d go back to the first, second, or even third class because you forgot all the information.</p><p>But if you make it through “Class 3” with the 60% probability, congrats! You’ve now passed the class and will go to sleep.</p><p>This is an example of a Markov chain. Let’s look at a <strong>sample (a sequence of states)</strong> that take you from in → out. For example:</p><ul><li>C1 C2 C3 Pass Sleep</li><li>C1 FB FB C1 C2 Sleep</li><li>C1 C2 C3 Pub C2 C3 Pass Sleep</li></ul><p>Think of each jot note as a random sequence that samples these dynamics i.e. the dynamics of that Markov chain (image). The diagram is just one way to look at this chain.</p><p>Here’s what that would look like in “state transition matrix form”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*pHbqvMKkIqAytXcU" /></figure><h4><strong>Markov Reward Process</strong></h4><p>Mark reward processes are generally a normal Markov chain (process) with values. There’s some value judgement that tells you how much reward will you accumulate given a Markov sequence?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*rnteyuU6cwpg2xFZ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Remember that the reward is immediate. “If we start in some state S, how much reward would get from the exact state, exact moment?” What we care about is maximizing the accumulative sum in reward over time. We’re just showing this in one step (Rₛ).</p><p>Going back to our chain, let’s [hypothetically] create rewards for every action we take.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/0*a7-cWt3GsrpM9VjL" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>We don’t care about the immediate reward we get. We care about the total reward. We can define that by creating the return variable, Gₜ that gives us the total discounted reward from time-step t:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/0*hHU3t9Yka9yotqqq" /></figure><p>where γ ∈ [0, 1] is the present value of future rewards (recap: γ is a value randomly sampled between 0 and 1 which is the <strong>discount factor</strong> → what this means is that over time, because we’re exponentiation γ, the value of the rewards at k time steps in the future will ultimately equal 0, thus indicating that they are irrelevant).</p><p>Therefore, the value of receiving <strong>reward R after k+1 timesteps would be γᵏR.</strong></p><p>This concept of judgement via discount factors emphasizes that we care more about <strong>short-term rewards more than long-term rewards.</strong></p><p>Why do we do this discount factor? Because we do not have a perfect model. ex. if we were building a Markov process to model the environment and we don’t have a perfect model of the environment, <strong>we are uncertain about the future.</strong> That’s what short-term outcomes are more important to us. It’s also mathematically convenient and avoids infinite rewards (the point where <strong>γᵏR=0</strong> is when <strong>we don’t care about time step k</strong> anymore).</p><p>The value function v(s) is quite different in this context — <strong>it gives us the long-term value of the state s.</strong> If you’re in some state s, how much value will get from there on? What’s the total reward that you’d get from that state on.</p><p>If we go back to the chain diagram, how much reward will you get from ex. class 2 to the end?</p><p>Formally, the state value function v(s) of a Markov Reward Process is <strong>the expected return starting from state s.</strong> Formally, we write that down as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/0*kFp5Ik_z2oY-yyhC" /></figure><p>The reason why we take the expected return is that <strong>we’re in a stochastic environment.</strong> We don’t know about every single path that we can take and how that shapes the reward.</p><p>Let’s try sampling rewards from our Student Markov Chain (let’s say γ=1/2):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sEXhSiGp4ZVC36c8" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>What if γ=1?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*Wp5l6b69MnioW5Fe" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>This all falls under the <strong>Bellman equation.</strong> What this states is that the <strong>value function can be decomposed into 2 separate parts:</strong></p><ol><li>the immediate reward Rₜ₊₁</li><li>discounted value of successor state γv(Sₜ₊₁) → the value you get from the immediate reward onwards</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*L494Dud5L_31oqLS" /></figure><p>Essentially what happens is that v(s) is equal to the expectation of the total reward given the state Sₜ.</p><p>Then we simply expand on that equation and can simply factor γ out of the equation (note that Rₜ₊₁ has already been taken out before that factoring).</p><p>After factoring that expression under γ, you can notice that the same original equation we defined for v(s) above shows up again → but works when now we’re in time t+1.</p><p><strong>This is exactly why the equation makes sense; the value function of state s is equal to its immediate reward + its value function of the next state. That’s what the Bellman equation is.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/0*u0YavmF6Wbw6uIJt" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>In this example here, we’re using an undiscounted factor i.e. γ = 1. In the highlighted case of 4.3, what’s going on is that the moment we enter node [which we already have], we get a reward of -2 automatically. Then from there, we multiply the probability of going to the next state and the reward we’d get from that (which is 0.6<em>10 and 0.8</em>0.4). From there we sum it all up to get 4.3</p><p>This is how it’s normally written:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/0*eab3WZkepjzQaBqP" /></figure><p>We can also express the Bellman equation in matrix form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*EJff_y7rAoNdycvC" /></figure><p>Where v(1)…v(n) is the value function for all specific states where R₁ to Rₙ is our reward function given that state, the P matrix contains the transition probability of going from one state to another, and the v vector at the end is the values of where you end up.</p><p>Because the Bellman equation is a linear equation, we can solve for v together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*IKr-1_02A0I_OG86" /></figure><p>The computational running time of this solution is defined to be O(n³) given n states, so it isn’t really practical; it’ll only work well with small Markov Reward Processes.</p><h4><strong>Going back to MDPs…</strong></h4><p>This is exactly what’s most commonly used in reinforcement learning. We’re also adding one more piece of complexity to MRPs now <strong>which is actions</strong>. So far, we’ve only looked at calculating the reward and understanding the environment, but <strong>we haven’t told the agent what to do with any of this information</strong>. We can do this by introducing <strong>decisions.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/0*508_gi8QcOXJbUOW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>The change here is that the probability transition matrix <strong>depends on which action you take.</strong> The reward function may also change based on the action.</p><p>Let’s redo our Student chain and make it an MDP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/0*7C1NKY4rpe6_73Mm" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>The decisions are now <strong>labelled in red</strong>. Now you have <strong>more control over the path that you want to take.</strong></p><p>In order to do this, we need to formalize a way to make decisions. You can’t follow a map if you’re not given the route to follow.</p><p>We can do this by creating policies. Policies are generally some sort of distribution over actions given some sort of states:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/0*pK7RoCJ9s_2_OY52" /></figure><p>Given some state s, the distribution we get would give us some sort of mapping to figure out the probability of taking a certain action. The policy function is generally stochastic and the reason why it is so is because it allows us to do things such as exploration.</p><p>So to summarize, a policy fully defines the behaviour of an agent. And those policy depends on the current state that we’re in [and not the history because we’re in a Markov state]. The reason why the reward isn’t modelled here is because the <strong>state s fully characterizes your future rewards and is time-independent.</strong> Our goal is just to maximize the reward in the future, and we don’t need the reward to do so.</p><p>The connection between MDP and MRPs is that we can always recover an MRP from an MDP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*0PriSSBI_Fw9Elkv" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>If we have some policy that helps us take some actions, the sequence of states that we draw when we follow that particular process is a Markov process.</p><p>Once we fix the policy, if we look at the states and reward sequence, that sequence we get is a Markov Reward Process (point #3).</p><p>Note that P^π and R^π are the transition dynamics and the reward function which just simply average over our policy. We have the transition dynamics of all the possible things that I could do (ex. if P(walking left) = 0.5 and P(walking right) = 0.5, then I’m just multiply all the probability of the states that I’m going to end up on the left by 0.5 and the same for the right. <strong>Those average dynamics define some Markov Reward Process.</strong></p><p>We originally defined the value function to be that of the MRP but now, we need to also define it for the MDP. So, our state-value function v_π(s) of the MDP is the <strong>expected return starting from state s</strong>, and then following policy π:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/0*C22tRQjDuOv6tM1u" /></figure><p>Now what v_π(s) tells us is, <strong>“how good is it to be in state s if I’m following policy π?”</strong></p><p>We also need to define the second type of value function, known as the <strong>action-value function.</strong> We need to also figure out <strong>“how good would a particular action be for a given state [that would give us more reward]?”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/0*4BUHxiH_cy096ZMq" /></figure><p>Let’s go back to our student MDP…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*SkRuzE2O2fN6cGAC" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Now we can also define the Bellman expectation equation — the state-value function can be further decomposed into the immediate reward + the discounted value of the next state:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/0*B6zc_CJBS2_bvcQp" /></figure><p>The same also goes with the action function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RjQlK8XCL6FOn9Pg" /></figure><p>The way v_π and q_π relate to each other can be shown using the graph below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*PKaLhucD8kTHzQrj" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>So, for our state-value function here, <strong>we’re going to average over the actions that we might take</strong> (black boxes) shown by its probability. the q value tells us <strong>“how good is it to take that action from that step?”</strong></p><p>Now let’s try doing the opposite (inverse) of this — what would happen if we were given a single action value?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/0*1nRhizMu8oreMRaZ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>What’s going on here is that you’re in a particular state and you’re considering how good is it to go right from that state vs left? We now have to average the dynamics over our MDP; <strong>for each situation that might occur for going right, what’s the value of going right which also follows my policy?</strong></p><p>We can do this by averaging over everything by taking the probabilities of the transition dynamics (Pᵃₛₛ’) which gives us the action-value function at those roots here.</p><p>Remember that v tells us how good it is to be in a particular state while q is telling us how good is it to take a particular action given a particular state.</p><p>Let’s put together those 2 diagrams together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/0*Vdhni7zXU1xcNnze" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Given the value function for a particular state, <strong>we can figure out how good of a state we’re in by taking a 2-step (row) look in it.</strong> We’re considering <strong>all the actions we might take next</strong> (go left/right) and then <strong>all the actions the environment might do to us</strong> (ex. we might get blown in one direction by the wind). <strong>For each of those things the environment might do, we want to know how good is it to be in that state</strong> (bottom row of white circles) and carry on with my usual policy (reward).</p><p><strong>If we average everything all together</strong> i.e. we’re weighing each of the arcs of the second row by the policy we select (left or right), we’re also averaging each arc of the final row, all by the transition probability, Pᵃₛₛ’, we then finally figure out <strong>how good is it to be in that particular state (the top node).</strong></p><p>Just like with the state-value function, we can also do the same thing with the action-value function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*GVmwT9c9RrH-Ajs9" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>What is all of this saying? <strong>The value function of the current time step is equal to the immediate reward + the value function of where you end up. That’s what everything above is saying.</strong></p><p>Going back to our Student MDP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*oUPN0qyw8jKbzjSu" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>The state value function (7.4) is broken down into 2 things: studying or going to the pub. That means that each part has a probability of 0.5. From there we need to sum up the probability of going to the next state multiplied by its value.</p><p>What we really want to know is the <strong>best way to behave in MDPs</strong>. We need to find the best behaviour to follow in an MDP.</p><p>We don’t really care about how much reward we get following these 50/50 policies. What we care about is the best path that we can take in a system to find the optimal way to solve the problem. Let’s define the optimal-state value function v*(s) is the maximum value function over all the policies. We care about snatching the maximum possible reward and to do that, we need the best policy function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MukRt6LI9V6LpxRB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Once you figure these values out, you’re done. If you went left and got +70 reward but going right would give you +80 reward, where would you go? Obviously right. So, we know that the MDP is solved when we’ve figured out the bets possible performance in the MDP.</p><p>Well, what is the optimal value here?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/0*d75EyO0dO7vMWIzj" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Remember that we’re subtracting at every time step i.e. 6 -(-2) = 8, 8 -(-2) = 10. The red text is exactly what v*(s) is → <strong>how good it is to be in each state. But it doesn’t tell you how to behave in it.</strong></p><p>We can take the q*(s,a) values to figure out the optimal values for every arc:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*n7Q3ZZrDnJPY6LNU" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>We also need to figure out the optimal policy; we talked about the maximum reward we can get, not the policy itself. What does it mean for one policy to be better than another?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*T7Y1Ro-3eO3YbaTU" /></figure><p>One policy is better than another thing if the value function of that policy is greater than the value function of another policy <strong>in all states.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6LTfg6Ti69pqqRdm" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>There is one “ultra” policy that’s better than all the other policies. Not for just specifics, but for all s.</p><p>To find the optimal policy, you just need to solve for q* and then you pick the action that gets you the most q*.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pk2h7ro29ek7lzed" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Remember that the policy tells us the optimal math that we should follow for every situation. Let’s do this one last time with the student MDP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/0*pGEiKTFh1O4F9gTj" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>There’s a special “version” of the Bellman equation known as the Bellman Optimality Equation that really tells us how to actually solve the MDP → how do you relate the optimal value function to itself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*ghqOzT7Axyff4Goi" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>The question will always be “what’s the optimal value of being in some state?” We can consider each of the actions to we might take [which take you to the black notes]. And then when we reach that action, we can look at its value (the number on the arc) and then simply take the maximum of the 2 values. <strong>Look at the value of each of the actions that you can take and simply pick the maximum q-value.</strong></p><p>Now let’s do the other way around q→v. Now we don’t know what the state will be; the wind will have to take us there. Each of those states that we end up in has an optimal value and if we know the value of each of those states that we might end up in, we can average both of them now. No maximum here; <strong>we don’t get to pick where the wind blows us.</strong> That’ll tell us how good our action is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*f9OzugSroL1cvmJM" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Therefore, the optimal action value is the immediate reward + the average over all the probabilities of the wind blowing us left or the wind blowing us right multiplied by the optimal value of being in that state.</p><p>Let’s put those 2 pieces together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/0*dRBU9dCzNBoBqjlx" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>Again, this equation relates v* to itself therefore it is recursive which can give us an equation we can solve. So, we’re looking ahead into the actions we can take and maximize over those while also looking over ahead at the dice the environment might roll [but we don’t control the dice therefore we can’t maximize] and instead, we average over the dice the environment throws at us. Now we look at the optimal value where we might end up and back these things all the way up to the initial state which tells us how good it even is to be in that state at the top here.</p><p>And now we do the same thing for q*:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/0*NtrHFzGDEym79qvV" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>If we want to figure out how good an action is, we first consider where the wind (environment) will take us and then average over the dice which the environment is rolling. Wherever the dice roll (the bottom row), we get to make a decision and maximize the decision we take. And for each of the black dot on the final row, we refer to the optimal action value → <strong>how good is it to be in this state and taking this particular action</strong> and trace it back up all the way to the beginning.</p><p>Back to our student MDP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*bmkmkH8v1U3sQUgg" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf">Source</a></figcaption></figure><p>So we start off with a reward of 6. We look ahead at one step and figure out that the value of the current state (6) is equal to the maximum of the value function of where we end up. You can either follow the arc going up with a reward of -1+6=5 or continue studying and get a reward of 8–2=6. 5&lt;6, therefore we continue studying.</p><p>The <strong>Bellman Optimality Equation is non-linear</strong>. There is no closed-form solution that you can follow. Instead, you need to use iterative solution methods i.e.</p><ul><li>Value iteration</li><li>Policy iteration</li><li>Q-learning</li><li>Sarsa</li></ul><p>Some quick extensions to MDPs (not expanding on this, just general info):</p><ul><li>Infinite and continuous MDPs</li><li>Partially observable MDPs</li><li>Undiscounted, average reward MDPs</li></ul><h3>Dynamic Programming</h3><p><strong>Dynamic:</strong> dealing with sequential + temporal components to a problem.</p><p><strong>Programming:</strong> using a problem i.e. the policy</p><p>Dynamic programming is all about <strong>finding the optimal program to solve a sequential complex problem</strong>. This is generally done by:</p><ol><li>Breaking the problem down into subproblems</li><li>Combine solutions to solve the problem</li></ol><p>Dynamic programming generally needs to have 2 properties:</p><ol><li>Optimal substructure → you can solve some RL problem by breaking it down into ≥2 pieces, solving them, and then combining them to find the optimal problem.</li><li>Overlapping subproblems → subproblems can occur multiple times and by breaking the problem down into subproblems, there is something for us to gain.</li></ol><p>The MDP satisfies both properties → the Bellman equation that we defined above gives us this way to break down the problem ex. “how can we break down the optimal reward function into 2 sub-pieces?” i.e. take one step to the left and then decide what step to take next.</p><p>Dynamic programming assumes full knowledge of the MDP and is used for planning in an MDP. There are 2 main cases:</p><ol><li><strong>Prediction</strong> → given a<strong>n</strong> MDP of our [state, action, transition dynamics, reward, discount factor] and policy, we want to figure out the <strong>optimal value function.</strong></li><li><strong>Control</strong> → full optimization, given a<strong>n</strong> MDP of our [state, action, transition dynamics, reward, discount factor] policy, we want to figure out the <strong>optimal policy function.</strong></li></ol><h4><strong>Policy evaluation</strong></h4><p>This is the case whenever someone gives you the MDP + policy, and given an action, you need to calculate the reward i.e. “if i continue walking straight ahead, how much reward will i get?”</p><p>What’s the problem? <strong>evaluate a given policy π</strong></p><p>What’s the solution? <strong>turn the Bellman expectation into an interactive update system</strong></p><p>Let v₁ → v₂ → … → v_π be the initial value function (what’s the value of all states in the MDP). Then, when we look one step ahead with the Bellman equation, we want to figure out a new value function v₂. This process is then iterated multiple times until we find the true value function.</p><p>This is done through something known as synchronous backups:</p><ol><li>At each iteration k + 1:</li><li>For all states s ∈ S:</li><li>Update vₖ₊₁(s) from vₖ (s’) where s’ is a successor state of s</li></ol><p>So, for every single iteration, we’re going to consider all the states in our MDP, we’re going to start with our previous value function and apply every single state in that value function so that we can produce a new value function of that state. TLDR; is that we run all the states sequentially through the initial value function and then update the value function over time to create new value functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*Kb4vMSU2u2oNHSgU" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>What we’re going to do is take the Bellman equation we had before to update the value function (turn it into an iterative function). We plug in the leaves of the previous iteration, vₖ(s’) next iteration vₖ₊₁(s) and back up those values up to vₖ₊₁(s). And now we simply reiterate that process to find the optimal value function.</p><p>Let’s say that we have a 4x4 grid with 2 shaded squares in the top and bottom corner. 4 actions (N,E,S,W). Regardless of every action you take, you get a reward of -1 and once you land on the shaded squares, you get no more reward and the episode is over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*y2EwQ4YYajObvXRp" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>We’re optimizing for the shortest path to get to the corner. We can use dynamic programming to solve this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/0*mDPjP3k3UmkrCtDv" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>Let vₖ be the random policy that we’re trying to optimize for. We’re going to start off assuming that every square will have no reward. We then apply the Bellman expectation equation as backup and then we pick up the new value of those states. If we pick one of the states i.e. the one highlighted in yellow, whatever direction that you go in you get a -1 reward but we also look at the value according to our previous estimate which says that we’d get 0 everywhere.</p><p>And now we plug in a new value here saying that “if we take a step up, we get a -1 step with a 0 reward.” And now we iterate to k=2.</p><p>Let’s do that same lookahead on the square highlighted in red. If we look N,E,S,W (each with probability 0.25), <strong>each square we move has an immediate cost of -1</strong> + the value where I end up. Therefore, the cost is -1–1 = -2. The value of my current state will also approximately =-2 given the average taken from looking at N,E,S,W.</p><p>Look at the top left square that has a value of -1.7. Remember that if we go north, we just end up in the same square we’re in so we’d just get an immediate cost of -1. Therefore, -1 (N) + -2 (E) + -2 (S) + -1 (W) = -7/4 = -1.75 (rounded up to -1.7).</p><p>After we keep iterating and find the optimal function…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*B7UohjzKHv7-fT_4" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>The right hand side of the images is all about finding the optimal policy using a greedy policy with respect to our value function, vₖ.</p><h4><strong>Policy iteration</strong></h4><p>Given a policy π, how can you return a new policy that’s better than the current one that we have?</p><p>This can be broken down into 2 main steps:</p><ol><li>Evaluate the policy π</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*l3zO9nZPBj_9wmHp" /></figure><p>2. Improve the policy by acting greedily with respect to v_π (look at actions and pick which one is best)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/0*e4cMrFBRTVuTp23z" /></figure><p>In the small grid world example that we have, the improved policy was optimal i.e. π’ = π* where we just followed the values based on the left hand side values. But <strong>the more we iterate over this 2 step process</strong>, we eventually find the optimal policy that converges to π*.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d0k_9ZQJ-FZ_uNnV" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>Let’s use the example of a car dealership:</p><ol><li>states: 2 locations, max 20 cars</li><li>actions: move 5 cars between locations overnight</li><li>reward: $10 for each car rented</li><li>transitions: cars returned and requested randomly</li></ol><p>What’s the optimal policy for shifting cars around each location to get the maximal reward (people get their car)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o82Q4b4DEucEH20M" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>The goal is to figure out <strong>what to do given a certain state</strong> i.e. move x cars to the other location. Immediately after 1 iteration, we notice that sometimes we should be shifting 5 cars to the other location by acting greedily. From there, we take another step and create a new value function V₄ and then look ahead to find the new policy and then reiterate until we find the optimal policy.</p><p>Consider a deterministic policy a = π(s). The way we improve the policy is by acting greedily → pick actions the way that maximizes the action value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/0*4_ksSf1UwixeYNmK" /></figure><p>Remember that q_π is the immediate reward + the value function.</p><p>If we act greedily, this improves the value from any state s over 1 step (looking at only one step for now):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/0*G7VyN6ZmEYnJFyMV" /></figure><p>If we’re taking the max of q_π, this must mean that atleast ≥ one particular q_π which will also improve the value function v_π’(s) ≥ v_π(s) — proof:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/0*2XceHtIsVqPjVZea" /></figure><p>The moral of the story: <strong>If we pick some greedy policy, the value of the greedy policy (the total of the amount of reward we get by acting greedily) is at least as much as the policy that we had before it.</strong> What does this mean? <strong>Acting greedily will always never make things worse.</strong></p><p>But if improvements stop if we have an equality — we want to <strong>make sure that when we hit equality we’ve reached the optimal equation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/0*cG_azpzkK_l7dXE1" /></figure><p>which also means that the Bellman optimality equation has been satisfied:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/0*dQvg7mhBDGrvorYA" /></figure><p>Does policy evaluation always need to converge to v_π and just approximate instead? This is generally known as <strong>modified policy iteration</strong>.</p><p>There are 2 ways people generally deal with this:</p><ol><li>Introduce a stopping condition → ϵ-convergence of value function</li><li>Simply stop after k iterations of iterative policy evaluation ex. k=3 was good enough to achieve the optimal policy</li></ol><h4><strong>Value iteration in MDPs</strong></h4><p>Any optimal policy can be subdivided into 2 components:</p><ol><li>Optimal first action A_*</li><li>Optimal policy from successor state S’</li></ol><p>This brings us to the theorem of the Principle of Optimality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F0imKzjDzLqmQ5-k" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>If we generally know that the solution to subproblems v_*(s’), how do we use that information to build an optimal value function from the previous step? We can find the solution v∗(s) by using the one-step lookahead using the Bellman optimality equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*7lo9ZaeXAk6cl6w9" /></figure><p>Remember that the idea of value function is to make sure that we apply these updates iteratively. <strong>We start with the final reward and work backwards by looping over our entire state space.</strong></p><p>Let’s create another grid world:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p63N7xTuA87qwbiB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>We’re trying to find the optimal (shortest) path to reach g. Generally, one you know that the top left is the goal, then the reward can just be done backwards i.e. the cost adjacent to g would be -1, -2 beside the states that are -1, and then you <strong>can simply propagate through</strong>.</p><p>In synchronous programming though, <strong>we don’t know in advance where the goal is</strong>. Instead, we need to update every single state of all 0’s, and then update by doing the 1 step lookahead and just working from there.</p><h4><strong>Summarizing Dynamic Programming Algorithms</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KD-hZ9f2pVfNqJ1H" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>Generally, all of these algorithms are based on some sort of state-value function i.e. v_π(s) or v_*(s). The complexity O(mn²) per iteration, for m actions and n states when we use synchronous evaluation. This could also apply to action-value function q_π(s, a) or q_∗(s, a) where the complexity is O(m²n²) per iteration.</p><p>Do we have to update every single state in our algorithm? <strong>Of course not!</strong> We can instead use <strong>asynchronous evaluation</strong> where we can just pick any state we want to be the roof of our backup, and then you simply backup on that state and move immediately. You just plug in the new value function at the bottom (remember our graph) and just work our way again. As long as you select every state sometimes, then your algorithm will still converge to the optimal value function with lower compute.</p><p>There are <strong>3 main ways </strong>this is being approached:</p><ol><li>In-place dynamic programming</li><li>Prioritize sweeping</li><li>Real-time dynamic programming</li></ol><p>In-place dynamic programming is more of a programming trick but generally, synchronous value iteration stores to copies of the value function → v_old and v_new. In place is simple; just simply <strong>store only one copy of the value function and just keep the latest value.</strong></p><p>We can update the states using <strong>Prioritized Sweeping</strong>. What this does is allow us to measure how important it is to update a certain state depending on how important we consider it to be. via the Bellman equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Aa8Aw4pkB4sRuoq0" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/DP.pdf">Source</a></figcaption></figure><p>Real-time dynamic programming is mainly about selecting the relevant states that are useful to the agent by using real-world experiences to guide the selection of states.</p><h3><strong>Model-free Prediction</strong></h3><p>This is all about taking <strong>estimating the value function</strong> of an unknown Markov Decision Process and its dynamics + reward.</p><h4><strong>Monte Carlo RL (MC)</strong></h4><p>MC methods are straightforward; you’re just simply learning directly from the episodes of experience. What makes it model-free is that there isn’t any knowledge of MDP transitions/rewards. It learns from complete episodes by using the simple idea that <strong>value = mean (average) return.</strong> Caveat: can only apply MC to episodic MDPs and all episodes must terminate.</p><p>The goal is to <strong>learn v_π from episodes of experience under policy</strong> π where S₁,A₁,R₂,…Sₖ ~ π. Look at the stream of states, actions, and rewards we sample from our experiences. What we look at is the return of the total discounted reward where Gt = rₜ₊₁, γRₜ₊₂ + … + γᵀ⁻¹Rₜ.</p><p>From there, we calculate the expected return of the value function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/0*GIQ5ROmudQcGQBCW" /></figure><p>Instead of using the expected return, the Monte-Carlo policy evaluation uses the <em>empirical mean return.</em> The question is how do we do this when we can’t reset our state back to the point repeatedly and we need to figure this out for all states in our environment. How do we get the mean states from just trajectories. There are 2 ways we can do this:</p><ol><li>First-Visit Monte-Carlo Policy Evaluation</li><li>Every-Visit Monte-Carlo Policy Evaluation</li></ol><h4><strong>First-Visit Monte-Carlo Policy Evaluation</strong></h4><p>Imagine that there is some loop in the MDP where you come back to the same state repeatedly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/0*kMmcofIHdIhBd7dH" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>So, by using more and more random variables, the mean of those random variables approach the actual expectation of the value. Therefore, as we see enough episodes, the idea of Monte-Carlo Learning will really converge on the true example.</p><h4><strong>Every-Visit Monte-Carlo Policy Evaluation</strong></h4><p>Now, the main difference is that we consider every time step instead of the first time step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*4DbmToqUlaQqPWt_" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p><a href="https://ai.stackexchange.com/questions/10812/what-is-the-difference-between-first-visit-monte-carlo-and-every-visit-monte-car/10818?newreg=a75f572243034c76aee759b0099c5e49">Here’s an insightful post</a> that better explains the difference between First and Every-Visit.</p><p>Let’s better understand this with blackjack:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/0*eSZTw0RgouH4GWOH" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Let’s see what this looks like with Every-Visit Monte-Carlo Learning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*QJ7P8GI2JPXxna_I" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Because the useable ace is rare, there’s initially a lot of noise (as indicated in the top-left graph) but over time, by trying to play blackjack and just learning by doing, the agent get’s the hang of it.</p><p>The value function depends on every single aspect of the environment; the dealer’s hand, the deck that you have, the amount of competitors. But we don’t tell the agent anything about this. It just learns over time…</p><p><strong>note:</strong> instead of using mean, we can also use incremental mean rather than the mean to update as we see the sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*vBE4P2PObh9L_WTW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>The new mean is the old mean + some set size towards the difference of the new element and the mean before it. We can implement this with Monte Carlo to get Incremental Monte-Carlo Updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*E_1jDiDA-PqXqYbi" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>By using α, we can introduce an exponential forgetting rate to make sure that we discard information in the past that we may not need.</p><h4><strong>Temporal-Difference Learning (TD)</strong></h4><p>Like Monte-Carlo, TD methods learn directly from episodes of experience + interaction of environment and is also model free i.e. no knowledge of MDP transitions/rewards are required. The difference is that TD learns from incomplete episodes by allowing the model to estimate. <strong>This process is known as bootstrapping.</strong> TD also updates the guess of the value function towards the guess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*A0pYBnV9QZMXiSOB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>We use the Simplest temporal-difference learning algorithm so that we update our value V(Sₜ) towards the actual estimated return after one step Rₜ₊₁ + γV(Sₜ₊₁) which breaks down into the immediate reward Rₜ₊₁ and the discounted value of the next step i.e. γV(Sₜ₊₁) (like Bellman equation).</p><p>We then simply substitute G<strong>ₜ</strong> by using the estimated reward instead of the real return. That is what the <strong>TD target is</strong> while we also have the <strong>TD error</strong> to check the difference.</p><p>Let’s use an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/0*wI7Al0uZePxwN-qx" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>How do we update our value function based on this experience?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*-gY9UsVb2WZvkX4M" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Notice that with Monte-Carlo, we update towards the actual outcome of 43 minutes, and update as you go through each part of the way. TD learning is different — you update based on every single step and then bounce around based on every step until you get to the end of the episode.</p><p>So then, what are the main differences? TLDR; <strong>Monte Carlo learns after the full episode, TD can learn after every single step.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/0*xuKYVVMo61p4Jv8W" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Let’s also take a look at the bias/variance tradeoff:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*bU3CfUB6ZCVDLd0Y" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Notice that the TD target (2nd jot note) has significantly less variance than the unbiased estimate (1st jot note). Why? Because Gₜ predicts infinitely while the TD target has a fixed range of predicting the reward one timestep into the future. <strong>There’s less noise.</strong> Monte Carlo needs more samples to learn. Generally,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/0*LwV28MdVtqh31oMQ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Let’s look at an example of a random walk, simple MDP to better elaborate our goal. You only get a reward of 1 when you get to the right side. This is what applying TD would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/0*9mq5na18Js3LManA" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Initially, at the 0th episode, every state has an equal estimated value of 0.5. But through more and more training, the TD algorithm understands that moving to the right gets you higher reward and will therefore help you reach the end of episode.</p><p>Let’s look at the Root Mean Square error of Monte Carlo vs Temporal Difference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*09li1L6aMTBRU4Ri" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>The RMS(E) is averaged and then taken with respect to the ground truth for the actual estimated values (look above). With more step sizes, you can see that <strong>TD is substantially more efficient than MC.</strong></p><p>Let’s say that we recorded only 3 episodes and fed them into MC and TD algorithms and keep iterating on those episodes for learning again and again. What would happen?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/0*lGk8RxA9VVWxoSBn" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>So initially, you start with state A and get a reward of 0. You then transition to state B but also get a reward of 0. But when you start with B, you get a reward of 1 and onward until the last step. What would the value function be?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/0*DObfueA67U_1zB_l" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>That is what your MDP model might look like. This is the best explanation to understand the amount of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*ImL_SzreK30spzoJ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Monte Carlo always converges to a solution with the minimized mean-square error while Temporal Difference converges to a solution of the MDP that best explains the data. Note that N is the amount of times that we’re in a certain state and action while the reward is just the mean reward.</p><p>The <strong>TD exploits the Markov property</strong> which makes is more efficient because it actually makes use of it. MC though, does not exploit the Markov property which can make it less effective, but it <strong>performs well in non-Markov environments.</strong></p><p>Let’s look at the Monte-Carlo backup:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*ZsuoLS1PWAz6J4bW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>We start with some state, we choose an action, and so on <strong>until that path gets terminated</strong>. So the question becomes “how can we figure out the value function of Sₜ?” What Monte-Carlo does is that it samples one trajectory here (highlighted in red) and then uses that sample to update the value function.</p><p>Temporal difference is quite different:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*mfRlONd1N6SVpeLU" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>We sample the environment, state, action and then back up towards the value function to calculate the reward.</p><p>Dynamic programming also doesn’t look ahead like TD but we also don’t sample:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/0*IXiUOD1rRo5AlRKe" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>We have to know the dynamics to compute the full expectation but now we consider rₜ₊₁, take the full expectation there, and then sum those values up, weight their probability, take the maximum of their actions, and figure out the optimal path. You can also Dynamic programming all the way to the end and this is generally known as an <strong>exhaustive tree search.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*Qa9beNegJHTIhcZR" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Monte-Carlo samples the full environment while TD and DC bootstrap. This is what the unified view of RL algorithms would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*e5TA5JRB4HrAnEXM" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>There are a spectrum of methods between the shallow backups and the deep backups. There’s a unifying algorithm that either end of the special case. This is known as the TD-λ algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/0*yxd1LdGJlmdVC3E-" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>We can define this to be the n-state prediction where we can look into as many states as we want. Remember that when n=∞, then we’re in Monte-Carlo Estimate and not TD-λ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*eYlniYb6NMQfIeDk" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Which n is the best? We need to simulate and see the best n via testing. Let’s use the random walk example again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*p3qykxKBKTjLkVO7" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Notice that as n approaches ∞, we get a very high error, just like MC. The goal is to figure out an algorithm that gets the best of all n at once. We need to average the n-step returns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/0*CbqvgFxaabXcw9r0" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>We don’t need to just take one n; we can use multiple and average over them and call that the target. How can we efficiently use multiple n and be able to properly average over them? We can use the λ return:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/0*3jzAb2C75OhpzQeh" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>The λ-return is this constant that is a geometrically weighted average of all n. The idea is that λ is between 0 and 1 which tells us how much we’re going to decay the weighting of each successive n. It’s simply a normalizing factor. What does that weighting look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/0*vgb9xSNtLxStv8di" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>Geometric weightings are easily computable and are efficient. We don’t need to store more compute for every single return. We can also have it so that λ varies bu we don’t atke that into account.</p><p>Now, we want to update our value function towards that λ-return which means that our forward-view looks into the future will be so that we can compute G^λₜ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/0*MzkTyD6V0dfjXNvP" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>So, the eligibility traces is an idea used to combine both the frequency heuristic and the recency heuristic together. Every time we visit that state, we increase the eligibility state. But then we decrease it over time (notice now eligibility state goes up based on the time of visit to a state).</p><p>Now, when we see an error, we update the value function in proportion to the eligibility trace; the proportion to how much critic was assigned to be in that state. This also brings us to the backward view TD algorithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*ZHPxa89t2okTlRyz" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>What we do instead is that we update the value function for every state in proportion to that TD error and eligibility trace that we get. It’s the one-step TD error because we look only 1 step into the future. We broadcast it back to every state in the past and the value function of each of those states are updated in proportion to the TD error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/0*wmlTs6Vi9ea2wN_4" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><p>The other extreme is that when λ=1 where in that case, the credit we get is deferred all the way until the end of the episode. If we’re in an episodic environment, what’s really cool is that if you accumulate all the updates that you do in the backward-view, you actually end up getting the <strong>same sum of updates</strong> as the forward view. Therefore, they’re both equal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*XTY3LN97WJdQg5lb" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MC-TD.pdf">Source</a></figcaption></figure><h3><strong>Model-free Control</strong></h3><p>Everything that we’ve talked about above is “leading up” to this moment. Model-free control is all about answering the question: “If you dropped an agent into some unknown environment, how can it figure out the right thing to do i.e. maximize its reward?” → <strong>optimizing the value function.</strong></p><p>There are 3 main ways we can address this problem:</p><ol><li>On-Policy Monte-Carlo Control</li><li>On-Policy Temporal-Difference Learning</li><li>Off-Policy Learning</li></ol><p>The main difference between on-policy and off-policy learning is that on-policy is about learning “on the job” while off-policy is more about learning while following someone else’s data.</p><p><strong>On policy</strong> → learn about policy π from experience sampled from π</p><p><strong>Off policy</strong> → learn about policy π from experience sampled from µ</p><p>When is Model-free control used? Generally, some problems that can be modelled as MDP include:</p><ul><li>Elevators</li><li>Helicopters</li><li>Protein folding</li><li>etc.</li></ul><p>For the majority of these problems, either:</p><ol><li>The MDP is unknown, but we can sample experience</li><li>The MDP is known, but is too big to use, except by samples</li></ol><p><strong>Model-free control can solve these problems.</strong></p><h4><strong>On-Policy Monte-Carlo Control</strong></h4><p>Is it possible to simply substitute the Monte-Carlo policy instead of the dynamic programming one to evaluate the probabilities? And then iterate from there until we reach the optimal policy?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*T924306Kottw5GBU" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>There are 2 problems with this:</p><ol><li>The only way you can act greedily for improvement over V(s) is by having an understanding of the MDP via the state value function. I can’t look one step ahead because I want to be model-free.</li><li>Exploration issue → the model does not think about sampling from other trajectories because it might not see a better path with faster reward</li></ol><p>What we can do instead is the action-value function by enabling us to do control over a model-free setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OIVPDn4CG8rQDhfC" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>What would that look like now?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*oJZir6plxWrjieNZ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>Now we start with our q-value function (action-value function) and at every step, we do a Monte-Carlo policy evaluation (running a bunch of episodes) and then take the mean of each state-action pair to figure out how good each action pair is.</p><p>This gives us a new q function of which we then act greedily with respect to that function which gives us a new policy. We continue to run that policy again and again to get q_* and π_*.</p><p>But there’s still another issue again — <strong>we’re acting greedily = we might get stuck.</strong></p><p>Let’s look at an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9V6dKDYxrpCHtoCr" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>We need to figure out which door to go through. If you open the left door, get a reward of 0. But if you open the right door, you get a reward +1. If you’re acting greedily, then there’s only one logical thing to do — <strong>continue opening the right door.</strong> And now you get a reward of +3. So now the Monte-Carlo reward is +2 (mean of the reward i.e. 3+1=4/2=2). And now you just keep on opening this door, forever.</p><p>The problem is that we don’t know the value of the left door — you need to carry on and explore everything so that you understand the q value of all the other actions. <strong>How do you do that?</strong></p><p>Use ϵ-greedy exploration → with probability ϵ, you just simply choose a random action. With probability 1-ϵ, choose a greedy action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/0*VOIk8s7A-2OuKJdz" /></figure><p>The guarantee here is that <strong>we keep exploring</strong> and that <strong>we get an improved policy.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3FOciJ3yh96G8JbZ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>ϵ-greedy really is an improvement. If you evaluate that policy, the question here is that can you be sure that by taking an ϵ-greedy step, can we be really sure that v_π’ is the better policy? The proof above says <strong>yes.</strong></p><p>The idea of Monte-Carlo is to always act greedily with respect to the most recent estimate of the value function. After one episode, if you can already update the value function to something better, why keep the old estimate of the value function?</p><p>How can we really guarantee the optimal policy so that we actually reach q_<em>, π_</em>? There are 2 main things we need to balance:</p><ol><li>Continue exploring</li><li>We want the best policy which might mean that we get to a policy where we’re not exploring anymore.</li></ol><p>We can balance this by using Greedy in the Limit with Infinite Exploration (GLIE). The idea is that we can come up with any “schedule” for learned exploration such that 2 conditions are met:</p><ol><li>We explore everything and all states + actions are visited often</li><li>The policy eventually converges on the greedy policy so that we can satisfy the Bellman equation.</li></ol><p>The best way to do this is by having an exponentially decreasing ϵ:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0lx1_qw8dewWobyM" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>What does that look like in practice?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6-N0fg0T9wkwgm3c" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>If you let this look in an MDP, it will find the right solution. It’ll also be more efficient this way where you update the q value every single episode and immediately improve the policy vs. generating several batches of episodes.</p><h4><strong>On-Policy Temporal-Difference Learning</strong></h4><p>We’ve already seen that Temporal-Difference (TD) learning is better than Monte-Carlo (MC):</p><ol><li>Lower variance</li><li>Online</li><li>Can deal with incomplete sequences</li></ol><p>So naturally, it would also make sense to sue TD instead of MC in our control loop. How?</p><ol><li>Apply TD to Q(S,A)</li><li>Use ϵ-greedy policy improvement</li><li>Update policy + Q function every time-step</li></ol><p>This is called <strong>SARSA.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*tmjIestUotVl7GSg" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>It’s pretty straightforward why it’s called SARSA. We start off with the state action pair as the black dot, we sample a new policy and get our reward R, and then sample our new state S’, sample the policy again of the next state, and we get the new action A’.</p><p>The equation is also intuitive. We’re going to “move” our Q value a little bit to the direction of our TD target (R+γQ(S’,A’)-Q(S,A)).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*pLOG_y1PQhbiSPYP" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>So at every single timestep, we’re going to move up and down. Each time we (agent) takes a step, we update our value function by applying one step of SARSA for the Q value, and then improve the policy greedily.</p><p>What does the algorithm look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fxk6QZ5jSQCkVG3b" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>Sarsa will converge to the optimal action-value function by using the GLIE policies while making sure that your step sizes follow the 2 conditions below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tALOSE-CNlHH1FYo" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>The second point states that eventually, your q values become smaller and smaller and smaller until they become 0 or else there’s going to be noise. In practice, we don’t really care about the second point and emphasize the first point but it works.</p><p>Let’s look at an example of the Windy Gridworld:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2_62JNhmXQp1L-rK" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>The goal is to get from S→G but with every single step that we take, the wind pushes us up the number of grid cells mentioned on the bottom. What would happen if we used the naive version of SARSA?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EUQrK8Kur09yHQGG" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>Notice that it took 2000 time steps until SARSA finished an episode. Once if completes it, it goes up exponentially.</p><p>Can we get the best of both worlds i.e. unbiased behaviour from MC and control the bias-variance tradeoff of TD? Let’s use the <strong>n-step SARSA algorithm:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M2iiekMkMaFKhJ5J" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>We look at the one-step return and then the Q value after one step (n=1) but we can continue doing that all the way down to ∞ which is essentially MC.</p><p>The Q return mentioned in the second point is the generalization to any n. The n-step Sarsa updates for each state-action pair, instead of updating towards the one-step estimate of our value function using Sarsa, we’re going to use the n-step return instead. This can be done for any n.</p><p>Let’s look at the λ version of this sampling:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j2Rz2XE5zksxjJb-" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>This is known as Sarsa(λ). The idea is that we average over all of our n-step returns by weighing everything with respect to λ (take a look at the expressions under each node tree). This is then plugged into the update where we take the weighted sum and update the q value in a little bit direction in the target (q^λₜ).</p><p>The problem is that we’re looking <strong>forward in time,</strong> therefore making this an online algorithm.</p><p>Just like what we did with TD-λ, we’re doing to do the same with Sarsa(λ) by creating an <strong>eligibility trace:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bngSy0WvOJznpfoH" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>Think of this as a way to quantify the amount of “blame” (critic) you should assign to every action you took in every state. The eligibility trace allows you to track + figure out “who is responsible for a certain state?” Generally, the action that we took most recently + the ones that we’ve repeated several times should be the ones blamed.</p><p>The second equation in the first point just states that “if I actually visited a certain state-action pair, then I’m going to increase the eligibility trace by 1.” The algorithm looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*07nFjIX499r5ki2y" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>We initialize the eligibility trace to 0 for each episode and for each step for the episode, we take our actions using the policy, compute our TD error, increment the eligibility trace, and then update the Q function and the eligibility trace.</p><p>What does this look like? Let’s look at another gridworld:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7qDDFjpKevLZn4_k" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>The goal is to get to the tree. Let’s say that initially, you randomize all your state-action values to 0. What we’re going to do is indicate the value of those state-action pairs by an arrow with the direction of that state-action pair (3rd grid) with the size of the arrow indicating how big the q-value is.</p><p>The updates would look like this — if you did one-step Sarsa, when you hit the end goal, you get a reward of 1 in a grid initialized of all 0’s. And now you need to adjust the values of your trajectory to what actually happened. What changes? The end grid cell is what gets updated while all the values become propagated backwards by one step.</p><p>The λ parameter is what indicates how far you should be propagating backwards.</p><h4><strong>Off-Policy Learning</strong></h4><p>We discussed so far about on-policy learning where “the policy which I’m following is the policy that I’m learning.” <strong>But there are cases in which we want to evaluate some other policy while following one policy.</strong></p><p>Why is this important?</p><ol><li>Can learn from observing humans or other agents and figure out how to solve the MDP</li><li>Re-use experience generated from old policies π₁, π₂, …, πₜ₋₁</li><li>Learn about optimal policy while following exploratory policy</li><li>Learn about multiple policies while following one policy</li></ol><p>There are 2 mechanisms that deal with this:</p><ol><li>Important Sampling</li><li>Q-Learning</li></ol><p>Importance sampling is all about estimating the expectation of a different distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/0*X7mTE3mg2ahtK8_b" /></figure><p>The expectation of your future reward is the sum of those probabilities multiplied by how much reward you got. We then just multiply and divide by some other distribution which can give us an expectation.</p><p>How is it used in Off-Policy Monte-Carlo?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WrD63HwqmiRuVMQp" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>We apply Importance sampling to the whole Monte-Carlo trajectory. The problem is that it just has extremely high variance because it samples throughout the whole episode which is why it isn’t used in practice.</p><p>This is why we have to use TD learning instead:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BMXpyuiMsKOqEvfH" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>Because we’re bootstrapping after one step, it becomes a lot simpler because we only need to report for one sample. We update the value function in the direction of the TD target and correct our distribution over it.</p><p>But the ideal way to work with Off-Policy Learning is Q-Learning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X1odsi0VbckmvFMj" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>We’re trying to look at a specific cause where we can<strong> leverage our Q values and the action values to do off-policy learning in an efficient way that doesn’t need Importance Sampling.</strong></p><p>The concept is straightforward; we select our next action using the behaviour policy while considering the alternative successor action had we been following our target policy. And then from there, we update the Q value based on the state we started in, the action we took and update towards the <strong>value of our alternative action.</strong></p><p>In the case we’re trying to learn a greedy behaviour, then we now allow both the behaviour and the target policies to improve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nBV1u9XIe2nJhSYD" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>At every step, we make our target policy greedy with respect to our Q function while the behaviour policy will be ϵ-greedy with respect to the Q function as well. This is known as SARSAMAX:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cxhS0Mxz8LINkF6z" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><p>We’re updating our Q values a little bit in the direction of the best possible next Q value that we can have after one step → like the Bellman Optimality Equation.</p><p>So, to recap, here are the relationships between DP and TD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0sVwP3osHZwrT1Zp" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf">Source</a></figcaption></figure><h3>Value function approximation</h3><p>Games have a lot of states ex. Backgammon has 10²⁰, Go had 10¹⁷⁰ states, but continuous state spaces such as controlling a car has an infinite number of states. <strong>You can’t create a table and just use that scale up. It isn’t gonna work.</strong></p><p>What’s the goal here? How can we scale up the model-free methods for prediction and control from the last two lectures?</p><p>The problem with large MDPs is that there are way too many states and/or actions that we need to store in memory. What’s the solution? <strong>We can estimate the value function with function approximation.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*az5G7TmeDRqFEjR1" /></figure><p>What we’re going to do is consider the true value function v_π(s) and create a new value function that estimates this thing everywhere (v hat). <strong>w</strong> is just some weight (think of it as the weights of a neural network) that we feed into v hat to estimate the value function of the state space. The goal is to generalize from seen states to unseen states while parameter <strong>w</strong> is updated using MC or TD learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZpoNaZIlYFkQ7QaG" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Think of this image as 3 different types of value approximation functions that you can sue. The idea is that if we’re trying to do state value approximation (v hat), we have the “black box” which takes in the state, the box that tries to map from the state using internal parameters <strong>w</strong>, and then outputs <strong>how good the state is.</strong></p><p>For value function approximation, there are 2 main ways we can approach it: action in (2nd box), action out (3rd box). In the action in case, we input the state and action with the goal of the black to figure out <strong>given the input state and considered action, how good would that be?</strong> We then fit everything in the black box which returns us those probabilities with parameter <strong>w</strong>.</p><p>Action out is different. Given an input state, we want our approximation function to give us all the actions that I might take all in one forward pass. <strong>You get all the information you need from just a single pass.</strong></p><p>What do approximation functions look like:</p><ul><li><strong>Linear combinations of features</strong></li><li><strong>Neural network</strong></li><li>Decision tree</li><li>Nearest neighbour</li><li>Fourier / wavelet bases</li></ul><p>We want to focus on <strong>differentiable approximation functions</strong> i.e. where we can update our parameter <strong>w</strong> because we know the gradient and how it’ll affect the output (the ones are bolded). The data that we’re dealing with is not supervised. We require a training method that is suitable for non-stationary data.</p><h4><strong>Gradient Descent</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w5sqHq9O-UBF9GJ9" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>The gradient will tell us the direction of the steepest descent; our goal is to follow this slope as far down as we can get to the local minimum of J(<strong>w</strong>) and then adjust from there.</p><p>What’s the goal? <strong>to find the parameter vector w which minimizes the MSE between approximate value function v hat and the true value function.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*EZ53HcbKI7AaoRvW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Each state can be represented as a feature vector that can contain info about our states (location, weather, etc.). The goal is to find a linear mapping between the features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NG8Nc9T3diD-ZP_9" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Because our objective function is MSE, that means that our objective function is quadratic which makes it really easy to find the optimal parameter values.</p><p>We cheated using these methods a lot. There is no supervised method in RL. We need to figure out how to directly find experience. How do we do this? <strong>Monte-Carlo and Temporal-Difference!</strong> Why? We can get a target to use as our function approximator instead of having labelled data (oracle).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uviKsN3LVo91Z9qD" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>For Monte-Carlo Learning, we change our weights of the parameter vector <strong>w</strong> in the direction of error between the actual return and our function estimator multiplied by its gradient.</p><p>TD Learning uses the bootstrap instead. What we’re going to say is instead of waiting for all the way until the end of the episode to figure out the total reward, <strong>I’ll just simply take a step, estimate the reward, and use that as the “ground truth”.</strong> We use the TD estimation as labelled data to update the gradient of our parameter <strong>w.</strong> Same idea with TD-λ where we use λ as the training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JL93evVbZhdLerJO" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>The same idea for TD-learning will work as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5J1UKtLsk3v6PxaB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Again, notice that our corresponding label is the TD target. Same idea with TD-λ.</p><p>What does control for the value approximation function look like? We’re still going to build on the idea of generalized policy iteration. But now we’re going to <strong>use approximate policy evaluation</strong> where we start off with the parameter vector <strong>w</strong> and act greedily with respect to our neural network to find a new value function and then update the policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g1ehBLvYdL91OSYi" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>We need to also approximate the value function. So, for each state in the action-value function, we’re trying to predict the expected reward and then minimize the MSE again to find the local minimum using SGD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TjIGAkLgppouI77r" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Now we need to represent our state and action as a linear combination of feature vectors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fs64xyuLjs-O6Z6N" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>We can use the same idea for Monte-Carlo Learning and Temporal-Difference:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jg6EEZv1_t8cj2z-" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Let’s look at the Mountain Car experiment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R4vpH37X7W-_2gTJ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>The state space is 2D: we have the position of the environment and the velocity we’re going at. The more episodes + steps we go at, the more distinct shape emerges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ECR93kwTCINKdUBD" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>TD doesn’t follow the gradient of any sort of objective function which is exactly why it can diverge when off-policy or using non-linear function approximation. We can instead use the Gradient TD function because it follows the true gradient of projected Bellman error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DgdnH-YyAHcXZMma" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>How does this compare against control algorithms?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NXrTmjIX2uAo9KbY" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Gradient descent is very simple to set up and is appealing to use. The problem? <strong>it isn’t sample efficient</strong>. When we see an experience once and take our update, we just simply adjust our function and then throw that experience away and move on. The problem? It isn’t data efficient — we haven’t found the best valuable fit to our value function. Batch methods seek to find the best fitting value function given the agent’s experience (“training data”).</p><p>What does it mean to find this best fit? We can use something like the least-squares algorithm to find a proper fit the value estimator to our true value function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cJRKtyCz9Q2hSSWr" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>There’s actually a very easy way to find the least-squares method: <strong>experience replay.</strong> What we do is that instead of throwing away our data, we turn it into an actual dataset. At every timestep, we sample a state + value from that experience and do an SGD update on it (<strong>just like supervised learning</strong>) until we find the least squares solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*rpc9fEZX5iAHTm-A" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Deep-Q-Networks (DQN) uses <strong>experience replay</strong> and <strong>fixed Q-targets:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H_0XINSizWKCzidR" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Experience replay stabilizes the network because it decorrelates the trajectories because you shuffle + sample different experiences for better correlation fit. The other idea is to have 2 different Q-networks where we basically freeze the old network and bootstrap towards our targets.</p><p>What does that look like in Atari? We use end-to-end learning of the values Q(s,a) given the pixels (which are our state s) that are fed into a Convolutional Neural Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_pyVDwGH6dEvn_Cv" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>How did it do compared to humans?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4bqyskjbAhcdEoEX" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>Everything to the left of that grey threshold are instances when the RL agent did better than a human.</p><p>We have this method where we use several iterations to hit the optimal solution. But what if you don’t have many iterations to train with? With more computation/iteration, the more you can do in one step to get to the local minimum faster. What does that look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qiL-0sxIXFHPfrHg" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>When we hit the minimum of LS(w) then that technically means we’ve found the optimal solution. To recap:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*g06D-5XN5BW7YbXq" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>These have better convergence policies compared to the normal, incremental MD/TD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_qxENtC7KhNe2DgJ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf">Source</a></figcaption></figure><p>We apply this by using the Least Squares Policy iteration algorithm where we evaluate the policy by least-squares Q-learning with a greedy policy improvement.</p><h3>Policy gradients</h3><p>Generally, we’ve focused on creating a policy from the value function ex. using ϵ-greedy. We actually parameterize the policy and not the value function i.e. we control the parameters of π which affects the distribution by which we’re picking actions (remember: this is model-free RL). We need to scale and find parameters that allow us to generally scale.</p><p>We need to better understand how we can actually take this parameterized policy (and its parameter vector <strong>u</strong>) and adjust its values to get more reward via. gradient ascent — if we follow the gradient, we move uphill in such a way that we improve our policy.</p><p>Sometimes, it’s more efficient to represent the values of the policy than the value function itself, we might also have situations where the value function could be complicated vs figuring out the action.</p><p>Main advantages:</p><ul><li>Better convergence properties</li><li>Effective in high-dimensional or continuous action spaces</li><li>Can learn stochastic policies</li></ul><p>Disadvantages:</p><ul><li>Converges to local rather than global optimum (naive policies)</li><li>Evaluating a policy is inefficient and has high variance</li></ul><p>How does all of this work?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LPgOh6cLpSasa9c4" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>If I always start in some state s₁ or if I have some sort of distribution over start state s₁, what’s the total reward that I’ll get from that state onwards?</p><p>The other way is the average value or we can also consider taking the average of my immediate rewards over the entire distributions of states that I visit.</p><p>How do we optimize all of these methods? Policy-based RL is <strong>an optimization problem.</strong> Our goal: <strong>find θ that maximizes J(θ).</strong> Generally, we can get greater efficiency when using gradients i.e. gradient descent, etc.</p><p>Let J(θ) be the policy objective function we’re trying to optimize for — the goal here is to search for a local minimum in J(θ) by ascending the gradient of its policy w.r.t the parameters θ:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*TgrrbVcRs9eAm5FP" /></figure><p>Where α∇_θJ(θ) is the actual <strong>policy gradient</strong> and α is the step size parameter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/0*tZDB4BhZXU2lP1bu" /></figure><p>The overall update process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hw5FTr4lkM3JWgo8" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>Let’s assume that we want to compute the policy gradient analytically [assuming that the policy is differentiable when picking actions] and that the now the gradient of our policy. We can use the likelihood ratio trick; when we want to take the expectations of the gradient of the policy, we can use this to move in the direction to get more of something.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/0*LGMkpZKZZ9g4rXxm" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>By rewriting the gradient in this way, we’re able to take expectations.</p><p>We can use the softmax policy to prioritize our policy in a discrete domain ex. for Atari, to know if we should go left vs. right, we can weigh the features of going left or right and whichever is more, we would pick that action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ZtSdnUHekU7samZ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>The score function is very intuitive for the softmax function; it’s just the feature of the action that we actually took (left) subtracted by the average feature for all the actions that we might’ve taken.</p><p>Another policy that we can use is the Gaussian policy. We just generally parameterize the mean of the Gaussian and then add some noise to make it stochastic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MdHxHrWqkTYtJfsk" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>Let’s look at this for one-step MDPs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ba_I2rVWerqQpYiH" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>If we want to increase our objective function (if we want more reward), we just need to move in the direction that’s determined by the score*immediate reward.</p><p>We want to do the same thing in multi-step MDPs and to do that, we need to replace the instantaneous reward (score*immediate reward) with the value function (long-term reward). This is defined in the policy gradient theorem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*08pfGkwF-9_LgkwF" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>If you start with some policy, then for any of the objective functions we considered early at the start state, the policy gradient of that is given by the equation in red (expectation of score function * value function Q).</p><p>Where do we use this? In the Monte-Carlo Policy Gradient algorithm (aka REINFORCE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M-jpGdZsdnXKYHUW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>The idea here is to update the parameters with SGD. What we do is that we sample the expectation by using the return as an unbiased sample of Q. We plug that into the policy gradient to give us a direction (estimate) of direction to move. <strong>We then adjust our parameters a little bit in the direction of this score*return.</strong></p><p>Let’s look at an example of a Puck World:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9PR7hU_YKq4i7veG" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>The idea is that we start with a puck and you need to adjust a little force on it every time step to find the target (pentagon) which moves every so now and then. We just run this and use the Monte-Carlo Policy gradient to update into the direction of those returns.</p><p>Notice that we get a really nice learning curve and the scale here (hundred million iterations). It’s slow.</p><p>Here’s the TLDR; of the main ideas:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QNv-czPzX0AIWHZB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>How do we estimate the value function Q? <strong>We can use MC policy evaluation, TD learning, TD-λ, or even least-squares policy evaluation. </strong>Here’s an example:</p><p>Let’s say that we were using a linear value approximation function for the critic; we can figure out Q by just using a state and action * critic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8twCLDzw0riaLxMJ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>Remember that policy gradient update for actor = score * critic. For every step of the algorithm (not until end of episode), we sample the transition and see what the reward was, we see where we ended up after picking the previous step. From there we pick our own policy and get the TD error between the value before that step vs. the value after that step, and then update our critic in the direction of the TD error * features [making is linear].</p><p>We also update the actor in the direction that gest us more of the things the critic says are good.</p><p>Generally, whenever we approximate the policy gradient, we end up introducing bias to the model. The problem with that? <strong>Biased policy gradients might not find the right solution. </strong>But if we choose value function approximation carefully, then we can avoid introducing any bias.</p><p>How do we make all of this better? We have this “family” of actor critic methods where the actor picks the actions and then we have a policy that helps pick those actions. We have a critic that evaluates this and tells us whether a certain action was good or not. And then the actor moves in the direction suggested by the critic.</p><p>But how do we really make this better? We can use a trick known as the <strong>baseline.</strong> The idea here is to subtract some baseline function from the policy gradient in such a way that it doesn’t change the direction of percent (expectation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GBs3XVSg7NXA_vbn" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>What this all tells us is how much better than usual is a particular action a and then <strong>how can we adjust our policy so that we can achieve action a.</strong></p><p>How do we estimate this function now with our critic? One way to do this is to learn <strong>both Q and V</strong> by having 2 sets of parameters. We can simply take the difference between those things as our estimate for the advantage function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yXQX2XX14LlMLncD" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>But the easiest [and most common] way of doing this is through this idea: <strong>the TD error is a sample of the advantage function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pTkodl0PUgUI70TJ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>If we use the true value function V^π, the expected TD error = expected reward + the discounted value at the next state — value of the state I was in. The expected reward + the state value at the next state is literally the definition of Q (refer to Bellman equation) which all defines the advantage function A^π.</p><p>What about using eligibility traces and different time scales to get a backward view? We can plug this idea into our actors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D8KY5a99p6Q2PK9u" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>We compute the TD error, we build up an eligibility trace, and then the eligibility trace now becomes an eligibility over our scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9prCmdm3l_ZNV1di" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/pg.pdf">Source</a></figcaption></figure><p>With actor-critic algorithms, there’s a really important question that really needs to be answered: how do we know really if the critic is pushing us in the right direction that we should be following? If you choose the value function approximation carefully, it would be possible to pick a value function approximator that doesn’t have any bias. This is known as <strong>Compatible Function Approximation.</strong></p><h3>Integrating Learning and Planning</h3><p>We’ve talked a lot about learning a policy directly from experience and learning a value function directly from experience. But how can we actually have a model learn directly from experience and construct a value function or policy? The goal here is mainly to integrate learning and planning into a single architecture. This can be done with <strong>Model-Based Reinforcement Learning.</strong></p><p>Well what’s the difference between that and Model-Free RL?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7cjTv3ECLx5R-GOd" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>This is what Model-Free RL looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*hRRZGtccDNCDLTv9" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>This is what Model-Based RL looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/0*uTJsJraX9BexuIkb" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>What’s the difference? <strong>We replace the real world with the agent’s perception of the world. </strong>We’ve replaced the real interactions (that the agent has with the world) and replace it with simulated interactions between the model and the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/0*V1haF7qjBOWEdNao" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>As the agent interacts with the real world (experience), it then builds up its own model about its dynamics, you get a model which you then use to plan. The interactions with the model then create the value and policy which is then used for the model to act. Think of the model learning as a learning MDP and the planning is then we’re solving that MDP. And then you update via iteration.</p><p>Main Advantages:</p><ul><li>can efficiently learn via supervised learning</li><li>can reason about model uncertainty</li></ul><p>Disadvantages:</p><ul><li>has to learn model, then value function</li></ul><p>A model M can be a representation of an MDP and its parameters i.e. &lt;S, A, P, R&gt; which are all parameterized by η:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0ytltmZWc5b8ySat" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>These are all one-step models ex. reward function would be “what would the reward be at the next time step given the previous state and action?”</p><p>What’s the goal here? <strong>estimate model M_η from experience i.e. {S₁, A₁, R₂, …, Sₜ}:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0tDtr8rpUlBfOBJB" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>General examples of models:</p><ul><li>Table Lookup Model</li><li>Linear Expectation Model</li><li>Linear Gaussian Model</li><li>Gaussian Process Model</li><li>Deep Belief Network Model</li><li>etc.</li></ul><h4><strong>Table Lookup Model</strong></h4><p>In the case of a Table Lookup Model, the model will always be an explicit MDP P hat, R hat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3u05zoMIOH5UdsnA" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>The transition dynamics is straightforward; take the average of how many times I ended up with s’ (state). Same thing with reward; it takes the average of all the times it was in a state and action given s and a. The alternative way is just to record every single experience that you get and just simply sample the model from there.</p><p>Let’s go back to the AB example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*taKZtbeEZCOH4D_d" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>What would our model planning look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/0*9OmkBNBzsorBhuyk" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Planning means to literally solve the MDP and find the optimal value function + policy. Remember that we said no to Dynamic Programming algorithms because you have to be told what the MDP is before you apply them. But what we’re saying is that you can to RL with them, all you need is to learn the model and then use that model to do Dynamic Programming.</p><p>We can use Sample-Based Planning. It’s straightforward — we use the model <strong>only to generate samples</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_xjuRj0q1jkrrR3e" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Going back to the AB example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*79hS4HoCIqHfZlcf" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>We start off with real experience, generate the model, and then use that model to sample experience. We’re extrapolating data (generation) that we can apply to MC learning and then retrain the model. Think of it like this: let’s say that the real experience that you get is from doing homework which then allows you to understand what you’re doing in class. Now, whenever you’re given another problem to solve, you already know how to solve it, you’re just training your mind (model) to become better and better. <strong>That’s exactly what’s going on here; we learn from data, create a model, sample from experience, and then retrain.</strong></p><p>What happens if our model isn’t right? What happens if it really is inaccurate? We should never expect to get the right answer with RL. T<strong>he</strong> performance of model-based RL <strong>is only as good as the optimal policy it is given.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KdPighLS8hDiXubG" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><h4><strong>Integrated Architectures</strong></h4><p>There are 2 main ways we can generate experience: real and simulated experience:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bbD1K0xpu-YYEt6z" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>What would happen if we combine both real + simulated experience together? <strong>Enter Dyna-Q:</strong></p><ul><li>Learn a model from real experience</li><li><strong>Learn and plan</strong> value function (and/or policy) from <strong>real and simulated experience</strong></li></ul><p>What does that look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/0*tL-KxQXyd0fFhCXM" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Here’s the rundown of the algorithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pkzNY_IbOffquipf" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>We start off with action-value Q and some model. All that we’re going to do is plan with our table-lookup model in addition to our real experience. So, at every step that we take, we take a real action in the world, see where we end up, and then do 2 things:</p><ol><li>usual Q update (Sarsa) towards our one-step lookahead</li><li>update our Model in a supervised fashion toward the reward + state that we observed after that one step. We have this in a loop.</li></ol><p>And then we can sample n different states, actions, rewards and then “imagine that transition” to apply the Q-learning step to that transition (re-learning from past data). The more we sample different paths, the better the model.</p><p>How does Dyna-Q do in the grid world?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tALCi3Eesy1N-2tK" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>If I don’t do any planning (n=0), then then we “converge” after 5 episodes. But by setting n=50, we figure out the optimal policy within 5 episodes.</p><h4><strong>Simulation-Based Search</strong></h4><p>We’ve talked a lot about model-based RL. Let’s scope down to the planning problem and how we can use samples of imagined experience for good planning.</p><p>We can use Forward search since they’re generally focused on selecting the best action by lookahead. How? We use a search tree with the current state sₜ at the root and then use a model of the MDP to look ahead:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8pm165Ej1jC1UulF" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Simulation-Based search simulates those episodes of experience from <strong>now</strong> with the model by applying model-free RL to those episodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*Gh5tz9HDZN7uj9Cf" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>What does this look like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Uv2TLP4qfTDvX4w" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Let’s start with the simplest possible version of all of this: Simple Monte-Carlo Search:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qrLh2ZuvsL7JCfDT" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Instead of just evaluating the root actions i.e. going left vs right, we’re going to evaluate every state-action pair that we visit and build a search tree containing all the states and actions that we’ve visited so far and evaluate them.</p><p>But the real question is: how do we apply this evaluation method to simulation?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-jSDD86uoHL5cl2m" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>After every simulation, we’re going to improve it. We look at the Q values, maximize over them, and then make them better. The key distinction here though is that we don’t have a complete table of q-values everywhere and are only in the search tree. That’s why there are 2 phases (in-tree, out-of-tree). When we’re in-tree, then we maximize the policy, but when I’m out-of-tree, then I behave according to my simulation policy.</p><p>At every simulation, we evaluate the states via MC evaluation, and then improve the tree policy via ϵ-greedy. This is literally <strong>MC-Control for simulated experience.</strong></p><p>Let’s look at Go (one of the hardest tasks for AI):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LFM_bMlRvVU-4FPq" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>What would policy evaluation look like here?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*10ssktytw1llCZUt" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>We’re going to create a reward function and simply base it off who wins. We’re playing both sides (black and white) and therefore define policies for both.</p><p>MC Evaluation will look like this: we’re in a current state s and we want to know whether we’re going to win or not. We we role out some possibilities via simulation (4 different versions) and then collect the possible reward which is a good estimation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/0*oG7ZerVT8Hq9FLbk" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>This approach is very naive. Let’s turn this into a Monte-Carlo Tree Search algorithm instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/0*hCZh6HO5ubTcl6Iv" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Because black won that game, notice that we get a 1 for it. And now we fill in the value in our search tree (1/1). And then we use the tree policy to guide us as we make more simulation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*oMfL677IBC91VIjQ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>What’s the general advantages of MC Tree search?</p><ul><li>Highly selective best-first search</li><li>Evaluates states dynamically (unlike e.g. DP) and not the entire state space (current position)</li><li>Uses sample-based planning methods to break curse of dimensionality</li><li>Works for “black-box” models (only requires samples)</li><li>Computationally efficient</li></ul><p>Monte-Carlo is just one way of a search algorithm. What about Temporal-Difference Search and other model-free RL approaches? We do our simulation-based search again but use TD (bootstrapping) instead of MC and apply Sarsa to sub-MDP from <strong>now.</strong> Why?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hGbtmdvEa-jOUrsW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><p>Again, we simulate episodes based on our current (real) state sₜ with the goal being to estimate the action-value fiction Q(s, a).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cY1rWQq7Msv3J2Kr" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/dyna.pdf">Source</a></figcaption></figure><h3>Exploration vs Exploitation</h3><p>What even is this Exploration vs Exploitation dilemma? Whenever we have online decision making, we’re left with 2 choices:</p><ol><li><strong>Exploitation →</strong> Make the best decision given current information</li><li><strong>Exploration</strong> → Gather more information</li></ol><p>I’ve stated this near the beginning of the article: <strong>the best long-term strategy might have short-term sacrifices.</strong> Why? So that you can gather enough information to make the best decisions.</p><p>There are 3 different approaches to this problem:</p><ol><li>Random exploration → explore random actions</li><li>Optimism in the face of uncertainty → estimate uncertainties based on values and explore the states/actions with the highest uncertainty</li><li>Information state space → given the agent’s information of its state, use lookahead to see how that information can help the overall reward</li></ol><h4><strong>Multi-Armed Bandit</strong></h4><p>It’s essentially a way to simplify the MDP network where we just have a tuple of &lt;A, R&gt; (throwing away state + dynamics matrix).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bEL1OVQfqWJDJ__5" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>Think of each machine as a one-armed bandit — each arm you choose results in a different layout. Which one do you choose next? Remember that the criteria here is the maximize cumulative reward…</p><p>We also have something known as regret which is essentially the difference between the optimal value and the action that we chose:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*CHjyZJN8kBXeB63o" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>Regret can be thought of as a function of these gaps and counts that exist:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/0*DFHhc_B8fxcyvzwX" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>Anytime a gap is large, you need to make sure that you pull that arm very few times; the goal is to always pick the arm with the smallest gap. What’s the problem? The gaps are unknown because we don’t know v*!</p><p>What does this regret look like over time?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*FG49DLu-PGTCf12E" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>For most naive algorithms, we just keep on adding regret linearly. Even when we act greedily because we might end up locking on the sub-optimal action. ϵ-greedy might not work as well because we end up exploring the whole time. We want something that has less and less regret given the more time steps of training. Is this possible?</p><p>Yes! We can use a <strong>Decaying ϵ-Greedy Algorithm</strong> instead where it decreases over time. What happens? We end up getting a <strong>logarithmic (as guessed from the graph) asymptotic total regret!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/0*rG3r02ZBQUo7wJlK" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>You can’t do the schedule in practice (because it requires knowledge of v*) but let’s say someone told us v*. We then use this information to figure out the size of all our gaps. Then, we can just simply invent a schedule to figure out the size of the smallest gap between the best action and the second best action.</p><p>But all algorithms have a lower bound meaning that it cannot perform worse than some sort of logarithmic threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mZ0VV0WbiIEOzout" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><h4><strong>Optimism in the face of uncertainty</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wVYWPn9jtSLPn3Jv" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>Imagine that there are 3 arms: the blue, red, and green arms. Each arm is showing the distribution over their q values. The x-axis is the corresponding reward for an action. Remember that the flatter the curve, the more uncertain we really are about picking a certain arm. Which arm should we pick next?</p><p>What does Optimism in the face of uncertainty say? Take the one that is the best (green), and take the one that has the potential to be the best (blue). And now, as you try each arm more and more and more, you narrow down the distribution to a specific reward.</p><p>How do we estimate this uncertainty though? <strong>We use the general idea of Upper Confidence Bounds (UCB)</strong>. The idea is that I’m not just going to estimate the mean of a particular arm, but also some sort of confidence of where it might be. So for each distribution, we’re not only going to estimate the q value + some boundary to figure out the tail of the distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*ZPABEbOK5bI5d9Y0" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>We pick the action with the highest confidence (we know where the reward will be at). The less we try the error, the larger the confidence will be (uncertain) vs smaller confidence (scoped down to figure out a good range of where our rewards might be).</p><p>We apply something known as Hoeffding’s inequality to figure out the rewards of a bandit that has to select action a.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8WN9rR1mYjr1ed4c" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>We want to figure out “what’s the probability that ‘”m wrong about my estimates of these q values more than my U value?”</p><p>From there, we pick a probability p and then solve for Uₜ(a):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*7v9-aceBiMw-s7hS" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>What’s nice about this is that <strong>we don’t need to know anything about gaps or rewards and boundaries and now a way to pick the optimal action</strong>. And notice that as we pick things more and more and more (Nₜ(a)), then Uₜ(a) will eventually hit 0. We also schedule our p as we observe more and more to become more confident in the q value.</p><p>This brings us to the UCB1 algorithm: at every step, you estimate the q value by taking the Monte Carlo estimate (imperial mean of everything you’ve seen so far) and then add the bonus term based on the amount of times you’ve pulled an arm (t) divided by the number of times you’ve pulled a certain arm (Nₜ(a)). And then you simply just pick the highest action value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BD8kRo8866EKkPA9" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>How does it do in practice?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*T0RctqNA1pTAA57F" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>What’s interesting is that if you tune ϵ-greedy well, then it can do as well as UCB1. But if not, then it’s significantly worse.</p><h4><strong>Bayesian Bandits</strong></h4><p>What we can do with this approach is be able to exploit prior knowledge about rewards p[Rᵃ]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VbR_WjzBda2u5iLH" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>What the Bayesian approach does is (with its experience), update <strong>w</strong> i.e. the mean μ + variance σ² of each of the arms we could pull.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XcekJHx562HRMdbx" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>We compute the posterior distributions of what each distribution looks like after we get information. What’s interesting is that we can use that posterior information to guide our exploration via UCB and/or probability matching. If someone gave us useful + accurate prior information, then it would make more sense to actually use this Bayesian approach. But if it’s wrong, then you should just go ahead with the UCB approach.</p><p>The other way to make use of our probability distribution is to use Probability matching. Essentially, Probability matching selects some action <strong><em>a</em></strong> according to the probability that <strong><em>a</em></strong> is the optimal action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R-TPpgcK8EJtmaR_" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>What’s really nice about probability matching is that it’s built to do optimism in the face of uncertainty i.e. the more uncertainty about something, the higher the probability that it could be the maximum value. We do this through Thompson sampling [which is a way to do sample-based probability matching].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RL4PVyfPzihc_116" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>This is generally the oldest algorithm for bandits but it actually achieves the lower bound for this algorithm (works optimally well). The way it works is that at every step, you sample from your posterior (randomly sample from distribution). And then whichever arm distribution has the highest sampled value, we just go with that.</p><h4><strong>Information State Space</strong></h4><p>We’ve explored 2 types of algorithms so far: randomized exploration algorithms (ϵ-greedy) and UCB. Now let’s look into how we can quantify the value of information.</p><p>Exploration is useful in an agent because it gains information about its environment. But how can we really quantify the value of information? 2 main ways:</p><ol><li>How much reward a decision-maker would be prepared to pay in order to have that information, prior to making a decision</li><li>Long-term reward after getting information — immediate reward</li></ol><p>If we can quantify how much long-term reward we might get by picking an unknown arm, how much would that information be worth for me? Can we quantify that?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cMh9X-MLRn-h_SXu" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>The best way to do this is to transform our Bandit problem back into an MDP (sequential problem) and use the information state s~ to summarize all the information that we’ve received so far.</p><p>But in order to solve Information State Space Bandits, we need to formulate the bandit as an infinite MDP over information states but it can be solved with Reinforcement Learning!</p><ul><li>Model-free RL → Q-Learning</li><li>Bayesian model-based reinforcement learning → known as Bayes-adaptive RL where you find the Bayes-optimal exploration/exploitation trade-off with respect to prior distribution</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xlTWKm2r5wl0m16W" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/XX.pdf">Source</a></figcaption></figure><p>The way Bayes-Adaptive Bernoulli Bandits works are that you have some sort of Beta prior. And then for each action a, we have the Beta prior telling us how good that action is. And then every time an action is selected, then we just simply update the Beta posterior.</p><p>We can solve Bayes-adaptive MDP can by using dynamic programming formally known as the Gittins index. But recent methods such as MC tree search works and can be used to play a lookahead tree to find the Bayes-optimal exploration-exploitation tradeoff.</p><p>To sum everything up so far, let’s just review the Multi-Armed Bandit Algorithms</p><p>Random exploration:</p><ul><li>ϵ-greedy</li><li>Softmax</li><li>Gaussian noise</li></ul><p>Optimism in the face of uncertainty:</p><ul><li>Optimistic initialization</li><li>UCB</li><li>Thompson sampling</li></ul><p>Information state space:</p><ul><li>Gittins indices</li><li>Bayes-adaptive MDPs</li></ul><h3>Applying RL in SOTA Games</h3><p>This is just a quick overview of some of the biggest games and how well RL agents have performed in them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jbPn2XIOADFq6TXL" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>What does optimality in these games even look like? How do we really know if we’ve actually hit “superhuman” level? Remember that the optimal policy depends on how the other player plays. But we can use the idea of <strong>Nash equilibrium to define what optimality looks like:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UVHozmszW6a9qWxY" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>If all the other players that you play against have a fixed policy π⁻ᶦ except for the ith player πᶦ, then you can create a <strong>“best response”</strong> against that set of policies. If you all are using the same strategy against me, then I can compute the best strategy against all of you. Nash equilibrium is defined when <strong>every single player plays the best response.</strong></p><p>The best response will only be the solution to a single-agent RL problem where all the other players become part of the environment (we’re trying to operate in a way so that we can beat our opponents so we need to take them into consideration. In this case, the best response is optimal policy for this MDP.</p><p>Nash equilibrium in this case would be the fixed-point in the self-play RL where we can generate experience by playing games between agents. Each agent tries to learn the best response to other players where <strong>one player’s policy determines another player’s environment.</strong> We can then create a system where all the players adapt to each other.</p><p>There are generally multiple classes of games that we’ll focus on:</p><ol><li>Perfect information games (fully observable by all players ex. chess)</li><li>Two-player games (player 1 = white, player 2 = black)</li><li>Zero-sum games where there are equal and opposite rewards for black and white (what’s good for black is bad for white)</li></ol><p>How can we find the Nash equilibrium in Zero-sum games?</p><ol><li>Game tree search (i.e. planning)</li><li>Self-play reinforcement learning</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8CkBkxGa3XE9D1r5" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>Minimax is straightforward; we maximize white’s expected return while minimizing black’s. ex. if black played the best possible mode for black and white played the best possible mode for white, <strong>what would the final outcome of the game be?</strong> The minimax pair is just a joint policy for black and white that achieves those minimax values while achieving a Nash equilibrium.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WxPMmzAfoh5J3llg" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>We can use a depth-first search to then figure out the optimal value from a given situation. The problem though is that search trees can grow exponentially and it would be extremely impractical to search until the end of the game.</p><p>What we can use instead is a value function approximation (or evaluation/heuristic function) where we use it to estimate the minimax value for each leaf node. You don’t need to search until the end of the game; we can use the value function approximator, get a value from that, and then propagate back upwards.</p><p>The simplest way these value function approximations were built was through something known as Binary-Linear Value Functions where we have a binary feature vector that contains information about which features are present in this position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UqpZjRhXjLB5I2S3" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>In this case here, the features present are the white rook, bishop, king, and then the black rook and king. We then multiply this by the actual point value for each piece and then take its product to figure out who’s winning (in this case it’s white with +3).</p><p>One of the most famous chess programs ever written was DeepBlue which was the first chess program to defeat a world champion (Kasparov):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/0*7kr-pH_-e8WMNFgJ" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>It looked ahead 16–40 searches into the future and used that information to determine the next sequence of moves.</p><p>What would happen if we applied value-based RL algorithms to games of self-play i.e. chess + checkers to find a minimax solution? We can use Self-Play Temporal Difference Learning:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*quy8ypz7Mfr35eQW" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>The idea again is simply to do something like gradient descent to update those parameters <strong>w</strong> (we’re assuming that all immediate rewards i.e. moving a piece is 0). One thing that’s useful to know especially in games is that you don’t need action-value functions. In deterministic MDPs (games), it’s actually just sufficient to have a state value function and use that instead.</p><p>Recently, computer scientists have been taking a radically different approach to game-tree search that has shown to be extremely effective in games + real-world domains known as <strong>simulation-based search.</strong> The idea is quite simple; self-play RL can actually replace search and we can simulate games of self-play from our root state Sₜ. We simulate experience that starts from now i.e. Sₜ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/0*c_GDt9ZMOk64wnj5" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>MC Tree-Search has actually become one of the best performing methods in challenging games such as Go. But many games can be solved with just a simple MC search where you have randomized rollouts to the end of the game.</p><p>Let’s look at this in scrabble:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/0*ehlj2Kbe1gtdOQl7" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><p>The way MCS would learn is by playing out the games and choosing what gives us the highest score. How did it go? It beat the world champion with such a simple policy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EX3y8TFkkxS_15SA" /><figcaption><a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/games.pdf">Source</a></figcaption></figure><h3>What’s next?</h3><p>Yay! Let’s assume that you’ve read through this whole article + went through Prof. Silver’s lectures. Now how do you really start building and implementing what you’ve really learned?</p><p>I would really recommend going through some of <a href="https://spinningup.openai.com/en/latest/index.html">OpenAI’s Spinning Up </a>documentation for people who are interested in Deep Reinforcement Learning. There’s a lot of valuable content surrounding that field + you get to code our several Deep RL algorithms!</p><p>This YouTube course is also a good resource that I’ve come across to get a good theoretical understanding of Deep Q Learning + Policy Gradients:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FELE2_Mftqoc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DELE2_Mftqoc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FELE2_Mftqoc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5be9b443a44b8ea09f040b5878ae5e8b/href">https://medium.com/media/5be9b443a44b8ea09f040b5878ae5e8b/href</a></iframe><p>Another course by freecodecamp that I’d really recommend is to go through + understand how you can implement RL papers. It’s a great resource to go through and implementing algorithms + papers is something that’s really valuable to do.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FGJJc1t0rtSU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGJJc1t0rtSU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FGJJc1t0rtSU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c26f9b1bd1823bcf1413d6f00e03fad/href">https://medium.com/media/0c26f9b1bd1823bcf1413d6f00e03fad/href</a></iframe><p>Hey 👋 thanks so much for reading through the article! Hope you learned a thing or two about RL and got some sort of direction with it! My goal is to help as many people as I can and with that being said, I’d really appreciate if you could share with others and help them better understand Reinforcement Learning as they explore what they might be interested in!</p><h3>Contact me</h3><ul><li>Connect with me on <a href="https://www.linkedin.com/in/srianumakonda/">LinkedIn</a></li><li>Follow me on <a href="https://twitter.com/intent/follow?original_referer=http%3A%2F%2Fsrianumakonda.com%2F&amp;ref_src=twsrc%5Etfw%7Ctwcamp%5Ebuttonembed%7Ctwterm%5Efollow%7Ctwgr%5Esrianumakonda&amp;region=follow_link&amp;screen_name=srianumakonda">Twitter</a></li><li>Stay on the loop with what I’m building on <a href="https://srianumakonda.substack.com/">Substack</a>!</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cd5a365e340" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2021 Annual Letter — Sri Anumakonda]]></title>
            <link>https://srianumakonda.medium.com/2021-annual-letter-sri-anumakonda-6c9886967fb?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c9886967fb</guid>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Fri, 31 Dec 2021 21:45:12 GMT</pubDate>
            <atom:updated>2022-01-01T19:22:52.627Z</atom:updated>
            <content:encoded><![CDATA[<h3>2021 Annual Letter — Sri Anumakonda</h3><p><strong>My goal with this letter is to show how I’ve spent the past 365 days understanding how I can level myself up, and executing on those ideas.</strong> I’ve had &gt;75 mentor calls and met some of the most inspiring people on the planet who’ve not only shifted my trajectory, but showed me that there’s more than one lens you can view the world in (s/o to Shreyas for showing me the best ways to maximize growth and for showing me what it means to create legitimate projects).</p><p>I’ve failed more this year than the other 14 years combined from building DataGAN to networking to even my sleep. And yet those failures were the ones that taught me more about myself than any experience I ever could have.</p><p>Through failure comes learning. Through failure comes understanding. Through failure comes appreciation. Through failure comes excitement towards my future.</p><p>This year has not only been the most rewarding one, but also the most enjoyable year I’ve lived so far.</p><p><strong>This letter will mainly be centred around these 3 things</strong>:</p><ul><li>an overall summary + highlights of what I’ve done this year</li><li>my failures</li><li>my game plan to make 2022 the best year yet</li></ul><h3>Looking through the eyes of a camera; seeing the world as a self-driving car</h3><p>01/10/2021 — I created a presentation explaining my progress with autonomous vehicles (AVs) and computer vision, something quite foreign but also something I really wanted to do.</p><p>So I built something: <strong>using semantic segmentation to predict lane lines.</strong> I remember crying at 12am on the day of the presentation after spending the past 6 hours restarting the project. I spent 2 weeks building it all up but had to restart because I couldn’t figure out why my model wasn’t learning. And now the new model still wasn’t learning properly. Fast forward and I eventually got decent results after overfitting. But the presentation itself? It was mediocre.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMivZG2pAXNc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMivZG2pAXNc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMivZG2pAXNc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ec2b4282c4ce904aa69100c55e80c22/href">https://medium.com/media/6ec2b4282c4ce904aa69100c55e80c22/href</a></iframe><p><strong>Failure hits fast. Failure hits hard. The more important thing though was that I restarted the project from scratch and was able to completely rebuild it.</strong></p><ul><li><a href="https://medium.datadriveninvestor.com/using-u-net-in-lane-detection-343bc7b08c2b">Using U-Net in Lane Detection</a></li><li><a href="https://github.com/srianumakonda/Lane-Detection-in-CARLA">GitHub - srianumakonda/Lane-Detection-in-CARLA</a></li></ul><p>The art of hustling through and pushing yourself to break that wall that’s stopping you from moving forward is truly beautiful. I learned so much about <a href="https://arxiv.org/abs/1505.04597">U-Net</a>, implementing segmentation networks, and just creating neat code stored across multiple .py files.</p><p><strong>Perseverance gives you the keys to the doors that you thought were always locked.</strong></p><h3>Becoming one of the youngest self-driving engineers IN THE WORLD</h3><p>In February now, filled with so much momentum after not only rebuilding the lane detection project but also meeting <a href="https://davidsilver.blog/about/">David</a>, one of the smartest people that I’ve ever met in self-driving. I didn’t know what was next until he told me that <strong>Udacity runs the biggest certified self-driving program in the world.</strong></p><p>The first thing I did coming off that call was making my Udacity account and signing up for the nanodegree. This ended up not being possible because of unforeseen circumstances. But I knew this was the thing that I wanted to do. I <strong>ended up taking a massive leap of faith and asking David, right on the call if there’s any way I could do the nanodegree via a scholarship.</strong></p><p>After 15+ emails back and forth and talking with <a href="https://www.linkedin.com/in/pwdono/">Patrick</a> [shoutout to you personally Patrick! super grateful for the effort you’ve put into this], <strong>I was able to do the nanodegree, start to finish, in 12 weeks.</strong> I spent &gt;5 hours every single day not only understanding the math behind algorithms but also debugging code and writing reflection pieces.</p><p>Where did that lead me?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eKJoN48OshhOQpEZ" /></figure><p>One of the proudest achievements that I’ve received so far. It was surreal to not only learn so much within a matter of a few weeks but also to the people that helped me and allowed me to pursue something that I’ve wanted to do for so long.</p><p>What else happened? <strong>I had the chance to meet </strong><a href="https://en.wikipedia.org/wiki/Sebastian_Thrun"><strong>Sebastian Thrun</strong></a><strong> himself!</strong> It was a dream come true. I’ve looked up to Sebastian for the past 12 months with the pioneering work he’s done with DARPA, but also his willingness to play long-term games with the goal of making an impact.</p><p><strong>I’ve never felt more ecstatic coming out of a meeting ever, in my life.</strong> Just being able to spend 20 minutes with someone who has not only changed the world but has provided opportunities for millions to learn via <a href="https://medium.com/u/1022fa5dee6d">Stanford University</a> and <a href="https://medium.com/u/2929690a28fb">Udacity</a> was amazing.</p><p>I also was featured on Udacity for the work I’ve been doing:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5JTAta6ZIb8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5JTAta6ZIb8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5JTAta6ZIb8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1693bb18f8fb44a1d0a759e30d1a45f/href">https://medium.com/media/f1693bb18f8fb44a1d0a759e30d1a45f/href</a></iframe><p><a href="https://www.udacity.com/blog/2021/04/the-autonomous-tech-dreams-of-the-14-year-old-sri-anumakonda-student-story.html">The Autonomous Tech Dreams of the 14-year-old Sri Anumakonda | Student Story</a></p><h3>And then came summer…</h3><p>I’ve gotten so much momentum. Not only finishing the nanodegree and meeting some remarkable people like <a href="https://www.mit.edu/~amini/">Alexander</a> has gotten me super hyped for the summer. I remember thinking about what I wanted to do and I eventually <strong>broke this down into 2 main things: fully understand high school math, and build a research project.</strong></p><p>I haven’t read many papers in the first half of 2021. Why? Because I didn’t understand that many of them. So I decided to reimplement a paper. What would I end up reimplementing? Lane Detection. How? By <strong>leveraging LSTM networks.</strong> I remember reading this paper on <a href="https://arxiv.org/abs/1903.02193">Robust Lane Detection from Continuous Driving Scenes</a> and I knew it was something I wanted to do, especially how well it intertwines with the lane detection project I built earlier this year.</p><p>I spent 2 months implementing the paper. Doing this in private repo’s, completely doing this in stealth. Yes, I had some really neat-looking code that was not only useable but also showed a successful implementation of their paper. <strong>The only problem? No one knew. Nobody even knew what I was building throughout all of summer and I made the assumption that people did.</strong></p><p><strong>Building in public is the most valuable skill that I’ve learned in 2021.</strong> Showcasing progress creates an online presence that topples the domino effect for serendipity.</p><p>The other mistake that I made was not getting an internship. I’ve spent the majority of 2021 in research and understanding the gap between academia and industry. <strong>To build in the industry, you should be in the industry to understand + find problems in it.</strong> Understanding what sorts of algorithms are used in developing self-driving cars would’ve allowed me to build more meaningful projects that can genuinely be used.</p><blockquote>I’m currently looking for summer ’22 internships in Computer Vision, robotics, or in self-driving! Shoot me an <a href="/service/mailto:sri.anumakonda06@gmail.com">email</a> if you’re intersested.</blockquote><p>I’ve gotten so many slumps throughout the past year because of online school causing me to feel exhausted. The root cause? Transitioning between school and other parts of my day has been difficult. I’ve concluded that the majority of times where I’ve felt tired come from times when I’m trying to switch from one task to another.</p><p>Working out has allowed me to ease that transition drastically. I sometimes like going on a quick 15min walk to just do reflections and post-mortem examinations on the work I’m building and the session that I’m coming out of.</p><p>I also moved houses over the summer. It doesn’t sound like much of a big deal but finally being able to have my own, proper room, with books, a bed, a whiteboard, is amazing. Here’s a pic of a sunrise drive that I took!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XWH2x8OTdzIecn3_7Nq6sA.jpeg" /></figure><h3>Understanding next-gen self-driving tech w/ Waabi!</h3><p><a href="https://www.linkedin.com/in/tawsif-kamal/">Tawsif</a> told me about <a href="http://www.cs.toronto.edu/~urtasun/">Prof. Urtasun</a> and her self-driving startup that came out of stealth. I was literally amazed. I just stared at their YouTube videos and printed &gt; 70 pages worth of papers to read. <a href="https://arxiv.org/abs/2101.06543">GeoSim</a> and <a href="https://arxiv.org/abs/2101.06557">TrafficSim</a> are examples of some of the most advanced simulation research I’ve ever seen with self-driving cars.</p><p>I sent cold emails to the authors of these papers and even had the chance to meet them! I not only had the chance to learn from the smartest people I’ve ever met in my life (h/t to <a href="https://www.alexcui.ca/">Alex</a> + <a href="https://siegedog.com/">Andrei</a>!) but also was able to implement their advice and see month-by-month growth:</p><h3>Sri Anumakonda on Twitter: &quot;I had the chance to meet some amazing people from @Waabi_ai for the past little bit! Insights from meeting @alexcdot @andreib @thekelvinwong @sivamanivasagam (https://t.co/Jzv6A6ATG7): / Twitter&quot;</h3><p>I had the chance to meet some amazing people from @Waabi_ai for the past little bit! Insights from meeting @alexcdot @andreib @thekelvinwong @sivamanivasagam (https://t.co/Jzv6A6ATG7):</p><h3>Sri Anumakonda on Twitter: &quot;@scale_AI was probably one of the best conferences that I was able to attend this month. Had some time the past few days to break down @Waabi_ai @RaquelUrtasun&#39;s approach on &quot;Solving the Last Mile for Self-Driving&quot; (a thread): / Twitter&quot;</h3><p>@scale_AI was probably one of the best conferences that I was able to attend this month. Had some time the past few days to break down @Waabi_ai @RaquelUrtasun&#39;s approach on &quot;Solving the Last Mile for Self-Driving&quot; (a thread):</p><p>Waabi would undoubtedly be one of the places that would be a dream come true for me to intern at. <strong>I believe that an AI-first approach will not only allow for robust modelling + deployment but being able to learn from the smartest people in self-driving would be a dream come true.</strong></p><h3>Creating next-gen ways to train AVs w/ DataGAN</h3><p>I started building DataGAN after my chat with Andrei in September. I personally found GANs to be not only something that interested me but also was a whole realm of Computer Vision that I had yet to explore.</p><p>And what’s the best way to understand GANs? BUILD THEM! I started DataGAN: Leveraging Synthetic Data for Self-Driving Vehicles as a way to create synthetic data that can be used to train robust neural agents.</p><p><a href="https://srianumakonda.medium.com/datagan-leveraging-synthetic-data-for-self-driving-vehicles-6e629968a567">DataGAN: Leveraging Synthetic Data for Self-Driving Vehicles</a></p><p>Whenever I generally build projects, <strong>I love building in the intersection of awesome tech and in a field that I don’t know much of.</strong> For the project back in summer with LSTMs (ConvLSTMs) in lane detection, I didn’t know much about spatiotemporal scene understanding. With DataGAN, I knew nothing about GANs. <strong>So I spent the first 3 weeks understanding them and the next 5 weeks building this out and getting validation.</strong> The result?</p><h3>Sri Anumakonda on Twitter: &quot;DataGAN: Leveraging Synthetic Data for Self-Driving Vehicles (a thread)Article: https://t.co/e7v0cFlU8O / Twitter&quot;</h3><p>DataGAN: Leveraging Synthetic Data for Self-Driving Vehicles (a thread)Article: https://t.co/e7v0cFlU8O</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/0*tZlXhCmbLPGhxMxh.gif" /></figure><p>I even had the chance to meet <a href="https://seung-kim.github.io/seungkim/">Seung</a>, who’s built some of the <a href="https://nv-tlabs.github.io/DriveGAN/">COOLEST work</a> I’ve ever seen with GANs in my life. He’s awesome, we had an amazing chat and being able to jam out about GANs in self-driving was awesome.</p><p>I eventually decided to drop the project. I’ve spent &gt;250 hours planning this project out, understanding GANs + the math behind them, and yet I decided to drop everything. Why? GANs are notoriously hard to train. I’ve sent &gt;150 emails regarding compute issues to labs and research institutes in Toronto but I didn’t even get a single response. <strong>I was stagnant, for 2 whole weeks.</strong></p><p><strong>It doesn’t make any sense to sit around and expect serendipity to show up before you.</strong> Serendipity is not a genie stored in a lamp; you cannot expect good things to happen if you don’t put in the work. I liked the idea of going on sprints like DataGAN where I’d level up my understanding in a foreign field. So I decided to build in a space that I’ve always wanted to understand: end2end learning.</p><h3>Hopping on the Wayve of end2end learning</h3><p>2 reasons why I wanted to get into end2end learning. 1) I thought it was fascinating how neural networks can map from image → steering, and 2) I had a meeting with <a href="https://www.linkedin.com/in/haiboe/">Haibo</a> in a few days.</p><p>Meeting Haibo and learning about Wayve [h/t <a href="https://www.linkedin.com/in/satvik-agnihotri/">Satvik</a>!] was amazing. I woke up at 5:30AM to have our meeting (which was at 6:00) and looking back at it, <strong>getting 45 minutes of her time to jam out on end2end learning was amazing.</strong> Definitely one of the best meetings that I’ve personally had.</p><p>I came out of that meeting knowing for a fact that I wanted to learn more about e2e systems. <strong>I’ve printed, annotated, and handwritten notes for &gt;250 pages worth of papers within the space of 4 weeks and started prototyping + building end2end models.</strong></p><p>Interpreting neural networks is one of the hardest problems that there is right now. It’s crucial to be able to understand these networks and the weights activations to better understand the “rationale” for a certain prediction made. I was specifically looking into seeing how we can use CNNs [and ConvLSTMs] to better understand + extract spatiotemporal context given images, along with using autoencoders to predict the next few frames in a driving scene given the predicted speed + steering (n=1 is what’s used in the diagram). <a href="https://www.notion.so/End2End-Self-Driving-a3e063810d164ea982a4157b0fc8e403"><strong>Proposal can be viewed here</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sytBJ8JrevsoiKy_" /></figure><h3>Sri Anumakonda on Twitter: &quot;End2End learning is super exciting. I showed a little bit of what I&#39;m working with steering a little while ago.My main goal right now is to build a fully end2end model, capable of predicting steering, brake, and acceleration given monocular camera inputs. / Twitter&quot;</h3><p>End2End learning is super exciting. I showed a little bit of what I&#39;m working with steering a little while ago.My main goal right now is to build a fully end2end model, capable of predicting steering, brake, and acceleration given monocular camera inputs.</p><p>I also got &gt;1.35k in AWS credits to build this project out which ultimately kickstarted the “paper instinct” and now I wanted to turn this idea into a large-scale project that I could test, validate, and write a paper on.</p><p>I spent 2 weeks playing around with the <a href="https://github.com/commaai/research">comma dataset</a> + <a href="https://github.com/udacity/self-driving-car">udacity</a> to see if I can really train a network for steering, speed, and next-frame prediction. It didn’t make sense for me to go that far for the sake of the paper. <strong>I realize now that even small-scale testing provides nearly the same benefits as building large-scale projects.</strong> I wanted to build a large-scale project so that I can show people that I wrote a paper and made a cool video + animations. But that didn’t fundamentally add value to the actual idea itself.</p><p>Just as in the case of DataGAN, by not creating accurate + quantifiable metrics, I wasn’t able to make proper progress and eventually decided it was best to take a break from this.</p><h3>Dropping 2 projects in 2 months</h3><p><strong>There are 2 reasons why I dropped these projects:</strong></p><ol><li>Assumptions that I’ve made</li><li>The value prop from building this</li></ol><p>The first problem that I had (with both projects) was the fact that I made assumptions. <strong>I spent the whole time assuming that a professor or grad student would magically respond to my email</strong>, let me borrow a PC with a GPU for research purposes (or even work at a lab!), and everything would be perfect. I thought that I’d be able to write a paper on DataGAN and the end2end idea that I’ve been playing around with. <strong>I looked at both of my projects from an ideal standpoint, not something that was even realistic.</strong></p><p>This has been my biggest regret of 2021. <strong>Not realizing that making hypothetical assumptions for projects that are completely outside of my scope won’t bring any sort of progress.</strong> Instead, explore! Do sprints and deep dives of different fields in Computer Vision.</p><p>I stopped DataGAN because I realized that it didn’t make sense for me to continue pursuing it. I wasted so much time sitting back and waiting for the emails to pop in. <strong>What’s the value in that when I can start learning calculus, </strong><a href="https://twitter.com/srianumakonda/status/1469075614233767939"><strong>learning data structures</strong></a><strong>, and even </strong><a href="https://github.com/srianumakonda/DMOJ-Solutions"><strong>doing programming challenges</strong></a><strong>?</strong></p><p>end2end learning is a bit different. <strong>This is a goal that I want to pursue long-term.</strong> I want to actually build end2end systems. I think end2end might be the way we truly achieve level 5 driving. I wanted to understand end2end systems and then build a paper.</p><h3>Sri Anumakonda on Twitter: &quot;I&#39;ve been focusing on understanding Data Structures + algorithms w/ Grokking algorithms! @_egonschieleSummarized my notes into a website + medium article for people to read + understand!Website: https://t.co/W1wQDgfbcLhttps://t.co/YjODWs4Nja / Twitter&quot;</h3><p>I&#39;ve been focusing on understanding Data Structures + algorithms w/ Grokking algorithms! @_egonschieleSummarized my notes into a website + medium article for people to read + understand!Website: https://t.co/W1wQDgfbcLhttps://t.co/YjODWs4Nja</p><p>end2end learning is a bit different. <strong>This is a goal that I want to pursue long-term.</strong> I want to actually build end2end systems. The 4-week “sprint” that I thought of doing was to understand end2end learning systems and then build a paper (here’s the proposal). I dropped the proposal because again, it didn’t make sense + I didn’t have the compute to train dense neural networks.</p><p><strong>I’m starting to build in this closed-loop system of having idea sprints + building small-scale projects.</strong> This is probably the best way for me to not only get technical understanding but also be able to show legitimate progress.</p><h3>Failing hard</h3><p><strong>The biggest thing that I’ve done this year is failing. Not just in the projects that I’ve built, but also in the way I chose to grow as a person. </strong>I’ve lost momentum several times, but that’s allowed me to not only understand myself better but also allow me to understand the goals and impact that I want to make on this world.</p><p>Lane Detection — one of the first failures that I’ve gotten in 2021. <strong>I’ve failed by not allocating enough time</strong> to build this project enough, optimized for creating “professional” github repos instead of focusing on getting the model working, and also failed in not choosing to reach out to mentors + people who might’ve been willing to help me out. The main takeaway here was that code does not always work; understanding + learning to debug problems is a crucial aspect in developing intuition behind the code you write. I’ve allowed myself ever since to explore + create neat code that people can easily reimplement and also just focus on understanding the goal. I didn’t fully understand U-Net before I dived into it, and as a result, I wasn’t able to accurately debug it in time.</p><p>The biggest thing I’ve learned over the summer though is <strong>how important it is to set clear intentions, KPIs, and how to really break down your time in chunks that allow you to explicitly see week-by-week growth.</strong> I’ve failed in that aspect completely. I woke up every single day without clear intentions of what I wanted to achieve and just spent 10 hours typing up as much code as I can and expected it to work. I’ve become so much more productive when I have clear-cut intentions set out for the day.</p><p>I’ve also failed a lot in DataGAN. <strong>I set unrealistic expectations</strong> without a defined path that would get me there. <strong>I made claims</strong> such as making a paper and doing large-scale testing, but I never sat down and brainstormed how I’d achieve what I wanted. <strong>I expected the puzzle pieces to fit themselves together</strong> without realizing that I’m the person that brings the pieces together. I need to complete the puzzle, it doesn’t happen itself.</p><p><strong>I failed in over-optimizing</strong> for computational power because I wanted to build large-scale projects. <strong>I wasted so much time waiting for an email</strong> when I could’ve been making so much more progress instead by creating a small-scale implementation and then moving on. <strong>I failed in assuming that the research paper is the end goal</strong> and serendipity stems from that outcome.</p><p>I’ve failed in taking opportunities for granted. I’ve failed in being passive. I’ve failed in understanding the world. I want to do better. I will do better. 2021 has shown me what my weakness really are. I want to strengthen my weaknesses and be able to learn from them.</p><h3>How I can make 2022 the best year</h3><p>I’ve learned a lot. I’ve failed a lot. I want to make next year the best year I could ever possibly live in. <strong>There are 3 main buckets that I want to optimize for 2022:</strong></p><ol><li>Become a self-driving “god”</li><li>Get an internship</li><li>personal health</li></ol><p>The first thing that I’m going to be focusing on is really developing my T. I’ve done that a lot this year. I learned so much about AI, Computer Vision, and even built my depth of expertise with the projects that I’ve been building. But I want to take it further. <strong>I want to start reading textbooks and understand so much more about self-driving + the fundamental principles of the components of self-driving</strong> [specifically looking into <a href="https://joshua.smcvt.edu/linearalgebra/book.pdf">linear algebra</a>, the <a href="https://mml-book.github.io/book/mml-book.pdf">mathematics of ML</a>, and <a href="http://lavalle.pl/planning/">planning algorithms for AVs</a>]<strong>.</strong></p><p>I want to research, build, and showcase projects in these fields:</p><ul><li>motion planning</li><li>reinforcement learning + policy development</li><li>creating variational autoencoders for image-to-image translation</li><li>end2end learning + conditional imitation learning</li></ul><p>There’s so much to learn, despite the fact that I’ve scoped down from AI → self-driving → computer vision. I’ve been super passionate about computer vision and yet there’s still so much to learn and understand. How do things like optical motion flow impact the perception of a self-driving car? Can we simply use Q-Learning with CNNs to train robust agents?</p><p>The main focus of all of this <strong>will be to understand fundamental concepts and the principles of mathematics and programming.</strong> This will be the most compounding skill that I’ll have right now. <strong>Understanding math is a superpower. </strong>Being able to understand the fundamental understanding of an algorithm, concept, and paper will not only allow me to comprehend lower-level details i.e. optimization algorithms + custom loss functions, but also will allow me to question the math and understand why it works.</p><p>Right now, I’m reviewing calculus + differentiation. I’m also thinking about reading “<a href="https://mml-book.github.io/book/mml-book.pdf">Mathematics for Machine Learning</a>” to become more fluent with linear algebra + applying the math in various ML algorithms. I’m going to be focusing on playing long-term games i.e. learning math. This is the best way for me to really make an impact in the summer and will bring my technical understanding to a whole new level.</p><p>The other focus of 2022 will be to get an internship where I can build next-gen self-driving tech. <strong>Getting the chance to learn from some of the smartest people in the industry will not only allow me to better understand self-driving as a whole, but it’ll really push my technical understanding.</strong> I specifically want to work in the end2end learning space, looking into the interpretability of neural networks and even build end2end systems myself!</p><p>Some of the companies on my radar right now are, without question, <a href="http://waabi.ai/">Waabi</a> + <a href="https://wayve.ai/">Wayve</a>. Both of them have been pioneering some really cool AI-first research. I’d love to play around with cameras and understand how we can use VAEs and spatiotemporal networks to allow for robust performance when driving.</p><p>Another possible route would be to get into is a research lab here in Toronto ex. <a href="https://vectorinstitute.ai/">Vector</a>! Some of the smartest people in self-driving are usually found in research labs (we have a lot of smart people in Toronto!) and being able to network + work with them on interesting research projects would be great.</p><p>The third thing that I really want to double down on for 2021 will be personal health. Something that I’ve neglected for so long. I want to focus on not only sleep + working out, but also nutrition.</p><p>It’s not just about physical changes, but mental too. Even going on 15min walks is a huge refresher for transitions throughout the day. And the best thing is? You feel happy. You feel grateful for where you are and you feel motivated to strive to do more.</p><p>The main KPIs I’m currently looking at are upkeeping proper sleep, developing close relationships with the people who I talk to, and maintaining physical health. I’ll be focusing on strength + cardio training along with sports!</p><p>This is what I want 2022 to be all about. <strong>It’s not just about the number of hours that you put in working, but it’s about how well you perform in those hours.</strong></p><p><strong>Words cannot describe how grateful I really am for the growth I’ve had over the past 12 months.</strong> Want to personally shout out <a href="https://en.wikipedia.org/wiki/Sebastian_Thrun">Sebastian</a>, <a href="https://www.shreyaskousik.com/">Shreyas</a>, <a href="https://www.linkedin.com/in/rahul-shambhuni-9247a120/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAARjSewBP8fWotjnmpRundCJ2c8er069RMs">Rahul</a>, <a href="https://siegedog.com/">Andrei</a>, <a href="https://www.linkedin.com/in/alex-cui-caltech/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAA_HL94BkBJhaoLlTWBrIBJgEwBjzQsOlAA">Alex</a>, <a href="https://www.linkedin.com/in/haiboe/">Haibo</a>, <a href="https://www.linkedin.com/in/dsilver829/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAAW_McBnFSaMDCCae-8pDOooVJCjkQ1RYw">David</a>, <a href="https://www.linkedin.com/in/olivercameron/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAQE5TwBsHjBWodcsS1hrgs3xqwJaJv7Iog">Oliver</a>, <a href="https://www.linkedin.com/in/navidnathoo/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAa6JeIBbGa58WG8GhuNVi889QmjIOyxI_E">Navid</a>, <a href="https://www.linkedin.com/in/michael-raspuzzi-b3409194/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAABPx5hgBnGaLVPlxRAQNNRxBVAGPkScuGsA">Michael</a>, <a href="https://twitter.com/_harrisonnolan">Harrison</a>, <a href="https://www.linkedin.com/in/kim-parnell/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAQuJ7wBcNGFhi9caeJ6l1p80k5MTfFvhhY">Kim</a>, <a href="https://www.linkedin.com/in/danjacob1/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAD-7aUBghhK7YQYJGN_sr6Op0737OCvXZw">Dan</a>, and <a href="https://www.linkedin.com/in/nadeemnathoo/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAMHF4oBDrZzIeSvhjPCAD91D-xjOoJmrx0">Nadeem</a> for the mentorship + guidance you’ve provided me with. The amount of time you’ve spent helping me level up myself is something that not only will I be eternally grateful for but also the mentorship and the endless experiences you’ve provided me with.</p><p>Also shoutout to <a href="https://www.linkedin.com/in/mikael-haji-a324b41b4/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADHoS8YBwHkWKYpeIVQsMGHoF4y2PwYAsFo">Mikael</a>, <a href="https://www.linkedin.com/in/zaynpatel/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAACgyc-cBXe3JJPleOtMH7CRcMf9M2ncydPA">Zayn</a>, <a href="https://www.linkedin.com/in/aahaanmaini/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADHy0RwBkE7MF-ZxlvBaekvnX4jQt6MrUH4">Aaahan</a>, <a href="https://www.linkedin.com/in/dev-patel-25456219a/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAC62UOABASJ4Lc3_DHoWePeKICL_WavlErQ">Dev</a>, <a href="https://www.linkedin.com/in/arnav-shah-937287157/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAACWYBHsBiU-Uq73FqWKlREO_1QL933tfMLk">Arnav</a>, <a href="https://www.linkedin.com/in/anushmutyala/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADJlkgEB_xmrHnYcEMw88JJuUhKbi6CEgJM">Anush</a>, <a href="https://www.linkedin.com/in/kabeer-makkar/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADJtx0IBDSGci8X1u9nyWoIMv9e-T5iGuNU">Kabeer</a>, <a href="https://www.linkedin.com/in/pavan-jayasinha-6b06b71b6/?originalSubdomain=ca">Pavan</a>, <a href="https://www.linkedin.com/in/sriya-chintalapalli/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADEhJNQBfB3RDogUKZl6UTfc5uW0SvrWKtw">Sriya</a>, and <a href="https://www.linkedin.com/in/ahmed-hassan-5826411b0/?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAADE3VfIB4TW_XDIUVOBS--Qv_vq32_SeX0U">Ahmed</a> for not only allowing me to accelerate on my trajectory but also for the insightful conversations that we’ve had all this time. The successes [and failures!] that I’ve gotten this year wouldn’t have come without the calls that we’ve had. I’ve learned a ton about the projects you’ve all been working on from fUS to computational biology and it’s just been simply awesome nerding out with you guys!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*U4zYDEMZvYrj-CI1tZAsQQ.gif" /></figure><h3>Concluding thoughts</h3><p>Being able to have a room to myself, a computer to type this letter on, and fingers to type has allowed me to do things that not many 15yo really think about. I’ve had countless opportunities that people my age would only be able to dream about.</p><p>Even typing this letter out has allowed me to reflect on my year and better understand myself a lot better than ever before. I’ve learned, failed, and grown in so many aspects. I’ve met some of the smartest people that I’ve ever met in my life.</p><p>I want to crush 2022. I want to optimize to make it the best year ever. I’ve gained so much momentum + trajectory over the past 12 months and I want to keep accelerating. <strong>Why be linear when you can be exponential?</strong></p><blockquote>Hey! 👋 If you’d like to follow along with the progress that I’ll be making in 2022, <a href="https://srianumakonda.substack.com/p/coming-soon">here’s my newsletter</a>! <a href="https://twitter.com/srianumakonda">Twitter</a> too. You can also reach out to me on my <a href="https://www.linkedin.com/in/srianumakonda/">LinkedIn</a> + <a href="/service/mailto:sri.anumakonda06@gmail.com">email</a>, free say to drop in and say hi! Oh, and <a href="http://srianumakonda.com/">personal website too</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c9886967fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Data Structures with Grokking Algorithms!]]></title>
            <link>https://srianumakonda.medium.com/understanding-data-structures-with-grokking-algorithms-57005ce03a0?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/57005ce03a0</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Thu, 09 Dec 2021 22:29:02 GMT</pubDate>
            <atom:updated>2023-08-01T20:01:53.411Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Hey! I’ve been reading up on Data Structures + Algorithms as a way to be a lot more competent + technical with the work I’ve been doing! This article summarized my notes in an easy-to-understand guide for people to follow. <a href="https://grokking-documentation.readthedocs.io/en/latest/"><strong>The main website of my notes have also been published</strong></a> with the <a href="https://github.com/srianumakonda/Grokking-Documentation"><strong>Github repository</strong></a>. <strong>Note:</strong> the digram/italic text will usually be from the textbook itself. I do not take any credit for the amazing illustrations provided!. Have fun reading!</blockquote><h3>Binary search</h3><p><strong>The search problem:</strong> <em>to find element </em><strong><em>x</em></strong><em> in given list </em><strong><em>i</em></strong><em>, should </em><strong><em>i</em></strong><em> start from the beginning or start near the middle and split each step up?</em></p><p>This is exactly what binary search is. Its input is a <strong>sort list of elements</strong> with the output being the <strong>index of where that element is located</strong>.</p><p>Simple (stupid) search is essentially when we go through every single index in the list up until we find the desired element.</p><blockquote>“If my number was 99, it could take you 99 guesses to get there!”</blockquote><p>The better technique for counting? <strong>Start with 50.</strong> If that number’s too low, then you just eliminated 50% of all the numbers!</p><p>Your next range is between 50–100. What’s the middle of that? 75. <strong>Too high? Then cut again?</strong></p><p>63? <strong>57? YES!</strong></p><p>You just guessed the number within 7 steps. Notice how you <strong>eliminated half the numbers every time with binary search.</strong></p><p>For any list of <strong><em>n,</em></strong> binary search will take <strong><em>log_2_n</em></strong> steps to run in the worst case, whereas simple search will take <strong><em>n</em></strong> steps.</p><pre>def binary_search(list, item):<br>	low = 0<br>	high = len(list)-1<br>	while low &lt;= high:<br>		mid = int((low + high)/2)<br>		guess = list[mid]<br>		if guess == item:<br>		  return mid<br>		if guess &gt; item:<br>		  high = mid - 1<br>		else:<br>		  low = mid + 1<br>	return None<br>my_list = [1, 3, 5, 7, 9]<br>print(binary_search(my_list, 3)) # =&gt; 1<br>print(binary_search(my_list, -1)) # =&gt; None</pre><h3>Running times</h3><p>Whenever the maximum amount of guesses is equal to the size of the list, we generally define this to be <strong>linear time.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*1iDJkHc_PF0jQwfS" /></figure><p>Whenever we want to determine how fast an algorithm is, we generally use something known as <strong>Big O Notation.</strong></p><blockquote>That’s why it’s not enough to know how long an algorithm takes to run — you need to know how the running time increases as the list size increases.</blockquote><p>Big O Notation tells you how fast an algorithm is. For example, let’s say that you have a list of size <strong><em>n.</em></strong> The run time in Big O Notation is <strong><em>O(n).</em></strong></p><p>Notice that Big O Notation doesn’t ask for the speed in seconds. <strong>Big O notation lets you compare the number of operations. It tells you how fast the algorithm grows.</strong></p><p>What does that look like for Binary search? <strong>O(log_2_n)</strong></p><p><strong>Note:</strong> Big O Notation establishes a <strong>worst-case run time.</strong> The worst case is that you iterate through every element until you find the desired one, the worst case is that you have to apply binary search indefinitely until you find your desired element.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*PuDfegCGJqWoQOkuJ1k-cg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*sw3lX1UtH5MwYPMU" /></figure><h3>Lists + Arrays</h3><p>Think of your computer as a giant set of drawers with each drawer having an address.</p><p>Every time you want to store an item in memory, you generally will ask the computer for some space (drawers). It then returns an address where you can store your items.</p><p>The 2 ways we store multiple items right now are <strong>arrays and lists.</strong></p><p>Adding items every single time to an array is very difficult. Think of it like the movies: if you sit with your friends and a friend shows up [but there are no spots for them], you have to move everyone to a new row. And what happens if another one shows up? <strong>You have to move again!</strong></p><p>Adding an item to a computer is very slow. The easiest way we solve this is through “holding seats.” <strong>Even if you have only 3 seats, you can ask the computer for 10 slots, just in case.</strong></p><p>The main problem with doing something like this is mainly the fact that you have extra slots that you might’ve asked for but haven’t used. On the other hand, <strong>you might have more than 10 items and would have to move anyway.</strong></p><p>We can solve this through the principle of <strong>Linked Lists.</strong></p><p>Each item stores the address of the next item in the list. A bunch of random memory addresses are linked together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*v-0gWj2xRrbpKKQs" /></figure><p>Adding an item to this linked list is quite easy; <strong>you just simply stick it anywhere in the memory and store the address with the previous item. This allows you to never have to move your items</strong></p><p>If linked lists are better than arrays (above), what would the purpose of arrays even be?</p><p>The problem with linked lists is that “<strong><em>if you’re going to keep jumping around, linked lists are terrible. Why? If I wanted to get to the last item in a linked list, I have to go through EVERY SINGLE ITEM because that’s what gives me the address to get to the next item.”</em></strong></p><p>Arrays are different. You are given the address for every single item in the array.</p><p>In this case, you’d use an array when you want to <strong>read random elements because you can look up any element in the array.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*vmyq5vCGV52AwEXv" /></figure><p>Why does it take O(n) time to insert an element into an array?</p><h4><strong>Insertion</strong></h4><p>Generally, with lists, insertion is quite easy. You just simply change the address the previous element points to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_njilcd5KdSELbWd" /></figure><p>But in the case of an array, you need to shift all the elements down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*4AB2UgRTWeqok0Q6" /></figure><p><strong>Verdict: lists are generally better when it comes to inserting elements into the middle.</strong></p><h4>Deletion</h4><p><strong>Again, lists are better.</strong> You simply just change what the previous element points to while with arrays, everything needs to be moved up when you delete the element.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*-b_RH93nI4aW13AA" /></figure><h4>Arrays vs lists?</h4><p>Arrays are generally used more because they can allow for <strong>random access</strong>. We generally have 2 types of access: <strong>random access</strong> and <strong>sequential</strong> [one by one] <strong>access.</strong> Linked lists can only do sequential access while random allows you to jump directly to your desired element.</p><h3>Selection sort</h3><p>Let’s say that I’m sorting music</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*0pzIth4Ohd7OS4Xm" /></figure><p>I go from choosing the most played songs down to the least. I start with my original list and then append all of that (in ordered form) to this new list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*EaVAzIJZHnL1q0go" /></figure><p>As we do this over and over again, every single time, this overall process ends up taking <strong>O(n*<em>n) or O(n^</em>2) time. </strong>Selection sort is quite neat, but not that fast. Quicksort is a lot faster, it only takes <strong>O(n log n) time!</strong></p><pre>def findSmallest(arr):<br>	smallest = arr[0]<br>	smallest_index = 0<br>	for i in range(1, len(arr)):<br>		if arr[i] &lt; smallest:<br>			smallest = arr[i]<br>			smallest_index = i<br>	return smallest_index<br>def selectionSort(arr):<br>	newArr = []<br>	for i in range(len(arr)):<br>		smallest = findSmallest(arr)<br>		newArr.append(arr.pop(smallest))<br>	return newArr</pre><pre>&gt;&gt;&gt; print selectionSort([5, 3, 6, 2, 10])<br>[2, 3, 5, 6, 10]</pre><h3>Recursion</h3><p>It’s where a function calls itself.</p><pre>def look_for_key(box):<br>     for item in box:<br>	if item.is_a_box():<br>	    look_for_key(item)<br>	elif item.is_a_key():<br>	    print “found the key!”</pre><p>Recursion is generally used when it makes the existing solution a lot clearer and easier to understand.</p><p><strong>There is no performance benefit to using recursion!</strong> In fact, loops may sometimes be better for performance.</p><blockquote>“Loops may achieve a performance gain for your program. Recursion may achieve a performance gain for your programmer. Choose which is more important in your situation!”</blockquote><p>What’s the main problem with recursion? <strong>You might get stuck in an infinite loop.</strong></p><p>That’s why most recursive functions have a <strong>“recursive case”</strong> so that function knows when to stop calling itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/0*G7sKHNXJQe_XdpdK" /></figure><p>Computers generally use a stack known as the <strong>call stack.</strong> When you call on a function, your computer allocates a box of memory for that function call with “sub-boxes” for variable names.</p><p>Now, every time you make a function call, your computer saves the values for all the variables in that “box”.</p><blockquote><em>“W</em>hen you call a function from another function, the calling function is paused in a partially completed state.”</blockquote><p>The values of that function are stored in memory but they’re instead <strong>put on pause.</strong></p><p>Recursive functions can also use this call stack as well. A perfect example of this is in <strong>factorials.</strong></p><pre>def fact(x):<br>    if x == 1:<br>	return 1<br>    else:<br>	return x * fact(x-1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/0*clLCjwCwfEPUJvIa" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*HOZy5njPM60Yu6Ji" /></figure><p>What’s the main problem with saving everything on the call stack? <strong>It takes up A LOT of memory.</strong> When your stick is too tall, that means your computer is saving information for many function cells.</p><p>2 things you can do:</p><ol><li>Rewrite your code to use a loop instead</li><li>Use something known as <strong>tail recursion. (only supports some languages)</strong></li></ol><h3>Divide and Conquer</h3><p>Divide and Conquer is generally used in situations where you need to solve a problem that <strong>can’t be solved with the existing algorithms you’ve learned.</strong></p><p>It can also be known as a <strong>recursive technique</strong> used to solve these problems. Think of D&amp;C like this whole new framework used for solving problems.</p><p>Let’s say that we have this farm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*fWkg9btAg2XqGji_" /></figure><p>You cut the 1680x640 farm into 2 squares, each of length 640. <strong>Now your problem is that you need to split up the smaller segment (640x400).</strong></p><p>If you find the biggest box for that size (640x400), <strong>then that’ll be the biggest box that will work for the entire farm!</strong> Notice how you reduced the problem from a 1680x640 farm to a 640x400 farm.</p><p>And now, you simply apply the algorithm again and again until you find the perfect case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/0*8mMuHyaFSAaLMMZ2" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*hN7rq6Fn_uu6RAFO" /></figure><p>So for recap, D&amp;C can be broken down into 2 main steps:</p><ol><li>Figure out a simple case as the base case</li><li>Figure out how to reduce your problem and get to the base case.</li></ol><p><strong>Remember that D&amp;C isn’t a simple algorithm that you can apply to a problem. Instead, it’s a framework that you can use to think about a problem.</strong></p><p>Here’s array summing in recursion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*ffEShRYWRsSV7cDO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/0*DNFUyLQk_QUX0fxn" /></figure><p>This is known as <strong>functional programming</strong> (Haskell).</p><h3>Quicksort</h3><p>Quicksort is generally known as a <strong>sorting algorithm.</strong> It’s a lot <strong>faster than selection sort</strong> and is generally used. It uses a recursive algorithm structure for fast + proper sorting.</p><p>Here’s what the process looks like:</p><ol><li>Pick an element from the array. <strong>This is generally known as a pivot.</strong></li><li>From there, you <strong>bin the smaller + larger elements into 2 separate arrays. (</strong>This overall process is known as <strong>partitioning.)</strong></li><li>You then run this <strong>recursively until you get sorted arrays</strong> and then you can simply <strong>concatenate everything</strong> and you now have your output.</li></ol><pre>def quicksort(array):<br>  if len(array) &lt; 2:<br>    return array<br>  else:<br>    pivot = array[0]<br>    less = [i for i in array[1:] if i &lt;= pivot]<br>    greater = [i for i in array[1:] if i &gt; pivot]<br>    print(pivot)<br>    print(less)<br>    print(greater)<br>    print(&quot;-&quot;*20)<br>  return quicksort(less) + [pivot] + quicksort(greater)<br>print(quicksort([10, 5, 2, 3]))</pre><p>The average runtime generally for quicksort is <strong>O(n log n)</strong> but the worst case is generally <strong>O(n²) time.</strong></p><p>Generally remember that when we call something like <strong>O(n)</strong>, it’s generally more of like <strong>O(c*n)</strong> where <em>c</em> is a constant multiplied by n but we usually don’t look at that for large numbers.</p><p>But sometimes, <strong>the constant can make the difference.</strong> Quicksort generally has a smaller constant than merge sort. So if both of them are in O(n log n) time, then <strong>quicksort is faster.</strong></p><p>Quicksort is also faster in practice mainly because it meets the average case substantially more than the worst case.</p><p><strong>The pivot matters.</strong> You want to have both arrays be as small as they can. <strong>That’s why you’d always choose the middle [or a random] number.</strong></p><blockquote><em>“In this example, there are O(log n) levels (the technical way to say that is, “The height of the call stack is O(log n)”). And each level takes O(n) time. </em>The entire algorithm will take O(n) * O(log n) = O(n log n) time. This is the best-case scenario.”</blockquote><h3>Hash tables</h3><p>The goal with hash tables mainly is to be able to <strong>retrieve information about an item instantaneously.</strong> Something that’s running time is <strong>O(1)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*K32wKRA3y3eAG-Ad" /></figure><p>Hash functions are essentially when you <strong>input a string</strong> and are returned <strong>a number.</strong> Think of it as a way to map from strings to numbers.</p><p>Why does this work?</p><ol><li>Hash functions consistently <strong>map a name to the same index (value)</strong></li><li>Hash functions <strong>map different strings to different indexes.</strong></li><li>Hash functions <strong>know how big your array already is</strong> and can only return valid indexes.</li></ol><p>Everything, as described above, is known as a <strong>hash table.</strong></p><p>Other common names include <strong>hash maps, maps, dictionaries, and associative arrays.</strong></p><p>Common use cases generally include <strong>lookups, translation, correlation, preventing duplicate entries, and cache.</strong></p><p>Sometimes, what happens is that whenever we bin values together (alphabet words), we end up with 2 or more words that fall under the same index. <strong>This is known as collision.</strong> (2 keys have been assigned the same slot)</p><p>The simplest solution used right now is to use <strong>linked lists</strong>. But there’s a problem here; if your data is not evenly distributed i.e. only one node has the majority of lists, <strong>you’re going to slow down your table.</strong></p><p>What do you do? <strong>Create a proper hash function that maps the keys evenly throughout the hash.</strong></p><p>Hash tables run in <strong>O(1) time aka constant time</strong> (which means instantaneously)<strong>.</strong> Your worst case is O(n) time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/0*jTt5uy9bW5AOPBPW" /></figure><p>It’s important you don’t hit worst-case scenarios when working with hash tables. To do that, you need to <strong>avoid collisions.</strong> To do that, you need <strong>a low load factor and a good hash function.</strong></p><p>The load factor is essentially the <strong>Number of items in the hash table / total number of slots.</strong></p><p>Ex. if I had a hash table with length 5 and I’ve filled in 2 slots, then my load factor is: <strong>2/5 = 0.4</strong></p><p>When your load factor starts to grow, you need to add more slots to your hash table. This is generally known as <strong>resizing.</strong></p><p><strong>Quick note:</strong> lower load factors lead to fewer collisions + result in increased performance. A rule of thumb mentioned in the book is to resize whenever the <strong>load factor ≥ 0.7.</strong></p><p>Yes, averaged out, hash tables will still take only O(1) for resizing</p><p>It’s important you don’t hit worst-case scenarios when working with hash tables. To do that, you need to <strong>avoid collisions.</strong> To do that, you need <strong>a low load factor and a good hash function.</strong></p><p>The load factor is essentially the <strong>Number of items in hash table divided by the total number of slots.</strong></p><p>Ex. if I had a hash table with length 5 and I’ve filled in 2 slots, then my load factor is: <strong>2/5 = 0.4</strong></p><p>When your load factor starts to grow, you need to add more slots to your hash table. This is generally known as <strong>resizing.</strong></p><p><strong>Quick note:</strong> lower load factors lead to fewer collisions + results in increased performance. A rule of thumb mentioned in the book is to resize whenever the <strong>load factor ≥ 0.7.</strong></p><p>Yes, averaged out, hash tables will still take only O(1) for resizing.</p><h3>Graphs + Breadth-first search</h3><p>Breadth-first search allows you to find <strong>the shortest distance between 2 values.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*bM0B9BKB0lfTGZXx" /></figure><p><strong>Note:</strong> the problem indicated above is known as a <strong>shortest-path problem</strong> (how can I get from x to y in the fastest way?)<strong>.</strong> The algorithm that we use to solve this? This is <strong>breadth-first search.</strong></p><p>To figure out how we solve the above diagram in the fastest way, there are 2 main steps:</p><ol><li>Model the problem as a graph</li><li>Solve the problem using breadth-first search</li></ol><p>What even is a graph? A graph is a way to <strong>set connections.</strong> They’re made up of <strong>nodes</strong> and <strong>edges</strong>. A node can be <strong>directly connected</strong> to other nodes. These nodes that are connected are known as <strong>neighbours.</strong></p><p>The 2 questions breadth-first answers?</p><ol><li><strong>Is there a path</strong> from node A to node B?</li><li><strong>What is the shortest path</strong> from node A to node B?</li></ol><h4>Linear programming</h4><p>Linear programming can be used to maximize something given the time constraints.</p><p>Your goal could be: <strong><em>given 2 meters of fabric + cotton, what’s the maximum amount of shirts and pants that I can make to maximize profit?</em></strong></p><p>These aren’t direct answers but rather a <strong>framework for thinking.</strong></p><p>This is quite similar to graph algorithms (actually linear programming is the greater subset for graphs)</p><h4>Queues</h4><p>The way we’d organize through different hierarchies (levels) of nodes is through <strong>queues.</strong> They’re similar to stacks but you can only perform 2 operations: <strong>enqueue</strong> and <strong>dequeue.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/0*xm4NUz27WveUn4hc" /></figure><p>It works in <strong>FIFO structure;</strong> first in, first out. Notice how as the 4 is added, the 1 is automatically removed.</p><p>The main concern people face is, <em>How do you express relationships (nodes)?</em> We can do this through <strong>hash tables.</strong></p><p>For example, with the diagram below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/0*9P93Tz-UAM7Z-fRe" /></figure><pre>graph = {}<br>graph[“you”] = [“alice”, “bob”, “claire”]<br>graph[“bob”] = [“anuj”, “peggy”]<br>graph[“alice”] = [“peggy”]<br>graph[“claire”] = [“thom”, “jonny”]<br>graph[“anuj”] = []<br>graph[“peggy”] = []<br>graph[“thom”] = []<br>graph[“jonny”] = []</pre><p>We also have something known as <strong>directed graphs.</strong> This is essentially when one of the nodes doesn’t connect to anything else (Anuj, Peggy, Jonny, Thom).</p><p>So to recap, here’s what the overall implementation looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/0*eo-WGuCRYMjfGPQV" /></figure><pre># A use case of this for &quot;mango&quot; sellers<br><br>from collections import deque<br>search_queue = deque()<br>search_queue += graph[“you”]<br><br>def person_is_seller(name):<br>	return name[-1] == ‘m’<br><br>while search_queue:<br>	person = search_queue.popleft()<br>	if person_is_seller(person):<br>		print person + “ is a mango seller!”<br>		return True<br>	else:<br>		search_queue += graph[person]<br>return False</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*WpoiR_9R4U1kLM9X" /></figure><pre>def search(name):<br>	search_queue = deque()<br>	search_queue += graph[name]<br>	searched = []<br>	while search_queue:<br>		person = search_queue.popleft()<br>		if not person in searched:<br>			if person_is_seller(person):<br>				print person + “ is a mango seller!”<br>				return True<br>		else:<br>			search_queue += graph[person]<br>			searched.append(person)<br>	return False</pre><p>Note that the running time for this algorithm will be at least<strong> O(n)</strong> because we search through and follow each edge.</p><blockquote><em>“</em>Breadth-first search takes O(number of people + number of edges), and it’s more commonly written as O(V+E) (V for number of vertices, E for number of edges).”</blockquote><h3>Dijkstra’s Algorithm</h3><p>4 main steps:</p><ol><li>Find the “cheapest” node. This is generally the node that you can get to with the shortest cost</li><li>Update the costs of the neighbours of the node.</li><li>Repeat this until you’ve done this for every node in the graph.</li><li>Calculate the final path</li></ol><blockquote>“In the last chapter, you used breadth-first search to find the shortest path between two points. Back then, “shortest path” meant the path with the fewest segments. But in Dijkstra’s algorithm, you assign a number or weight to each segment. Then Dijkstra’s algorithm finds the path with the smallest total weight.”</blockquote><p>Each edge in the graph has a number associated with it, known as weights (graphs with weights are known as weighted graphs, unweighted for no weights).</p><p>Whenever dealing with unweighted graphs, <strong>breadth-first</strong> works. For weighted ones, <strong>Dijkstra’s algorithm.</strong></p><p>Here’s an example of trading a <strong>piano book → actual piano:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/0*mULq1A6PLs6nCD5h" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*AVCGbnkwZdL3Sn4K" /></figure><p>We work in <strong>hierarchies.</strong> That means the first section (<strong>parent column)</strong> would be the book as <strong>that’s the parent for the poster + rare LP.</strong></p><p>What are the trades he needs to make?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*D-V7tHfV_gLPYV84" /></figure><p>We can also use <strong>negative signs</strong> when we want to show situations <strong>where we get something back</strong> (ex. I get paid back $7 for getting some item).</p><p>The only concern? <strong>Dijkstra’s algorithm doesn’t support negative weights.</strong></p><p>What do we do? <strong>Use the Bellman-Ford algorithm!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*jcMyhGrcGnqKKuCL" /></figure><p>What the algorithm looks like?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/0*9E_Hmue46W0P4_YF" /></figure><pre>def find_lowest_cost_node(costs):<br>	lowest_cost = float(“inf”)<br>	lowest_cost_node = None<br>	for node in costs:<br>		cost = costs[node]<br>		if cost &lt; lowest_cost and node not in processed:<br>			lowest_cost = cost<br>			lowest_cost_node = node<br>	return lowest_cost_node<br><br>node = find_lowest_cost_node(costs)<br>while node is not None:<br>	cost = costs[node]<br>	neighbors = graph[node]<br>	for n in neighbors.keys():<br>		new_cost = cost + neighbors[n]<br>		if costs[n] &gt; new_cost: #we want to reduce + lower the cost<br>			costs[n] = new_cost<br>			parents[n] = node<br>	processed.append(node)<br>node = find_lowest_cost_node(costs)</pre><h3>Greedy algorithms</h3><p>It’s pretty straightforward: <strong>at each step, you pick the locally optimal solution, and in the end, you’re left with the globally optimal solution.</strong></p><p>They’re also super simple to write and usually hit the solution.</p><h4>The Set-covering problem</h4><blockquote><em>“Suppose you’re starting a radio show. You want to reach listeners in all 50 states. You have to decide what stations to play on to reach all those listeners. It costs money to be on each station, so you’re trying to minimize the number of stations you play on. You have a list of stations.”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/0*3PPvFChScWOlH497" /></figure><p>What’s the problem? <strong>It takes an extremely long time to calculate every single possible subset.</strong> The running time is <strong>O(2^n)</strong>, it’s WAY TOO SLOW. That would be at least <strong>15 years to figure this out</strong> (assuming that you calculate 10 subsets/second)<strong>!</strong></p><p>How can we solve this with greedy algorithms?</p><ol><li>Pick the station that covers the most states that haven’t been covered yet. It’s OK if the station covers some states that have been covered already.</li><li>Repeat until all the states are covered.</li></ol><p>What we’re using is something known as an <strong>approximation algorithm.</strong> Calculating the exact solution takes way too long.</p><p>How do we judge the algorithm?</p><ol><li>How fast they are</li><li>How close they are to the optimal solution</li></ol><p>What makes greedy a good choice? It has an <strong>O(n²) running time</strong> while being able to have high running speeds.</p><pre>states_needed = set([&quot;mt&quot;, &quot;wa&quot;, &quot;or&quot;, &quot;id&quot;, &quot;nv&quot;, &quot;ut&quot;, &quot;ca&quot;, &quot;az&quot;])<br>arr = [1, 2, 2, 3, 3, 3]<br>stations = {}<br>stations[&quot;kone&quot;] = set([&quot;id&quot;, &quot;nv&quot;, &quot;ut&quot;])<br>stations[&quot;ktwo&quot;] = set([&quot;wa&quot;, &quot;id&quot;, &quot;mt&quot;])<br>stations[&quot;kthree&quot;] = set([&quot;or&quot;, &quot;nv&quot;, &quot;ca&quot;])<br>stations[&quot;kfour&quot;] = set([&quot;nv&quot;, &quot;ut&quot;])<br>stations[&quot;kfive&quot;] = set([&quot;ca&quot;, &quot;az&quot;])<br>final_stations = set()<br>while states_needed:<br>	best_station = None<br>	states_covered = set()<br>	for station, states_for_station in stations.items():<br>		covered = states_needed &amp; states_for_station # this is a set intersection<br>		if len(covered) &gt; len(states_covered): #check whether this station covers more states then the current station, best_station<br>			best_station = station #update if condition is met<br>			states_covered = covered<br>states_needed -= states_covered #update because the station has covered those states<br>final_stations.add(best_station)<br>print(final_stations)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/0*8Sa3a7ZsxaPSAIfj" /></figure><p>In this example, the travelling salesman is an <strong>NP-complete problem</strong>; a computational problem for which <strong>no efficient solution algorithm has been found.</strong></p><h3>Dynamic programming</h3><p>We talked about in the previous chapter (greedy algorithms) that sometimes, we can only come up with an <strong>approximate solution.</strong> But is it even possible to find + calculate the optimal solution? That’s where dynamic programming comes in.</p><p>Dynamic programming is essentially this method to be able to <strong>solve subproblems that build up to solving the big problem.</strong></p><h4>The knapsack problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*olv2DwENMq3zm4-r" /></figure><p><strong>Every dynamic problem starts with some sort of grid</strong>, here’s a grid for the knapsack problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*ccNl4Qxs8dLC0UCD" /></figure><p>The question that the grid is trying to answer is <strong>if your knapsack had capacity <em>x</em> lb, what is the maximum value that you can put in that bag?</strong></p><p>As you fill through the table, this is what you end up becoming at:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*6t_E-7WbT8k-pZDf" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/0*0gPvnRqzUaAQW0He" /></figure><p>According to your last best estimate, the best item you can put in there for that 1lb space is the guitar. This is what your now final grid looks like!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*gI4tGyq6zGU7DDjK" /></figure><p>What does that look like as a formula?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*yhGsx_tNIJV2t-4M" /></figure><p>The biggest question that pops up in your mind is <strong>can the value of a column ever go down?</strong></p><p><strong>NO.</strong> At every iteration, you’re storing the current maximum estimate. That estimate that you get will never be worse than what it was before. The same thing with the order of the table, <strong>it makes no impact.</strong></p><p>What about dealing with fractions of items? <strong>That wouldn’t work, dynamic programming allows you to only take the item or not.</strong> Instead, you can solve this with a <strong>greedy algorithm!</strong> Take as much as you can of one item, when that runs out, take as much as you can of the next most valuable item, and so on.</p><blockquote><em>“</em>Dynamic programming is useful when you’re trying to optimize something given a constraint. In the knapsack problem, you had to maximize the value of the goods you stole, constrained by the size of the knapsack.”</blockquote><p>General tips for dynamic solutions:</p><ol><li>Every dynamic-programming solution involves a grid.</li><li>The values in the cells are usually what you’re trying to optimize. For the knapsack problem, the values were the value of the goods.</li><li>Each cell is a subproblem, so think about how you can divide your problem into subproblems. That will help you figure out what the axes are.</li></ol><h3>Longest common subsequence</h3><p><strong>Suppose Alex accidentally searched for fosh. Which word did he mean: fish or fort?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*QcoOoJ9Tt7jtgqdN" /></figure><p>That isn’t gonna work. Let’s redo the algorithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/0*CCl1KxyLClETJfv6" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*vvlrpAUCmFy-1T5J" /></figure><p>Notice how it works in a diagram “way”. When a new match is found, that row + column is filled with that max value until another match is found.</p><p>What are some applications of dynamic programming?</p><ol><li>Biologists use the longest common subsequence to find similarities in DNA strands. They can use this to tell how similar two animals or two diseases are. The longest common subsequence is being used to find a cure for multiple sclerosis.</li><li>Have you ever used diff (like git diff)? Diff tells you the differences between two files, and it uses dynamic programming to do so.</li><li>We talked about string similarity. Levenshtein distance measures how similar two strings are, and it uses dynamic programming. Levenshtein distance is used for everything from spell-checking to figuring out whether a user is uploading copyrighted data.</li></ol><h3>k-Nearest Neighbors</h3><p>Generally, the main intuition behind KNNs is finding the <strong>closest neighbour</strong> of a point when graphed out.</p><p>On a high-level understanding, Netflix’s algorithm also works like this!</p><p>Generally, users are plotted out based on similarity so whenever you get recommended movies, Netflix <strong>looks at the people who share similar taste and choose a movie that’ll appeal to you.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/0*imKCig9SiCqW1nbU" /></figure><p>The way you figure out the similarity will be through the <strong>features you choose to filter by.</strong> The way we find the distances between two [or more] points can be through the <strong>Euclidian distance</strong> (aka Pythagorean formula):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*3YkGZvdz3pBRENK_ZBJXxg.gif" /></figure><p>With KNNs, there are 2 main things you can do: <strong>classification, and regression.</strong> In this case, above we did classification i.e. classifying genres of similar movies, but regression is super useful as well.</p><p>The main question right now is <strong>“how do I choose the right features to compare against?”</strong> 2 things:</p><ol><li>Features that directly correlate to the movies you’re trying to recommend</li><li>Features that don’t have a bias (for example, if you ask the users to only rate comedy movies, that doesn’t tell you whether they like action movies)</li></ol><p>Additionally, this can also fall into Machine Learning algorithms. By selecting accurate + extracting relevant features, you can determine its nearest neighbours to make an accurate prediction.</p><h3>Binary trees</h3><p>The main intuition behind binary search trees is mainly that <strong>whenever you do binary search, you have to add and then sort the array. That’s inefficient.</strong></p><p>For every node, the nodes on the left of it have a lower value while the ones on the right are a lot larger in value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/0*GUat5F9njP0EdU07" /></figure><p>So, if I wanted to search for Maggie, I’d follow <strong>David → Manning → <em>Maggie</em>!</strong></p><blockquote>“Searching for an element in a binary search tree takes O(log n) time on average and O(n) time in the worst case. Searching a sorted array takes O(log n) time in the worst case, so you might think a sorted array is better. But a binary search tree is a lot faster for insertions and deletions on average.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/0*7ivCoqVRIiUyAlGI" /></figure><p>One of the main downsides of binary trees is that you don’t get random access (indexing). The performance of a tree relies fully on <strong>how balanced the tree is.</strong> Look at the tree below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*CduPDcT8wU7w7Ecc" /></figure><p>Some trees are also able to balance themselves, like the red-black tree. B-trees are used in storing data in databases.</p><h3>Inverted indexes</h3><p>Let’s say that we have 3 web pages with this content:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/0*1CRqCLsV7qV0yxjm" /><figcaption>From there, let’s build a hash table from that:</figcaption></figure><p>From there, let’s build a hash table from that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/0*DpNPVYUuJfcnXhHC" /></figure><p>Now let’s say that the user <em>searched for </em><strong><em>there.</em></strong> Now we easily know that we can <strong>display pages A and C.</strong> We’re essentially creating a <strong>hash that can map words to places where they appear (web pages).</strong> This is exactly what inverted indexes are.</p><h3>Fourier transform</h3><p>Fourier transform is essentially this algorithm that: <em>given a smoothie, the Fourier transform will tell you the ingredients in that smoothie. Or given a song, the transform can separate it into individual frequencies.</em></p><p>What’s the use case? <strong>With songs, you can boost the individual frequencies you care about (boost bass, hide the treble).</strong></p><p>Can also be used in compressing music. Break down the audio file into its ingredient notes. From there, the transform will tell you exactly how much each note contributes to the song. Then you delete the irrelevant notes. That’s exactly how mp3 works!</p><h3>Parallel algorithms</h3><p>The main focus of parallel algorithms is to <strong>take advantage of the multiple cores in a computer.</strong></p><p>For example, the best time you can get with a sorting algorithm is generally <strong>O(n log n) time.</strong> You can’t generally sort an array in <strong>O(n)</strong> time <strong>unless you use a parallel algorithm!</strong></p><p>Parallel algorithms are generally super hard to design + get it working.</p><blockquote>“So if you have two cores in your laptop instead of one, that almost never means your algorithm will magically run twice as fast.”</blockquote><ol><li>Parallelism management — if you have to sort an array of 1,000 items, how do you divide this task among the two cores? Even if you give each core 500 items to sort and then merge them, merging itself takes time.</li><li>Load distribution amongst processors — if you have 10 tasks to do, you’d probably give each core 5 tasks. What’s the problem? The first core is given the easy tasks whereas the other core gets all the hard tasks. <strong>Now a core stays idle while it could be used to work.</strong></li></ol><h4>MapReduce</h4><p>A special type of parallel algorithm used is MapReduce that’s becoming increasingly famous, known as <strong>distributed algorithms</strong>. MapReduce is essentially a popular distributed algorithm to manage workloads amongst hundreds of cores.</p><p>The main benefit here is that distributed algorithms allow you to do lots of work while the objective is to speed up the time it takes to do it.</p><p>MapReduce works on 2 main functions: <strong>map</strong> and <strong>reduce:</strong></p><p>The map function <strong>applies a function to each item in an array</strong> while distributing the workload on each machine/core.</p><p>The reduce function <strong>downsizes a list of items into a singular output.</strong> An example might be…</p><pre>&gt;&gt;&gt; arr1 = [1, 2, 3, 4, 5]<br>&gt;&gt;&gt; reduce(lambda x,y: x+y, arr1)<br>15</pre><h3>Bloom filters + HyperLogLog</h3><p>The main problem is that <strong>you want to go through a large list to figure out whether an element is in a large set.</strong></p><p>The fastest way to do this could be with a has; you have 2 columns: {website_name: checked_or_not}. So, if I wanted to check if I already saw <a href="http://srianumakonda.com/">srianumakonda.com</a>, then I can just look it up in the hash.</p><p>Remember that the average lookup time is <strong>O(1).</strong></p><p>What’s the problem though: <strong>these hash tables are HUGE. It’s a space problem.</strong></p><p>Bloom filters are essentially probabilistic data structures. Although false positives might be possible, false negatives are NOT.</p><p>The main benefit here is that Bloom filters take up <strong>very little space to operate.</strong> Tables have to store every single URL by Google while it’s ok to be wrong in these scenarios.</p><p><strong>HyperLogLog is also quite similar.</strong> You can approximate the number of unique elements in a set (ex. Amazo wants to count the # of unique elements that users looked at today) <strong>with a fraction of the memory.</strong></p><h3>SHA Algorithms</h3><p>SHA (<strong>Secure hash algorithms</strong>) convert input to a hash for that string. <em>Note: hashes are just a short string in this context where the goal is string→string.</em></p><p>SHAs can be used in situations to check the similarity of files, especially in the case of extremely large files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*ZPzi17kZIQtg7EqT" /></figure><p>Remember that this is only <strong>one-way. If someone gets the SHA hash, they can’t convert that back into the original string.</strong></p><blockquote><em>“</em>SHA is actually a family of algorithms: SHA-0, SHA-1, SHA-2, and SHA-3. As of this writing, SHA-0 and SHA-1 have some weaknesses. If you’re using an SHA algorithm for password hashing, use SHA-2 or SHA-3. The gold standard for password-hashing functions is currently bcrypt (though nothing is foolproof).”</blockquote><p>The other thing to note that is <strong>even a single change of the character in a sting results in completely different outputs!</strong> In the case that you a locally-sensitive hash function, you can use something like <strong>Simhash.</strong></p><h3>Diffie-Hellman key exchange</h3><p>The DH key exchange solves a problem that’s lurking for quite some time now: <strong>How do you encrypt a message so it can only be read by the person you sent the message to?</strong></p><p>The easiest way to solve this would be to come up with a cipher i.e. a=1, b=2, etc. The main problem here is that <strong>both parties need to agree on the cipher.</strong></p><p>The benefits?</p><ol><li><strong>Both parties don’t need to know the cipher.</strong></li><li>The encrypted messages are <strong>extremely hard to decode.</strong></li></ol><p>Diffie-Hellman has 2 keys: <strong>a public key and a private key.</strong> Everyone can have your public key but when you want to send a message, you can encrypt it using the public key. <strong>Only the private key can be used to decrypt it.</strong></p><p><em>👋 Hey, thanks so much for reading this article! I’m a 15yo autonomous vehicle enthusiast, currently building </em><a href="https://srianumakonda.notion.site/End2End-Self-Driving-a3e063810d164ea982a4157b0fc8e403"><strong><em>end2end models for self-driving</em></strong></a><em> [</em><a href="https://github.com/srianumakonda/end2end-learning"><em>github</em></a><em>] with my most recent project being </em><a href="https://srianumakonda.medium.com/datagan-leveraging-synthetic-data-for-self-driving-vehicles-6e629968a567"><em>DataGAN: Leveraging Synthetic Data for Self-Driving Vehicles</em></a><em>. My ask for people reading this: </em><strong><em>I’m currently struggling with computational resources</em></strong><em> for building out my research, would love it if you could </em><strong><em>pass this article on to people who would be willing to give physical compute or even cloud computing via AWS/Azure ($2,000 — $5,000 in credits).</em></strong></p><p><em>Here’s my </em><a href="/service/mailto:sri.anumakonda06@gmail.com"><em>email</em></a><em>, </em><a href="https://www.linkedin.com/in/srianumakonda/"><em>linkedIn</em></a><em>, and </em><a href="https://twitter.com/srianumakonda"><em>twitter</em></a><em> if you would like to learn more about what I’m building. </em><strong><em>P.S. I’m also open to summer 20222 internships in self-driving/Computer Vision!</em></strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FvPxW7LrYtUU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DvPxW7LrYtUU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FvPxW7LrYtUU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fefc12d8d43fcdca5434a5121ef02e2/href">https://medium.com/media/6fefc12d8d43fcdca5434a5121ef02e2/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57005ce03a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataGAN: Leveraging Synthetic Data for Self-Driving Vehicles]]></title>
            <link>https://srianumakonda.medium.com/datagan-leveraging-synthetic-data-for-self-driving-vehicles-6e629968a567?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e629968a567</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[autonomous-vehicles]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Mon, 18 Oct 2021 20:35:24 GMT</pubDate>
            <atom:updated>2021-10-19T22:17:08.481Z</atom:updated>
            <content:encoded><![CDATA[<p>GANs haven’t really found their place in self-driving cars right now, especially with recent advances in simulation and the vast amounts of data that has been collected.</p><p>So far, <strong>more than </strong><a href="https://www.caranddriver.com/news/a30857661/autonomous-car-self-driving-research-expensive/"><strong>$16 billion</strong></a><strong> has been spent on self-driving research</strong>. What’s the problem? Self-driving is expensive? Why? Getting data and training these models are not only time-consuming but really expensive. You then also need to take into account the fact that <strong>Waymo’s spent more than 20 million miles on public roads</strong> for data gathering (talking about the amount of energy consumed is for a whole other article…)</p><p>What percentage of data is actually useful? Very minimal. Why? Most of the data is usually from “normal” driving scenes, <strong>not edge case scenarios</strong> such as car overtaking, parking, traffic, etc.</p><p>How do we solve that?</p><p>Generative adversarial networks. Focusing on road/scenes rather than edge case scenarios i.e. parking and overtaking. Being able to generate vasts amounts of self-driving data while not having to spend excessive amounts of capital gathering data can be crucial in self-driving deployment.</p><p>As an attempt to solve this problem, I’ve been focusing on building DataGAN out. Leveraging Generative Adversarial Networks to create self-driving data at scale is crucial. By emphasizing a focus on DCGANs, I’m focusing on creating high-quality self-driving images that can be used to train and improve the performance of computer vision models such as lane + object detection, and semantic segmentation.</p><h3>The boxing match: generator vs discriminator</h3><p>GANs, simply put, is essentially a type of unsupervised learning algorithm where a model learns to discover and learn patterns in its dataset in such a way that the model itself can create new, generative outputs that could look as real as the original dataset itself.</p><p>In a GAN, 2 neural networks constantly compete against each other. This can be described as:</p><ol><li><strong>The generator: </strong>this is the model that creates your generative outputs. By taking in latent (random) noise, the generator is then able to upsample it to create realistic images.</li><li><strong>The discriminator:</strong> the main goal behind the discriminator is to determine whether the image, created by the generator, is real or fake. This is where the term “adversarial” comes from in GANs → the generator and discriminator compete to see whether the generator can create images that can fool the discriminator.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*Ptpqp6_VOhth5IJl.png" /><figcaption>An example of how GANs work. <a href="https://www.tensorflow.org/tutorials/generative/dcgan">Source</a></figcaption></figure><p>In a sense, GANs are also supervised. The discriminator is a perfect example of that when distinguishing between real and generated images where all images that are generated can be labelled as 0 while the dataset images that the generator is training on can be labelled as 1.</p><p>The goal with GANs would be to reach an equilibrium where your discriminator can no longer distinguish real images from fakes while having a generator that can create realistic images as if it was taken from the original dataset.</p><p>Think of a GAN like a boxing match: if one opponent is stronger than the other, one of the boxers will get destroyed. But if the boxers are even in strength, then they’ll continuously try to find a way to be better than one another (although it’s not possible, because they’re equally matched!).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FIeG7go6Pwa3Lg9Bfxm%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FIeG7go6Pwa3Lg9Bfxm%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FIeG7go6Pwa3Lg9Bfxm%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="238" frameborder="0" scrolling="no"><a href="https://medium.com/media/875e5c41dd5c91ea1fa18b962b610dfe/href">https://medium.com/media/875e5c41dd5c91ea1fa18b962b610dfe/href</a></iframe><p>This is the same thing with GANs: <strong>you want your generator and discriminator to be even in strength; having one stronger than the other will not allow for proper training </strong>(aka mode collapse). Instead, you want both of them to be equally strong, allowing some images to fool the discriminator while generating realistic images.</p><p>Since their introduction into Deep Learning in 2014, several extensions of GANs have been created, including Deep Convolutional GANs (DCGAN), Conditional GANs (CGAN), StyleGANs, and a lot more. DataGAN focuses on leveraging the use of DCGANs for creating its synthetic data.</p><h3>Deep Convolutional Generative Adversarial Networks (DCGANs)</h3><p>Rather than using straightforward Dense layers for generating + discriminating images, DCGANs leverage the use of <a href="https://medium.datadriveninvestor.com/how-you-can-figure-out-your-teachers-age-without-embarrassing-yourself-f4f4363113d6">Convolutional Neural Networks</a> (CNNs) to accomplish this task. The TLDR; of CNNs is that Convolutional Neural Networks are essentially a method used to help break down images while capturing spatial + temporal dependencies an image has via its filters.</p><p>In a DCGAN, we would use upsampling techniques such as transpose convolutions and Leaky ReLU functions for the generator. On the other hand, we would use convolutions for downsampling the input until we get to a 1x1 output for the discriminator, therefore telling us if the input image is real or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*taYEMXmfmuxJPFxc.png" /><figcaption>Original proposal for DCGANs implementation. <a href="https://arxiv.org/abs/1511.06434">Source</a></figcaption></figure><p>Some of the main differences between your Vanilla GAN and a DCGAN can be summarized as follows:</p><ul><li>Both networks (discriminator + generator) are <strong>Fully Convolutional Networks (FCN)</strong>. What this means is that there are no Dense/maxpooling layers after convolutions.</li><li>ReLU is used as the activation function for generators while LeakyReLU is used for the discriminator</li><li>Batch normalization is now a method that’s consistently used to help with gradient flow and avoid vanishing/exploding gradient problems</li><li>Tanh is the output function and should be used instead of something like sigmoid (normalization between [-1, 1] instead of [0, 1])</li></ul><p>In the original GAN paper, the goal of the generator is to minimize the following function while the goal of the discriminator becomes to maximize it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*pmQftxo63MSnb4Fosj8HMQ.png" /></figure><p>D(x) is the discriminator’s estimate of whether data x is real or not. The goal of the discriminator is to maximize this function i.e. output 1, for the true x data. For fake images, we want to ensure that D(G(z))’s [discriminator output from generator for input latent vector x] output is near 0.</p><p>Here’s what that means:</p><p>Let’s say that the discriminator outputs D(x)=0.99, D(G(z))=0.02:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*OMNtMZjEuQ4ikCY3XX303w.gif" /></figure><p>For context, log(1)+log(1)=0. Notice how that the discriminator’s practically maximized this function. It’s predicted that x is a real image while the generated image is not.</p><p>For the generator though, the goal will always be to maximize D(G(z)). This essentially means that we want the discriminator to always predict that the generated image is always real. The closer D(G(z)) is towards 1, the more realistic the discriminator thinks the generator is.</p><p>If the generator’s fooled the discriminator, then this is “assurance” that the way the generator is learning is right, which allows the model to continue training while producing high-quality images.</p><p>When dealing with our loss and optimization functions, we can generally use something like Adam along with a learning rate of 2e-4 (mentioned in DCGAN paper) can allow for more stable training. Another interesting thing to note is that the <a href="https://arxiv.org/pdf/1809.11096v2.pdf">BigGAN paper</a> has shown that making the learning rate smaller for the discriminator (i.e. 5e-5) also helps with training and does not result in immediate collapse.</p><p>The overall loss metric generally used for doing our backward pass + gradient update though would be Binary Cross-Entropy (BCE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/0*q2E9EamlpdwWUmMo.png" /></figure><p>BCE essentially takes the negative average of the log of the predicted probabilities that were correct. For example, the closer D(x) is towards 1, the lower the loss function.</p><p>This is quite straightforward for the discriminator but what about the generator? For the generator, it would look like BCE(D(G(z)), 1). We want the discriminator output to be as close as it can be to 1 (indicating that it’s a real image). If we have a “perfect” generator, then our output values should always be close to 1.</p><h3>DataGAN</h3><p>DataGAN leverages the Fully Convolutional Network (FCN) architecture along with the Deep Convolutional Generative Adversarial Network to create trainable synthetic data.</p><p>With respect to training, I’m currently running DataGAN on a <a href="https://www.kaggle.com/dansbecker/cityscapes-image-pairs">modified version</a> of the Cityscapes Dataset. The quality and resolution of the dataset make it an ideal choice, especially considering the fact that generating urban scenes as opposed to highways, is a lot more beneficial for the overall AV industry.</p><p>For fast processing + training, I’m resizing the images to be 128x128 rather than the original 256x256 mainly because of hardware requirements.</p><p>My generator takes advantage of convolution transpose operations to help create realistic driving scene images while updating weights while training for robust outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WoTYLyKtE4dMeR2m77RECA.png" /></figure><p>The generator is mainly structured around a finite amount of blocks, where each block consists of a Conv Transpose operation followed by a BatchNorm and ReLU layer. This allows the model to not only deal with the vanishing/exploding gradient problem but also have overall better stability + generalization when training the network.</p><p>The more features, the better it generally is. I also chose to implement the idea of a kernel size of 4 and a stride of 2, as implemented in the DCGAN paper. Additionally, the conv kernel bias is set to False, mainly because its impact is not noticeable as a result of the Batch Normalization layer.</p><p>ReLU has been shown throughout various papers and projects to work in the generator models but not the discriminator. Although Leaky ReLU might be helpful in some scenarios, it’s not and might even be redundant.</p><p>The goal generally is to have both a strong generator and discriminator and that doesn’t happen most of the time. What we’d have to do instead is make the discriminator a lot weaker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMONsEKGbAgPoO-9LmLRCQ.png" /></figure><p>As seen, I’ve made the discriminator substantially weaker than my generator to allow for a more, balanced training.</p><p>Just as in the case of the generator, I’m following a similar structure with the conv blocks where I have a convolution followed by a batch normalization + leaky relu.</p><p>Throughout initial training, the biggest problem I noticed was the fact that the discriminator overpowered the generator, causing model collapse. In order to solve this, I initially introduced dropout (commented out) along with reducing the number of features substantially for smoother training.</p><p>We’ve seen that ReLU is used in the generator so why not in the discriminator as well? The problem with ReLU is that it can’t output negative gradients (anything &lt;0). Therefore, this makes the network get stuck in a “dying state.”</p><p>What this means is that we’d essentially face a vanishing gradient problem where the discriminator produces only 0’s as the output. The generator is supposed to receive the gradient values from the discriminator, and if those values are simply 0’s, model collapse is certain to occur.</p><p>Leaky ReLU prevents that by allowing negative values to pass through which can be modified based on a hyperparameter, α (alpha). This will introduce tolerance in the network (which is where the idea of negative values passing through begins). In the case of DataGAN, I choose to follow through with the principles of the DCGAN paper → setting α to be 0.2, which has been proven successful in several models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OWPOAS0gyAvv_VZn.png" /><figcaption><a href="https://androidkt.com/what-is-the-dying-relu-problem-in-neural-networks/">Source</a></figcaption></figure><p>In the BigGAN paper, their findings show that having a bigger batch size allows for not only better generalization, but also for overall better performance. In my case, I chose to use a batch size of 64 as opposed to something like 128 or 256 because of how small my dataset is.</p><p>In order to ensure that the discriminator is weaker than the generator, I choose to set different learning rates for the models: for the generator, I have set the learning rate to 2e-4 while the discriminator uses a learning rate of <br>1e-5. The loss function used is standard Binary Cross-Entropy.</p><h3>Current Progress</h3><p>I’ve been training for around ~4000 (3875) epochs with this dataset (around 3075 images for training). Although I haven’t faced mode collapse (which is a good indicator so far!), here are some sample images for what it’s looking like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATRS2A5S1Z0xAc-c6p7bkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6n9xEzX6puaz-7AMq0ct4w.png" /></figure><p>Notice how the images have multiple, different types of buildings. And then lighting + structure of the lanes as well. <strong>This is data quite similar to that of real, road data, trained from my computer.</strong></p><h3>Why I built DataGAN</h3><p>GAN’s haven’t really found their place in self-driving yet. With DataGAN, I propose a segway for the introduction of GANs in self-driving and show that they can play a vital role in generating synthetic data.</p><p>With recent advances in self-driving research, the main problem right now is deploying these vehicles at scale. Simulation research has been proven to play a huge role in LiDAR + camera research, but what if you can create real-world data without having to collect it yourself?</p><p>That’s the underlying goal of DataGAN → becoming so good at creating realistic driving scenes that one day, a full driving log can be made, just from GANs.</p><blockquote>More updates soon on <a href="https://github.com/srianumakonda/DataGAN">Github.</a></blockquote><h3>Next steps</h3><p>Right now, due to hardware constraints, I’m focusing on generating scene images from DataGAN. Next steps/potential pathways right now could be summarized as followed:</p><ul><li>Build a straightforward lane detection model, fully trained on DataGAN outputs that are functional</li><li>Create the first self-driving GAN dataset that can be used to train robust lane detection + computer vision models</li><li>Make an AC-GAN model where you could potentially input an edge case scenario (ex. overtaking a vehicle) and use GAN-<a href="https://arxiv.org/pdf/1506.04214.pdf">ConvLSTM</a> to help generate the given scene</li></ul><p>If this is a project that you’d like to collaborate and build, <a href="/service/mailto:sri.anumakonda06@gmail.com">please send me an email</a>! I’m always looking for collaborators along with lab resources, anything that you can do would be greatly appreciated.</p><p><em>Thanks for reading and learning more about what I’m building out. For more information about DataGAN, the code behind DataGAN can be found at its </em><a href="https://github.com/srianumakonda/Data-Generation-GANs"><strong><em>Github repository</em></strong></a><em>. If you’d like to build this out/inquire about DataGAN, here’s my </em><a href="/service/mailto:sri.anumakonda06@gmail.com"><strong><em>Email</em></strong></a><strong><em>, </em></strong><a href="https://www.linkedin.com/in/srianumakonda/"><strong><em>LinkedIn</em></strong></a><strong><em>, </em></strong><a href="https://twitter.com/srianumakonda"><strong><em>Twitter</em></strong></a><strong><em>, and </em></strong><a href="http://srianumakonda.com/"><strong><em>Website</em></strong></a><em>. If you’d like to stay updated about DataGAN, you can add your email to my </em><strong><em>newsletter </em></strong><a href="https://subscribepage.com/srianumakonda"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e629968a567" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a real, self-driving car!]]></title>
            <link>https://srianumakonda.medium.com/building-a-real-self-driving-car-1fc4c5fd2395?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fc4c5fd2395</guid>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cars]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Fri, 11 Jun 2021 19:57:17 GMT</pubDate>
            <atom:updated>2021-06-11T19:57:17.264Z</atom:updated>
            <content:encoded><![CDATA[<p>You’ve probably seen me write quite a few articles on the components of the self-driving process (which were quite technical!). For the past few weeks, I’ve been working on building my own, real self-driving car that can drive real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*49SLQfKQ2wO9taRF.gif" /><figcaption>Demo!</figcaption></figure><p>Let’s break this stuff down, so simple that even you’ll be able to understand.</p><h3>The TLDR; of how I built this</h3><p>I used ROS (<strong>Robotic Operating System</strong>), to help successfully communicate between the traffic light detection nodes, a waypoint updater node, and a drive-by-wire (DBW) node via the use of data from sensor fusion (don’t know what that is, click <a href="https://medium.com/analytics-vidhya/the-true-beauty-of-extended-kalman-filters-32e0c08d9258">here</a>). The self-driving car works by using a <strong>waypoint updater algorithm</strong> by creating a trajectory (set of waypoints), the car can follow the path correctly while ensuring passenger safety.</p><h3>System architecture</h3><p>Here’s a rough outline of how the self-driving outline worked and the ROS nodes + topics that were used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*WTRwfdbwl7PY3cDr.png" /><figcaption>Source: <a href="http://udacity.com/">Udacity</a></figcaption></figure><p>A ROS node is essentially an <strong>operator</strong>, given certain data, it performs certain operations (think of it as a calculator, in this case, it would be <strong>executable code</strong> that we’d write) and then uses that operation to publish it to a topic.</p><p>Generally, when we work with ROS, we can <strong>subscribe and publish messages</strong> to a certain “topic” (like a category, ex. the topic of a soccer node would be sports). In this case, the ROS topic allows us to <strong>provide information</strong> (and update) information (which would be the outputs of a ROS node) to the ROS nodes which are “subscribed” to the topic.</p><p>For example, if you look at the Traffic Light Detection node, you’ll notice that it’s subscribed to 3 topics: traffic_waypoint, current_pose, and image_color. That <strong>data is then shared</strong> among all the other nodes in the ROS system → for the traffic_waypoint topic, the 2 nodes subscribed are the Traffic Light Detection node and the Waypoint Updater node.</p><p>All this data is pulled from the Car (simulator).</p><h4>Traffic Detection</h4><p>Getting a close-up perspective on the Traffic Detection node, this is how it looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*39vO_HrNj2-oQh7T.png" /></figure><p>We’re taking in data from our current position, the base waypoints (the trajectory the car will be following), and the image colour (in the sense of the colour of the traffic light → we had the option to use a Deep Learning pipeline for traffic light detection so we’d have to take in the image colour). We would then publish (output) the <strong>locations</strong> where the car needs to stop for traffic lights.</p><h4>Waypoint Updater</h4><p>The purpose of the waypoint updater node is to basically allow us to update our <strong>target velocities</strong> for each waypoint given the traffic light + obstacle detection data (camera + sensor fusion input respectively). We would then publish a new list of waypoints ahead of the car with the desired velocities while incorporating the velocities of the car.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*4Jxm8cjPlaus0ez1.png" /></figure><h4>DBW</h4><p>Drive-by-wire (DBW) is a type of system that CARLA (Udacity’s self-driving car) uses. What this means is that the throttle, brake, and steering values all have an <strong>electronic control</strong> (no mechanical control, we’d use this electronic system to activate these functions).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*XngjZWc1HL-b-TkJ.png" /></figure><p>We would use our inputs from the current velocity to help generate target linear + angular velocities. We would then publish throttle, brake, and steering commands.</p><h3>Breaking all of this down</h3><p>As you might’ve noticed, there are <strong>3 main steps</strong> with respect to the Perception, Planning, and Control steps; the traffic light detection, the waypoint updater, and the DBW steps.<strong> Let’s break this down</strong> and how all of this really works and plays a part in the overall building of the self-driving car.</p><h3>Perception</h3><p>In the 2 nodes as seen in the System architecture diagram, we have both the Object Detection and the Traffic Light Detection node.</p><p>Let’s start with Object Detection. This does include Sensor Fusion and the use of LiDAR and Radar sensors to help figure out as to where we can drive and where we can’t.</p><p>Additionally, we can also use camera-input to detect object detection for cars and objects that sensor fusion might not even be able to detect. Although I <strong>did not opt in to chose camera-input</strong> because the Sensor Fusion algorithm worked fine, <strong>here’s how the camera rundown would look like.</strong></p><p>We’d use a special type of Convolutional Neural Network (read more <a href="https://medium.datadriveninvestor.com/how-you-can-figure-out-your-teachers-age-without-embarrassing-yourself-f4f4363113d6">here</a>) known as a <strong>MobileNet-SSD</strong>. It mainly follows 3 steps:</p><ol><li>Perform <strong>depthwise convolutions</strong> with 1x1 convolutions (this is also known as a pointwise convolution) vs. a vanilla convolution.</li><li>From there, use a <strong>“width multiplier”</strong> which allows the size of input + output channels to be thresholded between 0 and 1.</li><li>We’d also then use a <strong>“resolution multiplier”</strong> to reduce the original input size and threshold to 0 and 1.</li></ol><p>Remember to note that MibileNets and designed specifically for usage on <strong>mobile devices</strong>, not cloud GPUs. Even though more parameters would allow for a more accurate model, remember that we want to run this real-time, in <strong>dynamic environments</strong> where the FPS (frames per second) matter.</p><p>Here’s what the overall CNN architecture would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*wIbtW1PScr_6eeg8.png" /></figure><p>After training and testing the model, <strong>here’s what it looks like</strong> when run in real-time.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FgUz61HwXXlM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DgUz61HwXXlM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FgUz61HwXXlM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9622f3fba0cf783c7ef628a3eda138ae/href">https://medium.com/media/9622f3fba0cf783c7ef628a3eda138ae/href</a></iframe><p><strong>Github repository</strong> for additional information:</p><p><a href="https://github.com/srianumakonda/CarND-Object-Detection-Lab-master">srianumakonda/CarND-Object-Detection-Lab-master</a></p><p>Let’s move onto the <strong>Traffic Light Detection</strong> part of the perception subsystem. In the case of the simulator I was using, I was already <strong>given the data</strong> of the location of the traffic lights and their colour. Alternatively, I could’ve also built yet another <strong>Deep Learning pipeline</strong> to help manage the Traffic Light Detection system.</p><p>Using that traffic detection system, we’d simply just <strong>publish the state</strong> of the traffic light (red, green, or blue) via ROS to the subscribed nodes.</p><p>The way we’d get the data of the location of the traffic light is <strong>through our trajectory.</strong> If a traffic light is part of the path of the desired trajectory, then we’d be able to receive the state of the light which is published through ROS.</p><h3>Planning</h3><p>Just as we mentioned before, we’d use a <strong>waypoint updater algorithm</strong> to help create the desired trajectory.</p><p>When we plan for the desired trajectory, we generally want to ensure that the car does not move backwards and usually forwards when driving (obviously there are situations where taking a step back is good, but we’re ignoring that in this case).</p><p>Instead, what we can do is simply make a check and see if our current waypoint (think of it as an interval in the trajectory) <strong>is in front of the car or not</strong> (I did a dot product multiplication to help figure this out).</p><p>From there, I’d get an output value. If this value is &gt; 0, then it’s a waypoint in front of me allowing me to <strong>continue moving on</strong> that trajectory. If the output value is less than 0, then that waypoint can be discarded, therefore signifying that I’ve made progress on following the trajectory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*rYoj0lokbQ7tMDFBeLbiDA.png" /><figcaption>The green path is our trajectory with those spheres being the waypoints</figcaption></figure><p>Also in the case that there’s an object in front of me or I’m at a traffic light, I’d need to <strong>deaccelerate</strong> my waypoints (trajectory) and ensure that I’m in time before I hit a car or break a law. We can do special thresholding that our car’s <strong>speed becomes the actual speed limit</strong> and as I continue to drive, I need to make sure I’m moving at a slower speed than I was at one timestep before.</p><p>For figuring out how we even create the trajectory in the first place, we can use a <strong>Polynomial Trajectory Generator</strong> to solve this problem. We can use our sensor fusion data to help us figure out a path we can legitimately drive on.</p><p>Because the intention of the article is to keep this stuff extremely high-level, <a href="https://srianumakonda.medium.com/learning-to-drive-from-a-self-driving-car-a4dda9eb2307"><strong>take a look at this article for more info + technical logistics.</strong></a></p><h3>Control</h3><p>For the control subsystem, we’d be using a <strong>DBW node</strong> to help control the car’s throttle, brake, and steering. We can mainly do this through the use of a <strong>PID controller.</strong></p><p>Generally, when driving, we have something known as the <strong>crosstrack error</strong>. When driving, the crosstrack error (CTE) would be the <strong>center of our lane</strong>. We want to make sure that we’re driving in the center of the lane to ensure that we don’t veer off the center and potentially hit another car.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*PDgMwauOJgkuhLuZ" /></figure><p>The whole PID controller is based on this error: P standing for Proportional, I for Integral, and D for Derivative.</p><p>We say proportional with <strong>respect to the current error</strong> (crosstrack error) to help get an output proportional to the CTE. Think of the proportional (P) controller as a form of “feedback” as it’s completely based on the CTE.</p><p>The I part (Integral) of the PID Controller basically takes the <strong>integration of the CTE error</strong>. What this means is that we’d take the sum of all the CTE errors and use that as a variable to tweak. By using the deviations which are affected with proportion to the sum of their magnitude, we can eliminate any offset (swinging behaviour) when driving, allowing the car to be more centred.</p><p>The D controller (Derivative) should be pretty state forward. In the case of our integral and proportional values being extremely high, by taking the <strong>derivative of those values</strong> with respect to time, we can now get a smaller output for a steering angle. This ensures that we don’t make sudden changes when driving and if there is a desired steering that is to take place, it can be done safely without unusual steering behaviour.</p><p>We can also <strong>threshold our driving</strong>: in the case that our desired PID values is to continue driving straight and if our velocity is below some certain threshold, then we can put the car on brakes to assume that we’re either at a traffic light or that there’s an object in front of us.</p><h3>Results</h3><p>Feel free to take a look at the <strong>Github code</strong> to run the code yourself!</p><p><a href="https://github.com/srianumakonda/CarND-Capstone-Master">srianumakonda/CarND-Capstone-Master</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*49SLQfKQ2wO9taRF.gif" /></figure><h3>Planning for the future</h3><p>Due to COVID circumstances, I, unfortunately, <strong>haven’t been able to run </strong>the code on Udacity’s real, self-driving car (if you’re a Udacity employee reading this right now, s<a href="/service/mailto:sri.anumakonda06@gmail.com">end me an email please</a>!!!)</p><p>Obviously, this<strong> isn’t close to a car</strong> I can actually drive in myself and take me wherever I’d really want. I think there is <strong>massive potential</strong> for improvement and potential opportunities to continue exploring + making an impact in the self-driving car space!</p><p>I’m currently fascinated by using camera inputs to perform <strong>lane detection in extremely adverse conditions</strong> (snow, fog, etc.). If you do know anyone who’s doing some research in this space, I’d love to connect with them! Please send me an email to them, it’d mean the world! Here’s my email: <a href="/service/mailto:sri.anumakonda06@gmail.com">sri.anumakonda06@gmail.com</a>.</p><p>Additionally, I’m also open for <strong>internships anywhere in the self-driving space.</strong> If you know anyone (or you) who would like to take me in as an intern or talk about potential working opportunities, feel free to send me an email using the link above.</p><h3>Connect with me</h3><p><a href="/service/mailto:sri.anumakonda06@gmail.com">Email</a></p><p><a href="https://www.linkedin.com/in/srianumakonda/">Linkedin</a></p><p><a href="https://twitter.com/srianumakonda">Twitter</a></p><p><a href="http://srianumakonda.com/">Website</a></p><p>Find this stuff cool? <a href="https://subscribepage.com/srianumakonda">Subscribe to my newsletter.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fc4c5fd2395" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning to drive (from a self-driving car!)]]></title>
            <link>https://srianumakonda.medium.com/learning-to-drive-from-a-self-driving-car-a4dda9eb2307?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4dda9eb2307</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[autonomous-vehicles]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Tue, 18 May 2021 19:35:12 GMT</pubDate>
            <atom:updated>2021-05-19T16:55:18.092Z</atom:updated>
            <content:encoded><![CDATA[<p>Humans are the most complex organism known to the whole history of the universe. Our ability to make rational decisions and process millions of possible outcomes within a matter of seconds has allowed us to dominate the world, procreate, and advance technological evolution.</p><p>Although often overlooked, driving is one of the most remarkable actions a human can perform. The ability to process several different outcomes while being able to understand the location you’re at, understand the traffic signs and lights in front of you, and being able to predict the path other drivers will follow is simply exceptional.</p><p>But can you code the same, rational thinking process into an autonomous vehicle? How does a self-driving car think like a human? How does it make decisions with regard to the direction it should continue to drive in? Let’s take a look at Path Planning, an integral field of the development of autonomous vehicles.</p><h3>Note: Frenet Coordinate System</h3><p>The article will be covering the use of Frenet coordinates, an alternate coordinate system that can be used to help map out the car’s actions on a road in more of an intuitive way than the traditional x y coordinates.</p><p>This is how our car would look like when displayed in the Cartesian coordinate system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4-zPDdqTKusxnNXmjiHrqQ.png" /><figcaption><a href="https://fjp.at/posts/optimal-frenet/">Source</a></figcaption></figure><p>This is how we can represent a car using the Frenet Coordinate system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoUoyN7ZckHP5kW665mlLw.png" /><figcaption><a href="https://fjp.at/posts/optimal-frenet/">Source</a></figcaption></figure><p>As depicted from the second image, the frenet coordinate system allows for a proper system where the road can be successfully mapped and we would not have to deal with the recursive problem of mapping the road at every single timestamp as a polynomial.</p><p>The s coordinate is defined to be the length of the road and can be depicted as the longitudinal positions of the car with respect to its reference path. d is the lateral direction of the car with respect to the center of the lane. The center of the lane, as depicted in blue, can be thought of as the “y” axis where it takes the shape of the road.</p><p>We can now define our variables, s and d, and their accelerated positions, velocities, and accelerations. Because we initially don’t define the relationship between s and d, we can assume that both variables are independent and initialize their positions, velocities, and accelerations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*HvoMYO3l9QcFAWp4" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*a8Mk7nwrdjDH_CpM" /></figure><h3>The motion planning problem</h3><p>We can define the motion planning problem as given the start configuration (qstart → this will be given from localization and sensors), a goal configuration (qgoal), and the constraints in the environment (physics, map, traffic, etc.), find a sequence of moves in the configuration space (all possible configurations of the car → can be in [x, y, θ] where θ is our heading) that can move the robot from qstart to qgoal without colliding with any obstacles.</p><p>The overarching path planning process can be broken down into 3main subsystems: prediction, behaviour planning, and path planning.</p><h3>Predicting the actions of other cars</h3><p>We can first start off the overall problem through predicting the actions of other cars given a certain timespan. We want to be able to predict and deduce the actions of other vehicles so that the self-driving car is able to drive without colliding into other vehicles while optimizing for the best path required.</p><p>The way we can handle multi-model uncertainty beliefs of an object is through a method that we can use to help keep track of our prior beliefs through a Gaussian distribution.</p><p>This will allow us to model the car in such a way that we can successfully make a prediction as to where the car will go.</p><p>In the prediction pipeline, the 2 main processes we can generally take is a model-based approach and a data-driven approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAFFuKpX5YkfYGDu072oqw.png" /><figcaption>Example pipeline of a model-based vs data-driven approach. <a href="http://udacity.com/">Source</a></figcaption></figure><p>The model-based approach allows us to incorporate the knowledge we have about physics, constraints that are imposed by the road, etc. making this a computationally efficient algorithm.</p><p>On the other hand, the data-driven approach allows us to recognize patterns that might not have been recognized via a black-box Machine Learning algorithm pipeline. An example of a data-driven approach can be Trajectory Clustering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fqRJY7GfwXTECplE.jpg" /><figcaption><a href="https://dfzljdn9uc3pi.cloudfront.net/2019/cs-206/1/fig-1-2x.jpg">Source</a></figcaption></figure><p>By taking in input data of past polynomial trajectories taken by other vehicles, the unsupervised ML algorithm will allow us to characterize and define x amount of base paths (like an average) of the paths taken for every action (left, right, straight). We would then simply fit the action of a self-driving car against a probability distribution to figure out the likelihood of a certain direction. <a href="https://ieeexplore.ieee.org/document/6386017">Here’s the paper for more information.</a></p><p>What we can do is combine an ML classifier with a filter. This can be done using a Gaussian Naive Bayes algorithm to help successfully complete this task. Because of the fact that we can study and learn from input trajectory data while being able to select the key features that matter to us, the Gaussian Naive Bayes algorithm will allow the model to use probability distributions to help account for uncertainty while making proper predictions.</p><p>Given a new data point, the prediction model requires 2 steps:</p><ol><li>Compute the conditional probabilities for the given [x, y] pairings. This can be modelled as</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/1*YI0IfMebTF_ullWi6Q8m6w.png" /></figure><p>where x is the feature, C is the label, with mean μ, and standard deviation σ which is computing during the training process.</p><p>2. Use the traditional conditional probability in a Naive Bayes classifier to take the hypothesis which is the most profitable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*0jELbmgTjxsAXJa1c7oI8w.png" /></figure><h3>Behaviour Planning</h3><p>The behaviour planning problem can be summarized as given the input of the map, the route that we will be taking from start to end, and the predictions that we’ve acquired of the other objects, create a path that can bring the car to the destination, collision-free, smooth, and safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*0WmdgWPVQRBZ2_WWSULmxQ.png" /><figcaption><a href="http://udacity.com/">Source</a></figcaption></figure><p>We need to be able to suggest states that are:</p><ol><li>Feasible</li><li>Safe</li><li>Legal</li><li>Efficient</li></ol><p>The approach we can use to solve this problem is through the use of a Finite State Machine (FSM). We can essentially create a pipeline where we can make decisions based on the finite set of discrete states that we can take. Building an FSM will allow us to make logical and sequential states for high-level situations that the car might face when driving.</p><p>For example, if a car in front of us is driving at an extremely slow speed, then we might create a transition function that can bring us into either the left lane or the lane to the right of us. We might have another node branching out from this condition telling the pipeline that if there is a car in either the left lane or the right lane, then we need to go to the right lane.</p><p>The video linked below further explains the use of FSMs and how they work through the use of visuals:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4rNYAvsSkwk&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4rNYAvsSkwk&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2F4rNYAvsSkwk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c7ed368c456e4813beb51f37a670aa8/href">https://medium.com/media/2c7ed368c456e4813beb51f37a670aa8/href</a></iframe><h3>Polynomial Trajectory Generation</h3><p>Now that we’ve built a model that will plan for typical situations that might occur when driving, we now need to plan for a specific path that we can use in order to successfully move the car and adjust its speed.</p><p>One thing to note is that when we build a suitable trajectory, it needs to be in respect to time; therefore, we’d have a 3-dimensional vector consisting of [s, d, t] where t is the time. Our path has to be executed while taking into account where the other cars will be at a certain time step (thus, driving in traffic is a 3-dimensional problem).</p><p>In order to produce a successful trajectory, there are some requirements it must follow in order to create a successful path. The main condition we have to keep in mind is jerk. Having a higher jer has been shown to be uncomfortable. Focusing on this specific aspect can allow us to create a trajectory model while limiting the amount of jerk.</p><p>We define the total squared jerk to be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*XYwrT9GQsc92D5fK" /></figure><p>where the jerk value itself is defined to be as s(t)² (note that the s means the 3 dots in the equation → medium doesn’t support latex embedding) which can be otherwise known as the third derivative of our position with respect to time. The goal of the total squared jerk function is to be minimized by creating a function, s(t).</p><p>If you chose to go through the math behind the total squared jerk, you’ll realize that all the time derivatives of s after the 6th order will have to be 0 in order for s to be jerk minimal.</p><p>Using that realization, we can define s(t) with 6 coefficients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/0*ZzM1BeCRSVGyp0q6" /></figure><p>Note: this also applies to the lateral displacement, d(t).</p><p>Using the equations provided above, we can take the first and second-order derivatives in order to successfully get the equations for the velocity and acceleration equations respectively:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*VeutMqguuDh1yOBB" /><figcaption>Velocity equation (first-order derivative)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*QoVxIK16v_jmHbGa" /><figcaption>Acceleration equation (second-order derivative)</figcaption></figure><p>But what happens when we set the initial time as 0? When we do that, we find that,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/0*IiU-00fwrchwpOsX" /></figure><p>Because of this, we can now simply our equations to become</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*Hw75DKa-mh0XgPXs" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*I6kSlgBns4KJWdGL" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/0*WF9GrImoMNrcDh-a" /></figure><p>If we put this all in matrix form by separating the coefficients from the unknown variables (α), we get,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SkRTMhZTYVJCtY0e.gif" /></figure><p>The way we can grab the coefficients for α_3, α_4, and α_5 is by taking the inverse of the time coefficients and multiplying that be the output vector.</p><p>Isn’t that simply amazing!? By taking the inverse of the matrix on the left and multiplying that by the output vector, we’re not only able to get the 3 missing variables but we can now fit them into the original equation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/0*ZzM1BeCRSVGyp0q6" /></figure><p>and get an output polynomial that we can use as a viable path!</p><h3>Results</h3><p>I’ve implemented this in C++ and here you’ll be able to find the outcome of the Jerk Minimization Trajectory implemented in Udacity’s simulator below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUl4WQM2VOE1Ajejl7a6KQ.gif" /></figure><p>Feel free to take a look at my Github:</p><p><a href="https://github.com/srianumakonda/Become-a-Self-Driving-Car-Engineer-Term-2/tree/master/CarND-Path-Planning-Project-master">srianumakonda/Become-a-Self-Driving-Car-Engineer-Term-2</a></p><h3>Connect with me</h3><p><a href="/service/mailto:sri.anumakonda06@gmail.com">Email</a></p><p><a href="https://www.linkedin.com/in/srianumakonda/">Linkedin</a></p><p><a href="https://twitter.com/srianumakonda">Twitter</a></p><p><a href="http://srianumakonda.com/">Website</a></p><p>Want to hear more? <a href="https://subscribepage.com/srianumakonda">Subscribe to my newsletter.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4dda9eb2307" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who (and where) are we in this world?]]></title>
            <link>https://srianumakonda.medium.com/who-and-where-are-we-in-this-world-d5636f58455c?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5636f58455c</guid>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[udacity]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Sun, 09 May 2021 17:27:12 GMT</pubDate>
            <atom:updated>2021-05-09T17:27:12.180Z</atom:updated>
            <content:encoded><![CDATA[<p>We, humans, tend to have quite the memory. Especially when driving. We know what a red traffic light is and realize that we need to deaccelerate our cars, remember road names and even perform path planning with Google Maps, to even understand how to drive a car!</p><p>Let’s say I give you this image of the world map and ask you to pinpoint (to the nearest city) of where I am. You probably have no idea. I could be in San Francisco all the way to Sydney or even Buenos Aires.</p><p>But let’s say I give you this image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*l-1F6VEIsigf4eZXUR8U8A.jpeg" /></figure><p>Mathematically speaking, the Gaussian distribution of the initial prediction I asked you to make (when I asked you to tell me where I am by simply giving the world map) is completely even.</p><p>Let me give you one more image and now let’s see if you can figure out where I am:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*gnAxBopt4rO4vW7gi25znw.jpeg" /></figure><p>When I gave you a specific image, the Effiel Tower, you were able to pinpoint the exact city I was talking about given that data point. You’ve figured out the mean of the Gaussian ( μ) with very low variance (σ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*dL6qSrTO2c2JWmakwsH1Kg.jpeg" /></figure><p>This is essentially the heart of localization in autonomous vehicles. Given an initial GPS estimate of our location, we can use landmarks as a source to help localize and figure out where we (the self-driving car) are in the world.</p><h3>GPS sucks</h3><p>A lot of people generally use GPS to drive when on highways, that’s fine. But in the case of autonomous vehicles, GPS signals tend to have a variance (σ) of between 3 and 10 meters which is extremely inaccurate. The GPS might be telling me that I’m driving in the right lane but in reality, I’m completely off track and probably hit a few pedestrians.</p><p>GPS is a great place to start with in terms of localizing a self-driving car. The problem though is simply the fact that it’s an inaccurate guess to believe that GPS is the solution to localization.</p><p>GPS helps autonomous vehicles with a rough estimate as to where the car is (in math terms, that’d be a Gaussian with 0 means). Initially, the car could literally be anywhere in the world. But GPS allows the car to get a better, updated estimation as to where it is. The mean would be higher than what it would be before (because we’ve pinpointed the location) with a drastic decrease of variation (from any coordinate point in the whole world to coordinate points between 3–10meters).</p><p>In order to build a better estimation system, we’re going to have to start at our beautiful Kalman Filters.</p><h3>Using Sensor Fusion data</h3><p>The first step with localizing autonomous vehicles comes from gathering our data from Sensor Fusion. I wrote an article going more in-depth about this which <a href="https://srianumakonda.medium.com/the-true-beauty-of-extended-kalman-filters-32e0c08d9258">you can read here</a>. (<em>Note: this section assumes that you’ve read the article that I’ve linked on Sensor Fusion or you know how Kalman Filters + Sensor Fusion works).</em></p><p>When we take in Sensor Fusion data, we essentially get a list of all the objects that are detected through the LiDAR and Radar sensors of the car. We also get the coordinates of landmarks (ex. buildings) that don’t “move” over time to help figure out where we might be.</p><p>What’s really interesting is that we can use a nearest-neighbours algorithm to actually help us figure out where the car might be. For every single point in the Sensor Fusion data, the nearest-neighbour algorithm will associate the closest point to each landmark. Based on that, we’re able to figure out where we can potentially be.</p><p>The problem is simply the fact that this is not an extremely accurate method and we’ll need something even more accurate to make predictions. But, let’s use this as an initial estimate as to where we could be, on top of our GPS estimates.</p><h3>Particle filters</h3><p>The GIST of Particle filters is that they perform the realization of Bayes Rule. We scatter a bunch of points of all the possibilities we could be at (Gaussian with 0 mean) and associate each point with a weight of 3 dimensions [x, y, θ] where x and y are our coordinate points and θ is our heading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*2m4j1D-fEiTk-FY2.gif" /><figcaption><a href="http://udacity.com/">Source</a></figcaption></figure><p>What you notice in this image is that all the red dots placed around the whole map are all the possibilities where the car might be. The blue lines are landmark locations from Sensor Fusion data (pinpointing how far each landmark is from us) with the nearest neighbour implemented in order to better understand our distance from the landmark. Each particle has a weight associated with it which we can deem as important weights. The more important the weight, the higher probability it will survive (weight size is directly proportional as to how long the estimate will last).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*s9lj0WQvzSolCKcs.gif" /><figcaption><a href="http://udacity.com/">Source</a></figcaption></figure><p>This is exactly shown in the GIF below. The weights are resampled at every single timestep and the bigger the weight is, the higher probability that it actually survives.</p><p>This can be broken down into 4 main steps:</p><ol><li>Initialization</li><li>Prediction step</li><li>Update step (weights)</li><li>Resampling</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1mdLLWn9GWOgyR6T.png" /><figcaption><a href="http://udacity.com/">Source</a></figcaption></figure><h3>Initialization</h3><p>The first step with Initialization is to figure out how many particles (the “red dots”) we want to use for estimations (yup, this is a hyperparameter that we’d tune). You need to figure out the optimal value to use between accuracy and cost (computation cost → can make the car too slow).</p><p>The easiest way to initialize a car is simply to divide up the map/initial GPS estimate into evenly spaced grid sections while taking into account Gaussian sensor noise distribution.</p><h3>Prediction</h3><p>Now that we’ve initialized our map space and initialized all the sensor data which can be passed into the prediction step, it’s time to do exactly that; make a prediction for every single particle as to where we might be. We would update each particle using the velocity and yaw measurements with Gaussian sensor noise using a basic motion model.</p><p>The formula below helps us exactly with that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/0*md6EZEGb4e-R4CLu.png" /></figure><p>Earlier above in the article, I mentioned that each particle comes with an associated x, y, and θ while accounting for the Gaussian sensor noise. We use this simple motion model (which is updated based on velocity) to help estimate where the car might be.</p><h3>Update step (weights)</h3><p>Data association is the problem that we have to solve before updating our weights to help map up our LiDAR measurements with the map landmark. This is exactly where the nearest neighbour algorithm comes into play. We compare the magnitude of all the points and use that data to help figure out which LiDAR point is closest to the actual landmark itself. This nearest neighbour assumption model would play a role in terms of updating the weights of the particles.</p><p>In the update step, instead of using our feature measurements (LiDAR + Radar) to affect the predictions about the location of the car, we can use the measurements to update the weights of the particles.</p><p>The way we can update the weights of the particles is through the use of the Multivariate Gaussian Probability Density function for each measurement and combine the likelihood of all the measurements by taking their product. Remember that we assume each landmark observation is independent so we will simply take the product of all the likelihoods over all the measurements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Gpf1TFKdx4Bh9IKQnq8WA.png" /><figcaption><a href="http://udacity.com/">Source</a></figcaption></figure><p>xᵢ is our x measurement at time i with yᵢ is our x measurement at time i. Σ would be our covariance matrix, consisting of the Gaussian noise for x and y.</p><p>1 major thing to note is that we’d need to transform the car’s measurements from its local coordinate system to the map coordinate system (xy system → map).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oHDF6tUIaRMhLiTF.png" /><figcaption>Example of the car on the vehicle&#39;s coordinates. This needs to be transformed into the map’s coordinates. <a href="http://udacity.com/">Source</a></figcaption></figure><p>The image above is an example of the car’s coordinates. We now want to get that into the coordinates of a map. This can be done using a Homogenous transformation using the equations below (<a href="https://www.youtube.com/watch?v=h11ljFJeaLo">here’s a great video to learn more about this transformation problem</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yQ12edhHqzcrzgay.png" /></figure><p>So to summarize, the update step consists of 3 parts:</p><ol><li>Transformation from vehicle coordinates to map coordinates</li><li>Data association → use nearest neighbours algorithm to figure out which weights are closest to a landmark</li><li>Update step → use the Multivariate Gaussian Probability Density Function to update the weights of each particle</li></ol><h3>Resampling</h3><p>This is the real meat of localization; how do we get rid of the estimates and weights that don’t matter to us while keeping the estimates that help us localize the car?</p><p>Let’s say that we have a pie chart where we place all our weighted particles there (each particle takes the slice, the bigger the slice = bigger the importance weight %). What would happen is we’d define a variable, β (beta) that starts at 0. β’s value would then be updated by uniformly sampling a value between 0 and 2*the maximum weight (ex. if the biggest weight was 0.3, then the biggest weight would be 0.6).</p><p>We would also choose a random index, i for our weights. So, if we chose index 6 as an example (weight #6 on the pie chart), then we would compare the weight of that and β. If that weight is less than β, then we’d discard weight #6. Now, β is now interfering with weight #7 (the difference between β and weight #6 is what would move onto weight #7).</p><p>If β’s now updated value is less than weight #7, then we keep weight #7 and don’t discard it (it survives). Because now weight #7 is &gt; β, we would now resample a new value for β and continue from there.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwNQVo6uOgYA&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwNQVo6uOgYA&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FwNQVo6uOgYA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4375e5a48fc8ceca1f08f6e6278bd7f5/href">https://medium.com/media/4375e5a48fc8ceca1f08f6e6278bd7f5/href</a></iframe><p>Sebastian Thrun explains this in-depth in his A.I for Robotics Class. Feel free to watch the video for more information. The visuals he provides are extremely helpful with developing the logical intuition behind the Resampling algorithm.</p><h3>Calculating the error</h3><p>In terms of figuring out a method to measure the overall performance of our model, we can use the weighted average error of all the particles. This can be done by simply taking the root squared error between each particle and its ground truth and multiply it by the particle’s weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z84eV9g3ancPfxmOynfirg.png" /></figure><h3>Localization in action!</h3><p>As part of the Become a Self-Driving Car Engineer nanodegree I’ve been doing, I’ve been tasked with this exact challenge below. Feel free to take a look at the Github code and the video demonstration below!</p><p><a href="https://github.com/srianumakonda/Become-a-Self-Driving-Car-Engineer-Term-2/tree/master/CarND-Kidnapped-Vehicle-Project-master">srianumakonda/Become-a-Self-Driving-Car-Engineer-Term-2</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJv5Vw1aJ8jU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJv5Vw1aJ8jU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJv5Vw1aJ8jU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a580e8167af35bf1b10931315409326/href">https://medium.com/media/8a580e8167af35bf1b10931315409326/href</a></iframe><p>Note that the blue circle in the video is actually the estimate of the localized prediction of the car.</p><h3>Connect with me</h3><p>Keep up with my progress and journey to build a fully self-driving car below!</p><p><a href="/service/mailto:sri.anumakonda06@gmail.com">Email</a></p><p><a href="https://www.linkedin.com/in/srianumakonda/">Linkedin</a></p><p><a href="https://twitter.com/srianumakonda">Twitter</a></p><p><a href="http://srianumakonda.com/">Website</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5636f58455c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The True Beauty of Extended Kalman Filters]]></title>
            <link>https://medium.com/analytics-vidhya/the-true-beauty-of-extended-kalman-filters-32e0c08d9258?source=rss-ff11722a4de3------2</link>
            <guid isPermaLink="false">https://medium.com/p/32e0c08d9258</guid>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[kalman-filter]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sri Anumakonda]]></dc:creator>
            <pubDate>Thu, 08 Apr 2021 20:10:35 GMT</pubDate>
            <atom:updated>2021-05-19T17:35:28.813Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been working on the Become a Self-Driving Engineer nanodegree for some time now. Recently, as part of the Sensor Fusion module, I was tasked with building an algorithm that can predict a moving car’s trajectory and location given the previous inputs and uncertainty.</p><h3>What even is a Kalman filter?</h3><p>Kalman filters are a special type of algorithm designed to make estimates given uncertain measurements. Hardcoding the algorithms sounds easy and a lot more practical but what makes an estimation algorithm such as a Kalman Filter optimal for such a task is the fact that it takes in indirect and uncertain measurements to make such a prediction.</p><p>Sensors such as LiDAR and radar are susceptible to noise and drift. Kalman Filters can merge (fuse) these sensors together to help optimize estimations for positions/detections. That’s exactly what sensor fusion is about. Kalman filters play an integral role in the succession of it.</p><p>In the case of autonomous vehicles, Kalman Filters are extensively used in the prediction of motion for other vehicles and pedestrians. Calculating the trajectory and location of an object is crucial to self-driving especially since driving right through can (and will) cause fatalities to other vehicles/pedestrians.</p><h3>How Kalman Filters work</h3><p>Kalman filters follow through a 2 step iteration process: Measurement update and then predictions. It is represented via a series of Gaussian distributions.</p><p>Let’s take an example of a 1-D Gaussian below. Let’s say that this is the location of our car:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RAF8gXdQPNgk_E7U.png" /><figcaption><a href="https://www.youtube.com/watch?v=VFXf1lIZ3p8&amp;list=PLn8PRpmsu08pzi6EMiYnR-076Mh-q3tWr&amp;index=5">Source</a></figcaption></figure><p>This is our initial Gaussian estimate of where the car is at time <strong>xₖ-₁</strong>. Notice that there is a bit of variance σ² but the estimate seems to be pretty confident.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q6i2i9sZtee4UUie.png" /><figcaption><a href="https://www.youtube.com/watch?v=VFXf1lIZ3p8&amp;list=PLn8PRpmsu08pzi6EMiYnR-076Mh-q3tWr&amp;index=5">Source</a></figcaption></figure><p>Now let’s predict for time <strong>xₖ</strong>. Notice at the graph below that our variance σ² is actually even higher than it was before. Let’s now get our new, updated measurement, <strong>yₖ</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uof3UIDyOTOEsLx-.png" /><figcaption><a href="https://www.youtube.com/watch?v=VFXf1lIZ3p8&amp;list=PLn8PRpmsu08pzi6EMiYnR-076Mh-q3tWr&amp;index=5">Source</a></figcaption></figure><p>yₖ is now the measurement update input we get from our sensors. The variance of the measurement update isn’t too much although the root cause of it would mainly be because of the noise from the sensors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uZmcSlO4UTAGjKzK.png" /><figcaption><a href="https://www.youtube.com/watch?v=VFXf1lIZ3p8&amp;list=PLn8PRpmsu08pzi6EMiYnR-076Mh-q3tWr&amp;index=5">Source</a></figcaption></figure><p>Now, what the Kalman Filter does is that it tries to make an <strong>optical state estimate</strong> by combing the prediction at <strong>xₖ</strong> with the updated measurement at yₖ, therefore, producing a Gaussian when an even lower variance than before.</p><p>Let’s break down the Kalman Filter algorithm through the 2 main steps — Prediction then Measurement Update:</p><h3>Predictions</h3><p>Given an object&#39;s current position and velocity, let’s store that into a vector x. So, in a 1-D world, <strong>x = [xₚ,vₚ]</strong> where xₚ is the position of the car in the 1-D world with vₚ being the velocity of the car.</p><p>We would then have a state transition matrix, F, to incorporate time into the equation. Assuming that <strong>vₚ </strong>is constant, our state transition matrix would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*k_uZymJGU9uadJ0tnd7dHw.png" /></figure><p>For our measurement vector, z, we would only incorporate the measurement and not the velocity. This would look like,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*oIlOW-8cCrkjSJS4cDbazg.png" /></figure><p>This [1, 0] matrix is known as H which is being multiplied by our input position, xₚ, and the velocity, vₚ.</p><p>Now that we’ve defined and specified our input matrices, here’s how our position update would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*SajgMKmoZGaHL5FbYsNdRQ.png" /></figure><p>where u is our external motion vector.</p><p>The other matrix we’d be using for our predictions is P which is our estimate covariance matrix that helps characterize our uncertainty. This would look like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*SNnBTTTK2ciVsNqWEKwj3w.png" /></figure><p>Remember that F is the state transition matrix and we are taking the transpose of that of which would be multiplied by F and the initial covariance matrix P.</p><p>Now that we’ve made our initial predictions, it’s time to make the measurement updates for better predictions of x and P.</p><h3>Updating the measurements</h3><p>We can start making our measurement updates by defining our error vector, y. This is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*JrNx1yRXvW4wn_AepP9i-A.png" /></figure><p>Remember that z is our measurement input and H is the measurement function (matrix) that eliminates the velocity component of our input vector x.</p><p>We would now project our system uncertainty into the measurement space by using our existing measurement function projections by defining that as matrix S.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*cjTDBBvwjB9kZeJQDQ_pvw.png" /></figure><p>R is defined as the measurement noise which takes into account the noise of the input sensor.</p><p>All of this is actually then mapped into another matrix known as K otherwise known as the Kalman gain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*b-tWnbJUIcdiN7QBfaNq6Q.png" /></figure><p>Now that we’ve defined the Kalman gain, we can simply update our prediction variables, x and P using the following equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*RG3IpkYsGCHj42hTv0R6YA.png" /></figure><p>and updating our estimate covariance matrix P,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*uTDVAjNYKL9va9acZvS7OQ.png" /></figure><p>This is exactly how we’d program a Kalman Filter for detecting cars and vehicles in the field of autonomous vehicles given all of these inputs.</p><p>In an Extended Kalman Filter, we would perform Sensor Fusion by fusing LiDAR and Radar inputs to help make even better predictions than by simply using one than the other.</p><p>By combining the strengths of each sensor (LiDAR having high-resolution inputs and Radar providing the velocity input), predictions of an object’s (cars/pedestrians) location and velocity can be made even better, therefore, allowing for accurate estimation and limited fatalities/intervention in real-world situations.</p><h3>The real fun begins — Extended Kalman Filters</h3><p>The Extended Kalman Filter (EKF) uses the main equations/logic that we described above for a basic Kalman Filter. The reason why we’d use an EKF is mainly because of the fact that the predicted state x is mapped to a non-linear function (from the Radar inputs). The goal of an EKF is to solve this issue while being able to fuse the inputs together.</p><p>The overall process of the Fusion flow would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MnruxF39w9pARC0i.png" /><figcaption>Source: <a href="https://www.udacity.com/">Udacity</a></figcaption></figure><p>The rundown is that if we’re taking in our initial measurement for any sensor, we’d start by initializing all of our state and covariance matrices. From there, we’d have to then set up the matrices and perform the new predictions + updates. We’d repeat this process until the desired outcome is achieved (ideally avoiding a pedestrian and other objects in the real world). We’d repeat this process for both Radar and LiDAR.</p><p>LiDAR is actually very easy to use and program (in the sense of Kalman Filters) because it directly gives us the coordinates required that we can input into the x vector.</p><p>Radar is completely different than LiDAR. Let’s break that down a bit more.</p><h3>Radar</h3><p>Radar uses a completely different coordinate system than the traditional Cartesian system. Radar uses the <a href="https://en.wikipedia.org/wiki/Polar_coordinate_system">Polar coordinate system</a> which has a different measurement function. Here’s how z would look:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*1RKm25K7JZU2DRPLhLwucw.png" /></figure><p>p is known as the radial distance from the object to the pedestrian. This is essentially the magnitude of the object given that we plot the object’s location and take the vector of that.</p><p>φ (rho) is the measured angle between the ray (vector line) and the x-axis. Take a look at the diagram below for more information</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSiV9--9_sTk2aWQyHIaVg.png" /><figcaption>Source: <a href="https://www.udacity.com/">Udacity</a>. Note that we always put the object’s location on the x-axis and not the y.</figcaption></figure><p>ṗ is the range rate (velocity) along the ray. <a href="https://glossary.ametsoc.org/wiki/Doppler_velocity#:~:text=The%20radial%20component%20of%20the,Doppler%20radar%20or%20Doppler%20lidar.&amp;text=%2C%20where%20r%20is%20a%20unit,motion%20away%20from%20the%20radar.">Here’s more information</a> about the velocity (Doppler/radial velocity).</p><p>Our Radar noise model would look like something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/0*YACKOTJBkEiE4Au_.png" /></figure><p>Notice that we’re taking the variance of every single “coordinate point” for the inputted Radar measurement.</p><p>The question that pops up now is: “How do I map from the Cartesian space to the Polar system where we can work with Radar and get the exact measurements of the angles, velocity, and magnitude?” We can do this by defining a function, h(x) to help solve this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/0*3SGuRIgRBBwsrHtL.png" /></figure><p>The problem that we get by using such a function to map from Coordinate to Radar space (remember: that’s our measurement vector z) is the fact that <em>h(x) </em>is a non-linear function.</p><p>This causes an output that is not Gaussian as a result of not using the non-linear function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNisQr7i3yerp4iyndwblw.png" /><figcaption>Source: <a href="https://www.udacity.com/">Udacity</a></figcaption></figure><p>Here’s how it looks like after sampling 10,000 random values from a normal distribution with a mean of 0 through the non-linear function, <em>h(x) = atan(x).</em> Take a look at the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YWQq7TMG9_Z70H7N.png" /><figcaption>Source: <a href="https://www.udacity.com/">Udacity</a></figcaption></figure><p>Notice that the graph on the right is not Gaussian. The main problem with this is the fact that Kalman Filters are not applicable anymore — the outputs are not Gaussian since we’re using the nonlinear function of <em>atan(x)</em>.</p><p>How can we fix this? Let’s take the first-order derivatives (first-order Taylor Series expansion) of the non-linear functions. Here’s how the output looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WvKBUBNrawAw4Mt4.png" /><figcaption>Source: <a href="https://www.udacity.com/">Udacity</a> Much better!</figcaption></figure><h3>Taylor series expansions</h3><p>Let’s talk a bit more about Taylor series expansions and how this works. Here’s how the general equation looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/0*JK4gCCBSkynGcWaL.png" /></figure><p>We can simply replace <em>f(x)</em> for our function, <em>h(x)</em>, and apply the same logic. Let’s say that we were applying the first-order Taylor expression to the nonlinear function, <em>arctan(x). </em>What would its derivative (linear function) be?</p><p>Let’s say that μ = 0 and <em>σ</em>=3.</p><p>So, the function that projects <em>x</em> to <em>z </em>(the measurement space) is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*IAJMPg3SR7AtQ-sL.png" /></figure><p>The partial derivative of this would be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*YnD-o11c-qJ2lRTi.png" /></figure><p>When we take the Taylor series equation of this, our linear approximation function output would be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*xWssc-uSuCIjiLUsW983UQ.png" /></figure><p>And that would equal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/199/0*8-0E7U91FudkzIuh.png" /></figure><p>This is exactly how we’d perform First-order Taylor Series equations. Remember that the radar measurements are multi-dimensional equations. How could we work with that?</p><h3>Multivariate Taylor Series</h3><p>The general equation for a multi-dimensional Taylor series expansion is known as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*zwoEO_yMn4jlwiH7dAQsog.png" /></figure><p>where <em>Df</em>(<em>a</em>) is known as the Jacobian matrix and <em>D²f</em>(<em>a</em>) known as the <a href="https://en.wikipedia.org/wiki/Hessian_matrix">Hessian matrix</a> where the former represents the first order and the ladder part representing the second-order derivatives of the multi-dimensional equations.</p><p>The full Taylor series expression would actually include higher-order terms as well (third, fourth, fifth derivatives, …).</p><p>For deriving the linear approximation of h(x), we can simply use the Jacobian matrix ( <em>Df(a)</em> ) since we will assume that <em>(x — a)</em> is a small value.</p><p>Let’s talk a bit more about the Jacobian matrix:</p><h3>The Jacobian Matrix</h3><p>The GIST of the Jacobian matrix is this: take all the partial derivatives of 2<em> variables and store them in a matrix, Hj.</em></p><p><em>Here’s how </em>Hj would look like for our case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*xUnFSZZCYSCbitP-yAUdGA.png" /></figure><p>Here’s how the Jacobian matrix would look like in our case — we would take the derivatives of p, φ, and ṗ with respect to pₓ, pᵧ, vₓ, and vᵧ since our <em>x</em> vector consists of these 4 measurements and we want to convert them into our measurement space, <em>z</em> (which would be the radar coordinates). That’s exactly why the matrix is 3x4–the 3 rows would be part of the measurement space <em>z </em>with the 4 columns being the x vector.</p><p>Here’s exactly how <em>Hj</em> would look like after calculating the Jacobian (partial derivatives) of <em>Hj</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*DKs80YB1blM7VbeCYo6PMg.png" /></figure><h3>Calculating our final equations for the Extended Kalman Filter</h3><p>Now that we’ve defined all the matrices and set up everything for making predictions and updating our measurements, we can now put the equations together.</p><h3>Prediction equations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*NHzf8SJnGiFu9l_u.png" /></figure><p>Remember that <em>f(x,u)</em> is the function for the Taylor series expansions where x is our input of pₓ, pᵧ, vₓ, and vᵧ and <em>u</em> is the noise in our sensor model.</p><p>P’ would be our estimate covariance matrix for uncertainty.</p><h3>Measurement update</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/0*9qeb9mRQ5Emw85Sp.png" /></figure><p>Remember that we’d have to use the Jacobian matrix Hj instead of the normal H matrix when calculating the system uncertainty projection and the Kalman gain.</p><p>Also, note that R is the input measurement noise. We’re also using the h function we defined above to help calculate the error.</p><h3>Evaluating the EKF model — RMSE</h3><p>Now that we’ve defined an Extended Kalman Filter, an effective comparison metric to see how our EKF model performed would need to be in place. We can use the Root Mean Squared Error (RMSE) to help do this exact task.</p><p>RMSE can help measure the deviation of the estimated state from the true state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*Ei4PENgUGgq9YLB0.png" /></figure><p>The difference between the ground truth state (denoted as yᵢ) and the predicted state gives an output known as the residual. It is then squared, averaged, and then square rooted to give the error metric. Note: the lower the error, the higher the accuracy.</p><p>I got this all up and running in Udacity’s simulator using C++. Take a look at the Github repository and video provided below to see the code in action.</p><p><a href="https://github.com/srianumakonda/Become-a-Self-Driving-Car-Engineer-Term-1/tree/main/CarND-Extended-Kalman-Filter-Project">srianumakonda/Become-a-Self-Driving-Car-Engineer-Term-1</a></p><p>Udacity simulator result:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-5RJNJ_cyuA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-5RJNJ_cyuA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-5RJNJ_cyuA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd707309aa6b7f52ea64161d155df4a1/href">https://medium.com/media/dd707309aa6b7f52ea64161d155df4a1/href</a></iframe><h3>Connect with me</h3><p><a href="/service/mailto:sri.anumakonda06@gmail.com">Email</a></p><p><a href="https://www.linkedin.com/in/srianumakonda/">Linkedin</a></p><p><a href="https://twitter.com/srianumakonda">Twitter</a></p><p><a href="http://srianumakonda.com/">Website</a></p><p>Thanks to <a href="https://medium.com/u/2929690a28fb">Udacity</a>…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32e0c08d9258" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/the-true-beauty-of-extended-kalman-filters-32e0c08d9258">The True Beauty of Extended Kalman Filters</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>