<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lando L on Medium]]></title>
        <description><![CDATA[Stories by Lando L on Medium]]></description>
        <link>https://medium.com/@Lando-L?source=rss-395c5e41bd1e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sVDbaYP97xdUgqbsdWJ4tQ.jpeg</url>
            <title>Stories by Lando L on Medium</title>
            <link>https://medium.com/@Lando-L?source=rss-395c5e41bd1e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:35:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Lando-L/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond the Basics: Reinforcement Learning with Jax — Part II: Developing  an exploitative…]]></title>
            <link>https://medium.com/data-science/beyond-the-basics-reinforcement-learning-with-jax-part-ii-developing-an-exploitative-9423cb6b2fa5?source=rss-395c5e41bd1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9423cb6b2fa5</guid>
            <category><![CDATA[multi-armed-bandit]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[jax]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <dc:creator><![CDATA[Lando L]]></dc:creator>
            <pubDate>Fri, 02 Jun 2023 14:03:17 GMT</pubDate>
            <atom:updated>2023-06-02T14:03:17.537Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond the Basics: Reinforcement Learning with Jax — Part II: Developing an Exploitative Alternative to A/B Testing</h3><h4>A practical introduction to multi-arm bandits and the exploration-exploitation dilemma</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zhvIYqJjKHOK4yn4" /><figcaption>Photo by <a href="https://unsplash.com/@carltraw?utm_source=medium&amp;utm_medium=referral">Carl Raw</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In our last blog post, we explored the Reinforcement Learning paradigm, delving into its core concepts of finite Markov Decision Processes, Policies, and Value Functions. Now, we are ready to apply our newfound knowledge and discover an alternative approach to traditional A/B testing via multi-arm bandits.</p><h3>Course links</h3><ul><li><a href="https://medium.com/@Lando-L/beyond-the-basics-reinforcement-learning-with-jax-part-i-introduction-and-core-concepts-4f85f3478f5">Part I: Introduction and Core Concepts</a></li><li><strong>Part II: Developing an exploitative alternative to A/B testing</strong></li></ul><h3>Is A/B testing significantly overrated?</h3><p>Think back to the last time you scrolled through your video streaming service looking for a new movie to watch. If you did not already know what to look for, you may have been affected by <em>the paradox of choice</em>¹. Having such an array of potentially good movies can make it difficult to make informed decisions. Instead, we often rely on simpler decisions that can be made almost instantly. As a result, we tend to compare videos based on their thumbnails and titles. Knowing of this effect, video production and streaming companies strive to optimise thumbnails and titles in order to increase their movies’ click rates. Without prior knowledge of the audience’s preferences, the process of creating engaging thumbnails and titles becomes one of <em>trial-and-error</em>. (I hope this rings a bell by now!)</p><h4>The traditional approach</h4><p>Let us imagine ourselves in the shoes of the decision-maker tasked with choosing the right thumbnail and title for a newly-published movie on the streaming website. In addition to the original movie thumbnail and title, we are given a set of proposals consisting of two thumbnails and two titles. The traditional data-driven approach in this scenario would be to run an A/B test and compare the click rates of the original version with that of the proposed versions. Although the advantage of A/B tests is that, when given enough samples, it can identify whether or not the measured difference in click rate is statistically significant, it limits us to compare only two variants simultaneously and often requires extensive amounts of samples to determine its result. Furthermore, in the case one of the variants is performing drastically better than the other, we are still forced to serve the worse performing variant to half our customers until the end of the experiment, potentially leading to money losses.</p><h4>The Reinforcement Learning approach</h4><p>Alternatively, we can set up a <em>multi-arm bandit</em> (MAB) experiment. MABs are a simplification of Reinforcement Learning (RL) problems, as they can be defined entirely by a set of actions and a reward function. This makes them effectively like finite <em>Markov-Decision-Processes</em> (MDPs) with only one state. Unlike MDPs, the actions in MABs are independent of each other, meaning that at each time step, the same set of actions will provide the same reward distribution. Therefore, if we know the exact reward distribution of the available actions, we can simply use a greedy approach to choose the action with the highest payout. In contrast, MDPs require us to sometimes take ‘suboptimal’ actions to reach a highly rewarding state. However, the problems that can be modelled with MABs are fewer than those that can be modelled with MDPs. For example, in the music practice environment from the <a href="https://medium.com/@Lando-L/beyond-the-basics-reinforcement-learning-with-jax-part-i-introduction-and-core-concepts-4f85f3478f5">last post</a>, it was necessary to introduce states to model our friend’s mood, something which cannot be done with MABs.</p><p>We can model the challenge from our video streaming platform example by defining the actions and reward function as follows:</p><ul><li>Every time a user visits the website, we must choose which version to display. Our options consist of showing the original movie version, or one of the four variations created by combining the two thumbnails and two titles.</li><li>Once we selected the variant to display, the user has the option to watch or not watch our film. Consequently, our reward function is binary, yielding a reward of zero if the user choses not to view the movie and a reward of one if they decide to watch the movie.</li></ul><p>The advantages of the MAB approach compared with traditional A/B testing are numerous; it allows for an unlimited number of variants to be tested simultaneously, dynamically lowers the frequency of poorer performing variants, and converges with fewer samples — all of which lead to cost savings. The downside is that it does not provide the statistical significance of a traditional A/B test; however, when the implications of the experiment are not affecting people’s wellbeing (as is the case in most instances), statistical significance is not strictly necessary.</p><h3>Let me see some code</h3><p>Finally, we will start writing code. For the code examples of this course, we will be using Python as our programming language, and will largely use <a href="https://jax.readthedocs.io/en/latest/index.html">Jax</a> as our Machine Learning (ML) framework. Developed by Google, Jax is a Python library that is specifically designed for ML research. Unlike Tensorflow and PyTorch, it is built with the functional programming (FP) paradigm, making it highly composable and promoting the concept of pure functions that have no side effects. This means that all state changes, such as parameter updates or splitting of random generators, must be done explicitly. Although this may require more lines of code than their object-oriented programming (OOP) equivalent, it gives developers full control over state changes, leading to increased understanding of the code and fewer surprises.</p><p>(The full code is available as a Jupyter notebook on <a href="https://github.com/Lando-L/reinforcement-learning-with-jax/blob/91404625fd22fe62cb3e644c0ee8d51dd6bf2f24/2-developing-an-exploitative-alternative-to-a-b-testing.ipynb">GitHub</a> and <a href="https://www.kaggle.com/code/landol/exploitative-alternative-to-a-b-testing">Kaggle</a>.)</p><h4>Implementing the environment</h4><p>As the first step, we implement the environment for our video streaming platform challenge. At a high-level, whenever someone visits the platform to scroll through the available movie options, the environment needs to ask the agent which thumbnails and titles to display, and then communicate to the agent which movie the visitor chose. The complexity of this task will depend on the maturity of the platform’s architecture, which could range from simply changing a few lines of code to developing an entirely new service.</p><p>For the purpose of this course, we will keep the environment simple by implementing it as a three-step process:</p><ol><li>Asking the agent which of the five variants it wishes to display.</li><li>Simulating the visitor’s choice.</li><li>Communicating the agent its reward for the decision.</li></ol><p>Since we are using Jax as our main ML framework, we need to import Jax and the three modules <em>numpy,</em> <em>lax</em> and <em>random</em>²:</p><pre>import jax<br><br># Numpy API with hardware acceleration and automatic differentiation<br>from jax import numpy as jnp<br><br># Low level operators<br>from jax import lax<br><br># API for working with pseudorandom number generators<br>from jax import random</pre><p>Next, we set the constants of our environment, consisting of a random seed to ensure replicability, the number of visitors we want to simulate, and the expected click rates. It is important to note that, in the real world, the click rates are considered unknown. Since we are not running an actual experiment, we must simulate the visitors’ click behaviour. To do this, we define different click rates for the five variants to imitate the visitors’ preference, with the original variant having a click rate of 4.2%, and the four variants having click rates of 3%, 3.5%, 3.8%, and 4.5%, respectively.</p><pre># Random seed to make our experiment replicable <br>SEED = 42<br><br># Number of visitors we want to simulate<br>NUM_VISITS = 10000<br><br># Expected click rates for the five variants with the<br>CLICK_RATES = [0.042, 0.03, 0.035, 0.038, 0.045]</pre><p>Finally, we define a generic function that simulates a user visit, or a single step in our environment. This function comprises three steps and is close to the high-level implementation we set out earlier:</p><ol><li>Executing the agent’s policy to determine which variant to display to the user.</li><li>Randomly simulating whether the user clicked on the movie or not.</li><li>Updating the agent’s parameters based on the variant shown and the associated reward.</li></ol><pre>def visit(state, timestep, click_rates, policy_fn, update_fn):<br>    &quot;&quot;&quot;<br>    Simulates a user visit.<br>    &quot;&quot;&quot;<br><br>    # Unpacking the environment state into<br>    # the agent&#39;s parameters and the random number generator<br>    params, rng = state<br><br>    # Splitting the random number generator<br>    next_rng, policy_rng, user_rng = random.split(rng, num=3)<br><br>    # Selecting the variant to show the user, based on<br>    # the given policy, the agent&#39;s parameters, and the current timestep<br>    variant = policy_fn(params, timestep, policy_rng)<br><br>    # Randomly simulating the user click, based on<br>    # the variant&#39;s click rate<br>    clicked = random.uniform(user_rng) &lt; click_rates[variant]<br><br>    # Calculating the agent&#39;s updated parameters, based on<br>    # the current parameters, the selected variant,<br>    # and whether or not the user clicked<br>    next_params = update_fn(params, variant, clicked)<br>    <br>    # Returning the updated experiment state (params and rng) and<br>    # whether or not the user clicked<br>    return (next_params, next_rng), clicked</pre><p>Before we continue, let us discuss the function signature. If we fix the parameters <em>click_rates</em>, <em>policy_fn</em>, and <em>update_fn</em>, the <em>visit</em> function takes the environment state and the current timestep as its parameters and returns a tuple containing the next environment state and a boolean value that encodes the binary reward. In Haskell notation the function signature would looks like this:</p><pre>state -&gt; timestep -&gt; (state, Bool)</pre><p>Hence, to simulate the <em>n-th </em>step in our environment we pass the function the <em>n-th</em> state and <em>n-th </em>timestep<em>, </em>and we receive the <em>(n+1)-th</em> state and the <em>n-th</em> reward. For the <em>(n+1)-th</em> step, we use the same function, by passing it the <em>(n+1)-th</em> state returned by the last function call and the time step <em>n+1</em>.</p><pre># Initialising the state<br>s0 = ...<br><br># Simulating the time steps 0, 1, and 2<br>s1, r0 = visit(s0, 0)<br>s2, r1 = visit(s1, 1)<br>s3, r2 = visit(s2, 2)</pre><p>Having to pass a state and timestep parameter to every call of the <em>visit</em> function may seem cumbersome to people accustomed to OOP. However, in this example, using a pure function implementation for the environment has several advantages over an object-oriented approach. Firstly, it explicitly states which parameters the environment depends on, thus eliminating any hidden global variables that may influence the outcome. Secondly, it makes it easier to test the environment with various states and timesteps without having to set and read the environment’s internal state. Lastly, we will discover a useful function from the Jax library, which offers state management, thus drastically reducing the code needed at the call-side.</p><h4>Implementing the policies</h4><p>With the environment in place, we can now implement our decision-making process or policies. We have already identified an optimal policy for MABs. Given the action-value distribution, the best action to take is the one with the highest expected payout, which is referred to as <em>exploitation</em>. However, since the actual action-value distribution is unknown to us, we must try out all available options at least once in order to estimate the distribution, a process often referred to as <em>exploration</em>. The delicate balance between exploration and exploitation is a recurrent theme in RL and will be discussed in more detail throughout the course.</p><p>The three policies we will cover in this blog post are the <em>epsilon-greedy</em> policy, the <em>Boltzmann </em>policy, and the <em>upper-confidence-bound</em> policy. All of which are <em>action-value methods</em>, meaning they explicitly estimate the values of all actions and base their decision making on these estimates. At the end we will cover a bonus policy, which is based on the<em> Thompson sampling </em>heuristic and is considered a <em>bayesian method</em>.</p><h4>Action-value methods</h4><p>The simplest way for an agent to estimate action-values is to average the rewards it has received for each variant so far. Here, <em>Q </em>denotes the agent’s action-value estimate of variant <em>a </em>at timestep<em> t.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*fnGHl6VAkSoHJWVCUu-T9w.png" /></figure><p>Rather than re-calculating the average every round, we can implement action-value estimation in an incremental fashion. Where <em>Q</em> stores the current action-value estimate of each variant <em>a, N</em> counts how often <em>a</em> was shown, and <em>R</em> denotes the reward received at timestep <em>t</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*c8-4d0ppcHU21b46m0cgqA.png" /></figure><p>Let us implement two functions to work with action-value estimates. The first function initialises the lookup tables <em>Q</em> and <em>N</em> for each variant <em>a</em>, setting the estimates of all variants to an optimistic initial value of 1 (or a click rate of 100%). The second function updates <em>Q</em> and <em>N</em> according to the incremental update definition described above.</p><pre>def action_value_init(num_variants):<br>    &quot;&quot;&quot;<br>    Returns the initial action values<br>    &quot;&quot;&quot;<br><br>    return {<br>        &#39;n&#39;: jnp.ones(num_variants, dtype=jnp.int32),<br>        &#39;q&#39;: jnp.ones(num_variants, dtype=jnp.float32)<br>    }<br><br>def action_value_update(params, variant, clicked):<br>    &quot;&quot;&quot;<br>    Calculates the updated action values<br>    &quot;&quot;&quot;<br><br>    # Reading n and q parameters of the selected variant<br>    n, q = params[&#39;n&#39;][variant], params[&#39;q&#39;][variant]<br><br>    # Converting the boolean clicked variable to a float value<br>    r = clicked.astype(jnp.float32)<br><br>    return {<br>        # Incrementing the counter of the taken action by one<br>        &#39;n&#39;: params[&#39;n&#39;].at[variant].add(1),<br><br>        # Incrementally updating the action-value estimate<br>        &#39;q&#39;: params[&#39;q&#39;].at[variant].add((r - q) / n)<br>    }</pre><p>We chose to implement the estimate action-value initialisation and update using a functional approach, similar to the environment function implementation. Jax arrays, unlike numpy arrays, are immutable and therefore cannot be updated in-place; instead, each update returns a new copy with the applied changes, while the original array remains unchanged.</p><h4>Epsilon-greedy policy</h4><p>The epsilon-greedy policy defines a stochastic approach to balancing the exploration and exploitation trade-off. Guided by the hyperparameter <em>ε</em>, it randomly decided between selecting the variant <em>a</em> with the highest action-value <em>Q</em> or selecting a uniformly random variant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*Z1dQldLwzjwFS-PpY5gUHw.png" /></figure><p>In Jax we can define conditional policies using the <em>cond </em>function. It takes a predicate, two functions, and a variable amount of arguments. Depending on the result of the predicate <em>cond</em> employs one of the two functions passing it the given arguments.</p><pre>def epsilon_greedy_policy(params, timestep, rng, epsilon):<br>    &quot;&quot;&quot;<br>    Randomly selects either the variant with highest action-value,<br>    or an arbitrary variant.<br>    &quot;&quot;&quot;<br><br>    # Selecting a random variant<br>    def explore(q, rng):<br>        return random.choice(rng, jnp.arange(len(q)))<br>    <br>    # Selecting the variant with the highest action-value estimate<br>    def exploit(q, rng):<br>        return jnp.argmax(q)<br>    <br>    # Splitting the random number generator <br>    uniform_rng, choice_rng = random.split(rng)<br>    <br>    # Deciding randomly whether to explore or to exploit<br>    return lax.cond(<br>        random.uniform(uniform_rng) &lt; epsilon,<br>        explore,<br>        exploit,<br>        params[&#39;q&#39;],<br>        choice_rng<br>    )</pre><h4>Boltzmann policy</h4><p>The Boltzmann or softmax policy is similar to the epsilon-greedy policy in that it is a stochastic policy based on action-value estimates. This approach randomly samples a variant <em>a</em> from the probability distribution that results from applying the softmax function to the action-value estimates <em>Q</em>. The exploration-exploitation trade-off can be controlled through the temperature hyperparameter <em>τ</em>, where lower temperatures favour exploitation and higher temperatures promote exploration. The probability <em>P </em>of each variant to be selected is defined by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*09DRpOd5XgA0jAmMJkB_2g.png" /></figure><p>In Jax, this can be easily implemented by utilizing the <em>choice</em> function from the random module, parameterised by the softmax function applied to the action-value estimates.</p><pre>def boltzmann_policy(params, timestep, rng, tau):<br>    &quot;&quot;&quot;<br>    Randomly selects a variant proportional to the current action-values<br>    &quot;&quot;&quot;<br><br>    return random.choice(<br>        rng,<br>        jnp.arange(len(params[&#39;q&#39;])),<br>        # Turning the action-value estimates into a probability distribution<br>        # by applying the softmax function controlled by tau<br>        p=jax.nn.softmax(params[&#39;q&#39;] / tau)<br>    )</pre><h4>Upper-Confidence-Bound policy</h4><p>We will now discuss a policy with a deterministic approach to balancing exploration and exploitation. Like the formerly discussed policies, it encourages exploitation by prioritising variants with high action-value estimates. However, instead of relying on stochasticity for exploration, it leverages a heuristic that encourages the selection of variants with low selection counts.</p><p>The heuristic accomplishes this by <em>remaining optimistic in the face of uncertainty</em>. Meaning, every variant is given the benefit of the doubt of being better than our current action-value estimate. During the experiment, each time a variant is selected and a real reward is observed, we get more confident in our action-value estimate and decrease the benefit of the doubt for that variant.</p><p>Formally, we define this optimistic guess as the variants’ <em>upper-confidence-bound</em> (UCB) which is scaled by the confidence hyperparameter <em>c</em> and added it to the current action-value estimate. Finally, we select the variant with the highest sum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*aNRcMdumQskWpCDZEsmJYA.png" /></figure><p>The UCB policy is the first we discovered that rewards exploration as well as exploitation:</p><ul><li>Given two variants with the same action-value estimate <em>Q</em>, we will select the variant with a lower selection count <em>N</em>.</li><li>Given two variants with the same selection count <em>N</em>, we will select the variant with a higher action-value estimate <em>Q</em>.</li></ul><p>To ensure a consistent function definition for all policies, the UCB policy takes a random number generator parameter, even though it is a deterministic algorithm.</p><pre>def upper_confidence_bound_policy(params, timestep, rng, confidence):<br>    &quot;&quot;&quot;<br>    Selects the variant with highest action-value plus upper confidence bound<br>    &quot;&quot;&quot;<br><br>    # Read n and q parameters<br>    n, q = params[&#39;n&#39;], params[&#39;q&#39;]<br><br>    # Calculating each variant&#39;s upper confidence bound<br>    # and selecting the variant with the highest value<br>    return jnp.argmax(q + confidence * jnp.sqrt(jnp.log(timestep) / n))</pre><h4>Bayesian methods</h4><p>The discussed action-value methods make point estimates for the unknown click rates of our five variants. However, we now adopt a more Bayesian approach and treat the variants’ click rates as a set of independent random variables. Specifically, we define our current belief of a variants click rate <em>C </em>by modelling it as a random variable following a <em>beta</em> distribution².</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*iOBLqmA5o1hfTLpDcorceg.png" /></figure><p>The beta distribution is characterized by two parameters, <em>a</em> and <em>b</em>, which can be interpreted as the number of times variant <em>i</em> was clicked versus the number of times it was not clicked when shown. When comparing bayesian methods with action-value methods, we can use the expected value <em>E</em> of the random variable <em>C</em> to define our best guess, which can be determined by dividing the number of times a variant was clicked by the number of times it was shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*qv04dpczMeW-BQMD1jDxdQ.png" /></figure><p>We define two functions to work with beta distributions, analogous to the action-value methods. The first initialises a uniform beta prior for each variant, while the second function calculates the posterior beta distribution by incrementing either the <em>a</em> or the <em>b</em> parameter by one.</p><pre>def beta_init(num_variants):<br>    &quot;&quot;&quot;<br>    Returns the initial hyperparameters of the beta distribution<br>    &quot;&quot;&quot;<br><br>    return {<br>        &#39;a&#39;: jnp.ones(num_variants, dtype=jnp.int32),<br>        &#39;b&#39;: jnp.ones(num_variants, dtype=jnp.int32)<br>    }<br><br>def beta_update(params, variant, clicked):<br>    &quot;&quot;&quot;<br>    Calculates the updated hyperparameters of the beta distribution<br>    &quot;&quot;&quot;<br><br>    # Incrementing alpha by one<br>    def increment_alpha(a, b):<br>        return {&#39;a&#39;: a.at[variant].add(1), &#39;b&#39;: b}<br>    <br>    # Incrementing beta by one<br>    def increment_beta(a, b):<br>        return {&#39;b&#39;: b.at[variant].add(1), &#39;a&#39;: a}<br>    <br>    # Incrementing either alpha or beta<br>    # depending on whether or not the user clicked<br>    return lax.cond(<br>        clicked,<br>        increment_alpha,<br>        increment_beta,<br>        params[&#39;a&#39;],<br>        params[&#39;b&#39;]<br>    )</pre><h4>Thompson sampling policy</h4><p>The TS policy is based on a two-step heuristic, which works by drawing random click rate samples from our beta distribution, and then selecting the variant with the highest sample click rate. The feedback we receive is then instantaneously incorporated into the beta distribution of that variant, narrowing the distribution closer to the actual click rate.</p><p>Like the UCB policy, this approach rewards both exploration and exploitation:</p><ul><li>Given two variants with the same mean, the variant with a higher variance has a higher chance of being selected, since it has a broader distribution and will lead more often to higher action-values when sampled.</li><li>Given two variants with the same variance, the variant with the higher mean is selected more often, as it is more likely to sample a greater action-value.</li></ul><p>For the implementation of the TS policy we use Jax’s <em>random</em> module to sample random click rates based on the variants’ beta distribution and then select the variant with the highest sample.</p><pre>def thompson_policy(params, timestep, rng):<br>    &quot;&quot;&quot;<br>    Randomly sampling click rates for all variants<br>    and selecting the variant with the highest sample<br>    &quot;&quot;&quot;<br><br>    return jnp.argmax(random.beta(rng, params[&#39;a&#39;], params[&#39;b&#39;]))</pre><h4>Implementing the evaluation</h4><p>With the environment and policies in place, we can finally conduct our experiment and compare the results. Before we continue, I want to highlight that this experiment is intended to demonstrate how the algorithms work, not to empirically evaluate their performance. In order to execute the following implementations, we need to import the <em>partial</em> function from Python’s <em>functools</em> library and <em>pyplot </em>from<em> matplotlib</em>:</p><pre>from functools import partial<br>from matplotlib import pyplot as plt</pre><p>The <em>evaluate</em> function is responsible for executing the <em>visit</em> function, as guided by the set of constants, the parameter initialisation and update functions, and the policy. The output of the evaluation is the final environment state, which includes the policy’s final parameters and final random number generator state, as well as the click history. We leverage Jax’s <em>scan</em> function to ensure that the experiment state is carried over and user clicks are accumulated. Moreover, <em>just-in-time</em> (JIT) compilation is employed to optimise performance, while <em>partial</em> is utilised to fix the <em>click_rate</em>, <em>policy_fn</em>, and <em>update_fn </em>parameters of the <em>visit</em> function so it matches the expected signature.</p><pre>def evaluate(policy_fn, init_fn, update_fn):<br>    &quot;&quot;&quot;<br>    Simulating the environment for NUM_VISITS users<br>    while accumulating the click history<br>    &quot;&quot;&quot;<br><br>    return lax.scan(<br>        # Compiling the visit function using just-in-time (JIT) compilation<br>        # for better performance<br>        jax.jit(<br>            # Partially applying the visit function by fixing<br>            # the click_rates, policy_fn, and update_fn parameters <br>            partial(<br>                visit,<br>                click_rates=jnp.array(CLICK_RATES),<br>                policy_fn=jax.jit(policy_fn),<br>                update_fn=jax.jit(update_fn)<br>            )<br>        ),<br>        <br>        # Initialising the experiment state using<br>        # init_fn and a new PRNG key<br>        (init_fn(len(CLICK_RATES)), random.PRNGKey(SEED)),<br>        <br>        # Setting the number of steps in our environment<br>        jnp.arange(1, NUM_VISITS + 1)<br>    )</pre><p>The <em>regret</em> function is the last component of our evaluation. In RL lingo, regret is defined as the amount of reward we miss out on by taking a suboptimal action, and can only be calculated when the optimal action is known. Given the click history, our <em>regret</em> function calculates the regret of the action taken for every step of the environment.</p><pre>def regret(history):<br>    &quot;&quot;&quot;<br>    Calculates the regret for every action in the environment history<br>    &quot;&quot;&quot;<br><br>    # Calculating regret with regard to picking the optimal (0.045) variant<br>    def fn(acc, reward):<br>        n, v = acc[0] + 1, acc[1] + reward<br>        return (n, v), 0.045 - (v / n)<br>    <br>    # Calculating regret values over entire history<br>    _, result = lax.scan(<br>        jax.jit(fn),<br>        (jnp.array(0), jnp.array(0)),<br>        history<br>    )<br>    <br>    return result</pre><p>Next we run the evaluation for all four policies and visualise the regret. Note that the hyperparameters for the policies have not been fine-tuned, instead they are set to generic default values that are suitable for a wide variety of MAB problems.</p><pre># Epsilon greedy policy<br>(epsilon_greedy_params, _), epsilon_greedy_history = evaluate(<br>    policy_fn=partial(epsilon_greedy_policy, epsilon=0.1),<br>    init_fn=action_value_init,<br>    update_fn=action_value_update<br>)<br><br># Boltzmann policy<br>(boltzmann_params, _), boltzmann_history = evaluate(<br>    policy_fn=partial(boltzmann_policy, tau=1.0),<br>    init_fn=action_value_init,<br>    update_fn=action_value_update<br>)<br><br># Upper confidence bound policy<br>(ucb_params, _), ucb_history = evaluate(<br>    policy_fn=partial(upper_confidence_bound_policy, confidence=2),<br>    init_fn=action_value_init,<br>    update_fn=action_value_update<br>)<br><br># Thompson sampling policy<br>(ts_params, _), ts_history = evaluate(<br>    policy_fn=thompson_policy,<br>    init_fn=beta_init,<br>    update_fn=beta_update<br>)<br><br># Visualisation<br>fig, ax = plt.subplots(figsize=(16, 8))<br><br>x = jnp.arange(1, NUM_VISITS + 1)<br><br>ax.set_xlabel(&#39;Number of visits&#39;)<br>ax.set_ylabel(&#39;Regret&#39;)<br><br>ax.plot(x, jnp.repeat(jnp.mean(jnp.array(CLICK_RATES)), NUM_VISITS), label=&#39;A/B Testing&#39;)<br>ax.plot(x, regret(epsilon_greedy_history), label=&#39;Espilon Greedy Policy&#39;)<br>ax.plot(x, regret(boltzmann_history), label=&#39;Boltzmann Policy&#39;)<br>ax.plot(x, regret(ucb_history), label=&#39;UCB Policy&#39;)<br>ax.plot(x, regret(ts_history), label=&#39;TS Policy&#39;)<br><br>plt.legend()<br>plt.show()</pre><p>The resulting graph is shown below, plotting the regret of our policies over the number of visits. We can clearly see that all policies outperform a hypothetical A/B testing scenario in terms of regret. The epsilon-greedy and TS policies appear to perform slightly better than the boltzmann and UCB policies in this particular scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xUbqe3eOZKM0fzY3YsuOPw.png" /><figcaption>Visualisation created by the author.</figcaption></figure><h3>Beyond the regret</h3><p>When conducting machine learning experiments, it is generally advised to define a set of metrics that measure qualitative performance aside from the objective function. For RL experiments, however, it is often not so straightforward to identify suitable metrics. In our case, I chose we dig deeper into the final parameters of our policies and compare the action-value estimation to the actual click rates and how often each variant was selected under the different policies.</p><h4>Action-value estimates</h4><p>We compare the accuracy of the policies’ action-value estimates by calculating the <em>root mean squared error</em> (RMSE) against the ground truth click rates, as seen in the table below.</p><pre>|      Name      | Original |   V1   |   V2   |   V3   |   V4   |  RMSE  |<br>|----------------|----------|--------|--------|--------|--------|--------|<br>| Ground truth   |   0.0420 | 0.0300 | 0.0350 | 0.0380 | 0.0450 | 0.0000 |<br>| Epsilon greedy |   0.0420 | 0.0367 | 0.0378 | 0.0256 | 0.0375 | 0.0072 |<br>| Boltzmann      |   0.0397 | 0.0291 | 0.0383 | 0.0346 | 0.0449 | 0.0024 |<br>| UCB            |   0.0399 | 0.0259 | 0.0247 | 0.0390 | 0.0518 | 0.0060 |<br>| TS             |   0.0390 | 0.0425 | 0.0370 | 0.0393 | 0.0441 | 0.0059 |</pre><ul><li>Surprisingly, despite performing well overall, it turns out that the epsilon-greedy policy has prematurely honed in on exploiting the second-best variant, while the other policies have correctly identified the last variant as the optimal one.</li><li>The Boltzmann policy performed the best in terms of predicting the variants’ click rates.</li><li>The UCB policy and TS policy performed comparably well. While UCB appears to overestimate the value of V4 and underestimate V2, TS seems to overestimate V1 and underestimate the original version.</li></ul><h4>Variant counter</h4><p>The second characteristic I want to discuss is how often each variant was selected under the different policies. Here, we simply compare the absolute variant counts of our experiment with 10,000 visitors.</p><pre>|      Name      | Original |  V1  |  V2  |  V3  |  V4  |<br>|----------------|----------|------|------|------|------|<br>| Epsilon greedy |     8334 |  409 |  291 |  352 |  614 |<br>| Boltzmann      |     1991 | 1998 | 2012 | 2024 | 1984 |<br>| UCB            |     2079 | 1701 | 1661 | 2051 | 2508 |<br>| TS             |     1901 |  963 | 1324 |  735 | 5077 |</pre><ul><li>The epsilon greedy algorithm was found to exploit too early, as it selected the original version over 83% of the time.</li><li>The Boltzmann policy was remarkably accurate at predicting the variants’ click rates, likely due to its uniform exploration of all variants.</li><li>The TS policy was found to have a lower regret value than the UCB policy, likely due to its more extensive exploitation of the optimal variant.</li></ul><h4>Conclusion</h4><p>Upon further examination of our observations, we have identified several areas for improvement in our current experiment setup.</p><ul><li>The epsilon-greedy policy appears to be overly exploiting; as a result, we should increase the epsilon hyperparameter to encourage a more comprehensive exploration.</li><li>The Boltzmann policy appears to have explored much more than it has exploited, resulting in accurate predictions of the click rates, albeit missing out on rewards. This may be an indication that its temperature hyperparameter should be increased.</li><li>The UCB and TS policies performed well, but there is still room for improvement. We can fine-tune the confidence hyperparameter and the initial beta prior respectively to further optimize their performance.</li></ul><p>In addition to the suggested hyperparameter changes, I encourage the interested reader to explore more sophisticated improvements as an exercise:</p><ul><li>Utilizing a variety of random seeds and averaging the outcomes to reduce bias in the experiment results.</li><li>Implementing a dynamic scheduler to gradually reduce exploration over time, such as decreasing the epsilon or confidence parameter in epsilon-greedy and UCB policies, and increasing the temperature parameter in the Boltzmann policy.</li></ul><h3><strong>What’s next?</strong></h3><p>MAB are an incredibly versatile tool for solving a wide range of problems. In this post, we discussed four MAB policies that could be used to solve the challenge of the video streaming platform scenario. Other MAB use cases include ad selection for online advertisement, prioritisation of query results for search engines, and allocation of resources for different projects.</p><p>Moreover, MABs can be further extended to incorporate additional information. In our example, the decision-making process could be improved if we had access to more contextual data, such as individual user preferences and feature vectors for each variant. Such problems are referred to as contextual bandit problems, which we will explore further in later posts focusing on approximate solution methods.</p><p>[1] Schwartz, Barry, and Barry Schwartz. “The paradox of choice: Why more is less.” New York: Ecco, 2004.</p><p>[2] We chose the Beta distribution to model the click rates, as it is the conjugate prior of the Bernoulli distribution which is our likelihood function of the visitor clicking or not clicking the shown variant. For a better understanding of probability distributions and Bayesian inference, I recommend consulting Christopher M. Bishop and Nasser M. Nasrabadi’s book “Pattern Recognition and Machine Learning”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9423cb6b2fa5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/beyond-the-basics-reinforcement-learning-with-jax-part-ii-developing-an-exploitative-9423cb6b2fa5">Beyond the Basics: Reinforcement Learning with Jax — Part II: Developing  an exploitative…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Basics: Reinforcement Learning with Jax — Part I: Introduction and Core Concepts]]></title>
            <link>https://medium.com/data-science/beyond-the-basics-reinforcement-learning-with-jax-part-i-introduction-and-core-concepts-4f85f3478f5?source=rss-395c5e41bd1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f85f3478f5</guid>
            <category><![CDATA[jax]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Lando L]]></dc:creator>
            <pubDate>Tue, 02 May 2023 14:54:20 GMT</pubDate>
            <atom:updated>2023-05-11T18:30:17.405Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond the Basics: Reinforcement Learning with Jax — Part I: Introduction and Core Concepts</h3><h4>An overview of Reinforcement Learning fundamentals: Markov Decision Processes, Policies, and Value Functions</h4><figure><img alt="Interface of an agent with its environment" src="https://cdn-images-1.medium.com/max/1024/1*LH6oBldCTb5rkcgEIeq9cg.jpeg" /><figcaption>Diagram created by the author using Miro.</figcaption></figure><p>In this series of blog posts, I am excited to share with you my passion for the Reinforcement Learning (RL) paradigm. My mission is to provide an in-depth exploration of RL, combining theoretical knowledge with practical examples that tackle real-world problems.</p><h3>Course details</h3><h4>Who is this course intended for?</h4><p>This course is for anyone who wishes to learn about RL. It is recommended that you have some prior coding experience in Python and be familiar with the use of Jupyter Notebooks. Furthermore, it would be beneficial to have a basic understanding of neural networks when tackling the chapters in the second part of this course. If you lack prior knowledge of neural networks, you can still use them and treat them as black box function approximators; however, I suggest taking the time to learn more about them before progressing through the latter chapters.</p><h4>Why should I care about Reinforcement Learning?</h4><p>RL is an incredibly exciting field that has seen exponential research growth in recent years. Despite the successes that have been achieved, bridging the gap between academia and industry is not as straightforward as is the case of other Machine Learning fields. RL is often viewed as too impractical for commercial applications, but I am here to dispel that notion and arm you with practical knowledge of RL methods to bolster your problem-solving toolbox.</p><h4>What will I learn?</h4><p>This is the first in a series of blog posts where you will:</p><ul><li>Learn to identify use cases of RL and model your real-world scenarios to fit the RL paradigm</li><li>Become familiar with the various RL algorithms and understand their advantages and limitations</li><li>Understand the principles of exploration and exploitation and how to balance them for optimal performance</li><li>Analyse and interpret the results of RL experiments</li></ul><p>Here is a sneak peak of the techniques we will cover:</p><ul><li>Multi-arm bandits</li><li>Dynamic Programming</li><li>Monte Carlo Methods</li><li>Temporal Difference Learning</li><li>Deep Q-Networks</li><li>Policy Gradient Methods</li><li>Actor-Critic Methods</li></ul><h3>Introduction</h3><p>Some of the most significant breakthroughs in artificial intelligence are inspired by nature and the RL paradigm is no exception. This simple yet powerful concept is closest to how we humans learn and can be seen as an essential element of what we would expect from an artificial general intelligence: Learning through trial and error. This approach to learning teaches us about cause and effect, and how our actions impact our environment. It teaches us how our behaviour can either harm or benefit us, and enables us to develop strategies to achieve our long-term goals.</p><h4>What is RL?</h4><p>The RL paradigm is a powerful and versatile machine learning approach that enables decision makers to learn from their interactions with the environment. It draws from a wide range of ideas and methodologies for finding an optimal strategy to maximize a numerical reward. With a long history of connections to other scientific and engineering disciplines, research in RL is well-established. However, while there is a wealth of academic success, practical applications of RL in the commercial sphere remain rare. The most famous examples of RL in action are computers achieving super-human level performance on games such as chess and Go, as well as on titles like Atari and Starcraft. In recent years, however, we have seen a growing number of industries adopt RL methods.</p><h4>How is it used today?</h4><p>Despite the low level of commercial adoption of RL, there are some exciting applications in the field of:</p><ul><li>Health: Dynamic treatment regime; automated diagnosis; drug discovery</li><li>Finance: Trading; dynamic pricing; risk management</li><li>Transportation: Adaptive traffic control; autonomous driving</li><li>Recommendation: Web search; news recommendation; product recommendation</li><li>Natural Language Processing: text summarization; question answering; machine translation; dialog generation</li></ul><h3>Sometimes less is more</h3><p>A good way to gain an understanding of RL use cases is to consider an example challenge. Let us imagine we are trying to help our friend learn to play a musical instrument. Each morning, our friend tells us how motivated they feel and how much they have learned during yesterday’s practice, and asks us how they should proceed. For reasons unknown to us, our friend has a limited set of studying choices: Taking a day off, practicing for one hour, or practicing for three hours.</p><p>After observing our friend’s progress, we have noticed a few interesting characteristics:</p><ol><li>It appears that the progress our friend is making is directly correlated with the amount of hours they practice.</li><li>Consistent practice sessions make our friend progress faster.</li><li>Our friend does not do well with long practicing sessions. Every time we instructed them to study for three hours, the next day they felt tired and unmotivated to continue.</li></ol><p>From our observations, we have created a graph modeling their learning progress using state machine notation.</p><figure><img alt="Visualisation of the music practice environment using state machine notation" src="https://cdn-images-1.medium.com/max/1024/1*te8K5mRK0kkEOWSR8yP00w.jpeg" /><figcaption>Diagram created by the author using Miro.</figcaption></figure><p>Let us discuss again our findings based on our model:</p><ol><li>Our friend has three distinct emotional states: neutral, motivated, and demotivated.</li><li>On any given day, they can choose to practice for zero, one, or three hours, except when they are feeling demotivated — in which case studying for zero hours (or not studying) is their only available option.</li><li>Our friend’s mood is predictive: In the neutral state, practicing for one hour, will make them feel motivated the following day, while practicing for three hours will leave them feeling demotivated und not practicing at all will keep them in a neutral state. Conversely, in the motivated state, one hour of practice will maintain our friend’s motivation, while three hours of practice will demotivate them and no practice at all will leave them feeling neutral. Lastly, in the demotivated state, our friend will refrain from studying altogether, resulting in them feeling neutral the next day.</li><li>Their progress is heavily influenced by their mood and the amount of practice they put in: the more motivated they are and the more hours they dedicate to practice, the faster they will learn and grow.</li></ol><p>Why did we structure our findings like this? Because it helps us model our challenge using a mathematical framework called finite <em>Markov decision processes</em> (MDPs). This approach helps us gain a better understanding of the problem and how to best address it.</p><h4>Markov Decission Processes</h4><p>Finite MDPs provide a useful framework to model RL problems, allowing us to abstract away from the specifics of a given problem and formulate it in a way that can be solved using RL algorithms. In doing so, we are able to transfer learnings from one problem to another, instead of having to theorise about each problem individually. This helps us to simplify the process of solving complex RL problems. Formally, a finite MDP is a control process defined by a four-tuple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/1*-dn8vgQLFPW8gq74XJPUvQ.png" /></figure><p>The four-tuple (<em>S</em>, <em>A</em>, <em>P</em>, <em>R</em>) defines four distinct components, each of which describes a specific aspect of the system. <em>S</em> and <em>A</em> define the set of <em>states</em> and <em>actions</em> respectively. Whereas, <em>P</em> denotes the <em>transition function</em> and R denotes the <em>reward function</em>. In our example, we define our friend’s mood as our set of states <em>S</em> and their practice choices as our set of actions <em>A</em>. The transition function <em>P</em>, visualised by arrows in the graph, shows us how our friend’s mood will be altered depending on the amount of studying they do. Furthermore, the reward function <em>R</em> is used to measure the progress our friend has made, which is influenced by their mood and the practice choices they make.</p><h4>Policies and value functions</h4><p>Given the MDP, we can now develop strategies for our friend. Drawing on the wisdom of our favorite cooking podcast, we are reminded that to master the art of cooking one must develop a routine of practicing a little every day. Inspired by this idea, we develop a strategy for our friend that advocates for a consistent practice schedule: practice for one hour every day. In RL theory, strategies are referred to as <em>policies</em> or <em>policy functions</em>, and are defined as mappings from the set of states to the probabilities of each possible action in that state. Formally, a policy <em>π </em>is a probability distribution over actions <em>a </em>given state <em>s.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*dkn7w7em5fpEIvy3cvhDNQ.png" /></figure><p>To adhere to the “practice a little every day” mantra, we establish a policy with a 100% probability of practicing for one hour in both the neutral and motivated states. However, in the demotivated state, we skip practice 100% of the time, since it is the only available action. This example demonstrates that policies can be deterministic, instead of returning a full probability distribution over available actions, they return a degenerate distribution with a single action which is taken exclusively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*Eq0BZSme4ZY5In61cOayug.png" /></figure><p>As much as we trust our favourite cooking podcast, we would like to find out how well our friend is doing by following our strategy. In RL lingo we speak of evaluating our policy, which is defined by the <em>value function</em>. To get a first impression, let us calculate how much knowledge our friend is gaining by following our strategy for ten days. Assuming they start the practice feeling neutral, they will gain one unit of knowledge on the first day and two units of knowledge thereafter, resulting in a total of 19 units. Conversely, if our friend had already been motivated on the first day, they would have gained 20 units of knowledge and if they had started feeling demotivated, they would have gained only 17 units.</p><p>While this calculation may seem a little arbitrary at first, there are actually a few things we can learn from it. Firstly, we intuitively found a way to assign our policy a numerical value. Secondly, we observe that this value depends on the mood our friend starts in. That said, let us have a look at the formal definition of value functions. A value function <em>v</em> of state <em>s </em>is defined as the expected <em>discounted return </em>an agent receives starting in state<em> s </em>and following policy <em>π </em>thereafter. We refer to <em>v </em>as the<em> state-value function </em>for policy <em>π.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*VKfheWImv2wRVKkc9aA2-w.png" /></figure><p>Where we define the state-value function as the expected value <em>E </em>of the discounted return <em>G </em>when starting in state<em> s. </em>As it turns out, our first approach is in fact not far off the actual definition. The only difference is that we based our calculations on the sum of knowledge gains over a fixed number of days, as opposed to the more objective expected discounted return <em>G</em>. In RL theory, the discounted return is defined as the sum of discounted future rewards:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*GMwVlaA3o_Fzg-A62Gllxw.png" /></figure><p>Where <em>R</em> denotes the reward at timestep <em>t </em>multiplied by the <em>discount rate</em> denoted by a lowercase gamma. The discount rate<em> </em>lies in the interval of zero to one and determines how much value we assign to future rewards. To better understand the implication of the discount rate on the sum of rewards let us consider the special cases of assigning gamma to zero or to one. By setting gamma to zero, we consider only immediate rewards and disregard any future rewards, meaning the discounted return would only equal the reward <em>R</em> at timestep <em>t+1</em>. Conversely, when gamma is set to one, we assign any future rewards their full value, thus the discounted return would equal the sum of all future rewards.</p><p>Equipped with the concept of value functions and discounted returns we can now properly evaluate our policy. Firstly, we need to decide on a suitable discount rate for our example. We must discard zero as a possible candidate, as it would not account for the long-term value of knowledge generation we are interested in. A discount rate of one should also be avoided, as our example does not have a natural notion of a final state; thus, any policy that includes regular practice of the instrument, no matter how ineffective, would yield an infinite amount of knowledge with enough time. Hence, chosing a discount rate of one, would make us indifferent between having our friend practice every day or once a year. After rejecting the special cases of zero and one, we have to choose a suitable discount rate between the two. The smaller the discount rate, the less value is assigned to future rewards and vice versa. For our example, we set the discount rate to 0.9 and calculate the discounted returns for each of our friend’s moods. Let us start again with the motivated state. Instead of considering only the next ten days, we calculate the sum of all discounted future rewards, resulting in 20 units of knowledge. The calculation is as follows¹:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*DFgAVgHHoQnGNe5H-go4EQ.png" /></figure><p>Note, by introducing a discount rate smaller than one, the sum of an infinite number of future rewards is still constant. The next state we wish to analyze is the neutral state. In this state, our agent choses to practice for one hour, gaining one unit of knowledge, and then transitions to the motivated state. This simplifies the calculation process tremendously, as we already know the value of the motivated state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*tUZpYj-NM-1pxGleOPIpsg.png" /></figure><p>As a final step, we can also calculate the value function of the demotivated state. The process is analogous to the neutral state, resulting in a value of a little over 17.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*yfPCvq89mXIeZrQ3Zi4oZQ.png" /></figure><p>By examining the state-value functions of our policy in all states, we can deduce that the motivated state is the most rewarding, which is why we should instruct our friend to reach it as quickly as possible and remain there.</p><p>I encourage you to consider developing alternative policy functions and evaluating them using the state-value function. While some of them may be more successful in the short term, they will not generate as much units of knowledge as our proposed theory in the long-term². If you want to dig deeper into the math behind MDPs, policies and value functions, I highly recommend “Reinforcement Learning — An Introduction” by Richard S. Sutton and Andrew G. Barto. Alternatively, I suggest checking out the “RL Course by David Silver” on YouTube.</p><h3>What’s next?</h3><p>What if our friend was not into music, but instead asked us to help them build a self-driving car, or our supervisor instructed our team to develop an improved recommender system? Unfortunately, discovering the optimal policy for our example will not help us much with other RL problems. Therefore, we need to devise algorithms that are capable of solving any finite MDP².</p><p>In the following blog posts you will explore how to apply various RL algorithms to practical examples. We will start with tabular solution methods, which are the simplest form of RL algorithms and are suitable for solving MDPs with small state and action spaces, such as the one in our example. We will then delve into deep learning to tackle more intricate RL problems with arbitrarily large state and action spaces, where tabular methods are no longer feasible. These approximate solutions methods will be the focus of the second part of this course. Finally, to conclude the course, we will cover some of the most innovative papers in the field of RL, providing a comprehensive analysis of each one, along with practical examples to illustrate the theory.</p><p>[1] In our case, the state-value function is simply equal to the discounted return. However, if the environment or the policy were to be nondeterministic, then we would need to sum over all the discounted returns, each weighted by their likelihood.</p><p>[2] Proving that the proposed policy is truly optimal in the long run is beyond the scope of this blog post. However, I encourage those interested to seek out a proof as an exercise. I would start by defining a hypothetical optimal policy, then try to demonstrate that the value function of the proposed policy is greater than or equal to the value function of the optimal policy for every state of the environment.</p><p>[3] Not all RL problems fit the mathematically idealised form of finite MDPs. Many interesting RL problems require a relaxed formulation. Nevertheless, algorithms developed for finite MDPs will often perform well on RL problems that do not fit the strict formulation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f85f3478f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/beyond-the-basics-reinforcement-learning-with-jax-part-i-introduction-and-core-concepts-4f85f3478f5">Beyond the Basics: Reinforcement Learning with Jax — Part I: Introduction and Core Concepts</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>